Math2Sym: A System for Solving Elementary Problems via Large Language Models and Symbolic Solvers

Authors: minh phu nguyen
minh phuong pham
tuan minh kha
minh man ngo
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-19 17:41:39

Abstract: Traditional models for solving math word problems (MWPs) often struggle to capture both linguistic context and arithmetic reasoning. We propose Math2Sym, a novel approach integrating large language models (LLMs) with symbolic solvers. This method leverages LLMs' language comprehension and symbolic computation's precision to efficiently convert MWPs into solvable symbolic form. We introduce the EMSF dataset for training models to formalize math problems across various complexities. On our defined test set benchmark, fine-tuned models outperform GPT-3.5 by 17% in few-shot tasks and perform comparably to GPT-4-mini on elementary math problems.

Main paper: 
 introduction: 
 Math word problems (MWPs) present a unique challenge in artificial intelligence, requiring the integration of linguistic comprehension and mathematical reasoning to solve questions based on contextual descriptions [1] . The primary obstacles lie in understanding the problem's context and translating it into appropriate mathematical operations, particularly when mapping linguistic information to complex mathematical expressions [2] .MWP solving has evolved from early rule-based systems like STUDENT [3] to machine learning methods [4] , improving accuracy but still facing challenges in complex domains. Recent advancements include Chain-of-Thought (CoT) prompting [5] , which enhances reasoning by breaking down problems into structured steps, and Program-Aided Language models (PaL) [6] , which generate Python code for external computation.Our work advances the field with the following key contributions:• We introduce a novel approach that fine-tunes models to generate symbolic forms of MWPs, enhancing language models' capabilities in converting problems into representations compatible with our custom SymPy-based solver [7] . • We present the EMSF dataset for converting elementary math word problems into symbolic form across five problem types, facilitating improved formalization of MWPs. • We demonstrate that fine-tuning 7B-parameter models on EMSF outperforms larger models such as GPT-3.5 in MWP solving.This approach enhances MWP solving accuracy and versatility, paving the way for more robust AI systems in mathematical problem-solving. Its transparent step-by-step reasoning also offers educational value, fostering a deeper understanding of problem-solving processes.2 Related Work 
mwp solvers: 
 MWP solving has evolved from early rule-based systems like STUDENT [8] , which relied on predefined schemas, to statistical machine learning models [4] , improving the mapping from linguistic input to mathematical representations. Deep learning approaches, such as encoder-decoder architectures, further advanced the field [9] . However, many models remain limited to basic arithmetic problems or linear equations, struggling with more complex tasks like systems of equations or inequalities.Recent research has focused on enhancing language models (LMs) by integrating external tools like calculators, search engines, and symbolic solvers to address limitations in precise calculations or accessing real-time information [10, 11] . Two main approaches for training LMs to use these tools have emerged: creating large, supervised datasets with explicit examples of tool usage and using few-shot learning with prompts demonstrating tool use [6, 12] .Auto-formalization, the task of converting natural language into symbolic representations, plays a central role in mathematical reasoning. Recent work in this area leverages symbolic manipulation tools like SymPy [7] , alongside proof assistants such as Isabelle/HOL [13] , to enable computational formal reasoning. Unlike [14] , which uses BERT for simpler problems, and [15] , which employs LLMs via prompting without any fine-tuning, our method targets more complex problem types with enhanced LLM-based approaches.Formalize extracted elements into precise mathematical expressions, adhering to the problem's logic and conditions.3. Conversion to Symbolic Form: Transform mathematical expressions into a predefined Symbolic Form compatible with a symbolic solver.To illustrate this process, consider the following word problem and its symbolic formalization:Word Problem: The length of a rectangle is equal to triple the width. Which system of equations can be used to find the dimensions of the rectangle if the perimeter is 86 centimeters?Answer: Define the variables and formulate the linear system of equations: Let variable x represent the length of the rectangle and variable y represent the width of the rectangle. The length of a rectangle is equal to triple the width, so the equation is x = 3y. The perimeter of the rectangle is 86 centimeters, leading to the equation 2x + 2y = 86. System of equations: {x = 3y, 2x + 2y = 86} Symbolic Form: [x -3y, 2x + 2y -86, x, y, solve] This structured symbolic form provides the solver with a clear and unambiguous mathematical representation, ensuring consistent and accurate solutions across various problem types. 
math2sym: 
 Math2Sym integrates large language models (LLMs) with symbolic solvers to address math word problems (MWPs). By transforming natural language problem descriptions into symbolic representations, this approach tackles two key challenges: understanding linguistic complexity and ensuring precise computation. LLMs extract variables and conditions from word problems, while a symbolic solver handles mathematical computations.Math2Sym converts natural language word problems into standardized Symbolic Forms through three core steps:1. Extraction of Variables: Identify relevant entities (variables, constants, relationships) from the natural language description.The problem-solving process involves the language model systematically normalizing problems into standard forms, which are then converted into predefined Symbolic Forms. To mitigate the frequent arithmetic errors produced by language models, our approach trains the model to formalize problems while avoiding direct calculations. Unlike other LLMs that attempt computations within their reasoning steps, we delegate all arithmetic to an external solver in Symbolic Form, allowing the model to focus on formalization.Our solver is built using SymPy [7] , a Python library for symbolic computation. SymPy's capability to handle a wide range of mathematical problems and its ease of use make it suitable for both current needs and future scalability. The Symbolic Form follows this structure:Each mathematical problem type is associated with a specific action, which corresponds directly to a SymPy method (e.g., 'solve' for equations or inequalities, 'igcd' for greatest common divisors).To address LLMs' tendency to produce lengthy outputs, we enclose Symbolic Form answers in double square brackets. This formatting is achieved through prompting or fine-tuning with structured data, facilitating the consistent conversion of problems into Symbolic Forms. 
experiments: 
 We developed a custom test dataset of 92 questions across five categories: greatest common divisor, least common multiple, systems of equations, linear inequalities, and compound inequalities. Questions vary in difficulty (levels 1-5) and include both word and purely mathematical problems to assess generalization. The questions in the dataset were inspired by problems found in high school mathematics textbooks and reputable online educational resources.Language models ranging from a few hundred million to approximately 7 billion parameters were fine-tuned on the EMSF dataset using Low-rank adaptation (LoRA) [16] . Detailed parameters are in A.The EMSF dataset consists of three parts, as detailed in B:• Pretrain: Focuses on standard mathematical formalization.• Basic: Synthetically generated using Mixtral 8x7B [17] , involves direct extraction of simple problem elements.• Advanced: Generated using LLaMA3 70B [18] , requires reasoning steps for complex problems.In short, the basic dataset involves direct extraction and declaration of simple problem elements, whereas the advanced dataset requires reasoning steps and aggregation of information from more complex problems.Performance was evaluated through direct evaluation, few-shot learning, and two fine-tuned settings (on basic and on advanced datasets). Model-generated symbolic forms were compared to ground truths, both processed through the symbolic solver.Evaluation scores were weighted by problem's difficulty, with the total score S total calculated as:S total = N i=1 C i × D iwhere C i ∈ {0, 1} is the correctness score for problem i, and D i ∈ {1, 2, 3, 4, 5} is based on problem's difficulty. Enhanced Performance with Solver Integration: Our experiments demonstrate that integrating symbolic solvers with LLMs significantly improved performance across models like Mistral 7B, Qwen 7B, WizardMath 7B, and GPT-3.5. This integration outperformed approaches such as Programaided Language models (PaL) and Zero-shot Chain-of-Thought (CoT) prompting. For instance, Mistral 7B showed a 37% improvement with solver integration compared to Few-Shot PaL, while Qwen 7B demonstrated a 10% increase in performance. These findings underscore the efficacy of the Math2Sym framework, which leverages symbolic solvers to enhance the natural language comprehension and reasoning capabilities of LLMs. 
results: 
 Comparison with Qwen2-Math: Our approach outperformed Qwen2-Math-Instruct 7B [19] in overall performance across problem complexities. While Qwen2-Math-Instruct excelled in high-difficulty problems (scoring 170 and solving 5/6 of the most difficult problems), it showed inconsistencies on simpler tasks. In contrast, our models, particularly WizardMath 7B, maintained consistent performance across all difficulty levels, achieving a total score of 217. This consistency demonstrates Math2Sym's versatility in handling both simple and complex tasks.Dataset-Driven Success in High-Difficulty Problem Solving : Models fine-tuned on our EMSF dataset excelled in high-difficulty problems (levels 4 and 5). WizardMath 7B achieved a score of 217, significantly outperforming Zero-shot CoT (133) and GPT-3.5-turbo's best in-context learning (179). This success stems from our dataset's ability to teach diverse problem-to-symbolic-form mapping, enhancing solver utilization. Notably, our approach yielded results comparable to GPT-4o Mini, demonstrating its competitiveness in challenging problems.Performance of Smaller Models in Specific Contexts: In specific contexts, models with fewer than 1 billion parameters occasionally outperformed mid-sized models. For instance, the finetuned Qwen 0.5B model scored approximately 10% higher than the Qwen 1.8B model, suggesting that smaller models may benefit from improved learning efficiency before encountering overfitting issues. However, while smaller models excelled in simpler tasks, larger models like WizardMath 7B consistently outperformed them on complex problems, highlighting the importance of model size in managing problem complexity (See 1). For fine-tuning the models in our experiments, we used LLama-Factory [20] , an open-source framework for inference and fine-tuning of language models. All fine-tuning was performed on a single NVIDIA L4 GPU using the LoRA method. The hyperparameters used during training are as follows, with all other parameters set to their default values:• LoRA: rank = 16, alpha = 16, learning rate = 5e-05The training process consisted of three steps:1. Pre-training with a dataset containing non-word problems.2. Supervised fine-tuning (SFT) on the Basic dataset.3. Supervised fine-tuning (SFT) on the Advanced dataset.The system prompt used during training was:"You are a Math Teacher. Your goal is to understand a math word problem, recognize and distinguish the type of problem, define the variables (if needed), and formulate the problem in symbolic form."To mitigate data leakage in our dataset, we employed FAISS [21] and Sentence Transformers [22] for vector embedding and similarity filtering. Each mathematical word problem was embedded into a numerical vector using the all-MiniLM-L6-v2 pretrained model from Sentence Transformers.We then applied FAISS to detect and filter out similar data points, utilizing a cosine similarity threshold of 0.8. Any pairs of word problems exceeding this threshold were removed to prevent overlap between the training and test sets. This approach ensured the integrity of the dataset, minimizing leakage and maintaining valid evaluation metrics for model performance.The pre-train dataset comprises 1 million samples generated from standard mathematical problems with randomly selected constants and parameters. These problems are converted into symbolic form, providing the model with a foundation for handling structured mathematical tasks without natural language. The basic supervised fine-tuning (SFT) dataset, containing 149,000 samples, was generated using the Mixtral 8x7b model [17] with few-shot prompting. Data generation parameters were set at temperature = 1.5, top_p = 0.9, and max_length = 512. The dataset consists of 38% single-variable and compound inequalities, 28% two-variable system of equations, and 17% greatest common divisor and least common multiple problems. Word Problem: Today, both the soccer team and the basketball team had games. The soccer team plays every three days, and the basketball team plays every five days. When will both teams have games on the same day again?Find the least common multiple of 3 and 5. Symbolic form: [3, 5, ilcm]The advanced dataset was synthetically generated through a prompting process utilizing a larger language model due to the increased complexity of the problems. Specifically, we employed the LLaMA3-70B-Instruct model [18] . The data generation method mirrored that of the basic dataset, with parameters set to a temperature of 1.5, a top_p of 0.9, and a maximum length of 512 tokens, but with variations in the prompts to accommodate the more complex problem types.Each solution in the advanced dataset follows a structured approach:1. Analysis and Summarization: The system begins by analyzing and summarizing the key points of the word problem. 2. Problem Type Identification: It identifies the specific type of mathematical problem presented. 3. Information Extraction and Reformulation: Essential variables and information are extracted from the problem, which are then used to reformulate the word problem into a standard form. 4. Symbolic Representation: The standard form is converted into a symbolic representation, preparing it for processing by the symbolic solver to derive the solution.The advanced supervised learning data maintains a structure similar to the basic dataset, consisting of word problems and their conversion to symbolic forms. However, the problems in this dataset are more intricate, including types such as percentage interest problems.The advanced dataset comprises 79,000 samples, distributed as follows: 60% single-variable and compound inequality problems, 25% two-variable equation problems, and 15% least common multiple and greatest common divisor problems. Word Problem: The soccer team plays every three days, and the basketball team plays every five days.When will both teams have games on the same day again?Break down the word problem: 1. Find when both teams will have games on the same day. Given Problem: Anthony's mom gave him 30 dollars while Jennifer's mom gave her 36 dollars. They want to give the same amount of money to a charity. What is the greatest donation they can make? To solve this problem, follow the steps provided in the examples.Step 1:Find the prime factorization of each number.-30 = 2 * 3 * 5 -36 = 2 2 * 3 2Step 2:Identify the common prime factors and their lowest powers.-The common prime factors are 2 and 3, with the lowest powers being 2 1 and 3 1 .The greatest common divisor is the product of the common prime factors.-GCD = 2 1 * 3 1 = 6Step 3:To find the greatest donation they can make, divide the amount of money each person has by the GCD.-For Anthony: 30 / 6 = 5 -For Jennifer: 36 / 6 = 6The greatest donation they can make is the GCD multiplied by the smallest quotient fromStep 4.-Greatest donation = 6 * 5 = 30Final Answer: The greatest donation they can make is 30 dollars.Break down the word problem into smaller information:1. Our goal is to find the greatest donation Anthony and Jennifer can make. The problem asks for the number of cars the student government officers must wash to have enough money to pay for the trip which implies finding the minimum value that satisfies the constraints. The total cost of the trip should be greater than or equal to the total funds available is a lower bound constraint. We can use the inequality to solve this problem.Step 3: Calculate how many cars they need to wash to earn the remaining amount. Final Answer: They need to wash 69 cars to have enough money to pay for the trip.Final Answer: x >= 49 The least number of days it will take each person to save the same amount of money is the LCM of the number of days it takes each to save 36 dollars, which is 12 days.Final Answer: 12 days Break down the word problem into smaller information:1. Our goal is to find the least number of days it will take each person to save the same amount of money. 2. Steve can save 9 dollars every day.3. Maria can save 12 dollars every day.The problem involves two people saving money at different rates. The question asks for the least number of days it will take each person to save the same amount of money, indicating that we need to find the least common multiple of the number of days to determine when they will have the same amount of money.[ [9, 12, ilcm] ]Final Answer: 36The code and the new EMSF dataset are available at https://github.com/pepoo20/Math2Sym, https://huggingface.co/datasets/MathSymbol/EMSF † Corresponding author 38th Conference on Neural Information Processing Systems (NeurIPS2024). 
 
 Appendix:  



Reviews: review: title: title: The paper presents a novel approach combining LLMs with symbolic solvers for math word problems, showing promising results and contributing a new dataset, but lacks clarity in some evaluations and has potential limitations in handling integer-only solutions.
main_review: main_review: The paper presents an interesting and novel approach to solving math word problems by combining LLMs with symbolic solvers. The method shows promising results and has the potential for educational applications. The authors also contribute to a novel EMSF dataset for converting elementary math word problems into symbolic form. At the same time, this paper has some limitations in methodology and evaluation listed below.

# Pros

## Methodology

1. The integration of LLMs with symbolic solvers addresses both linguistic comprehension and precise computation in MWP solving.

2. This approach can solve various types of math problems, beyond simple addition and multiplication problems.

3. The authors contributed to a novel and comprehensive dataset that is proven to be useful in their evaluation chapter.

## Evaluation

1. The authors did extensive evaluation with different settings on different models and the results are sound and reasonable. The results are presented mostly clearly and concisely in table 1.

2. The authors included a few pairs of examples between their approach and Qwen2Math model in the appendix, enabling intuitive understanding of their approach.

## Writings

1. The paper is well-structured and free of grammar mistakes.

# Cons

## Methodology

1. The authors' solver is built using SymPy, but Sympy doesn't directly support specifying integer-only solutions. The authors' test set clearly contains problems that require an integral solution, but it's unclear whether and how the authors dealt with it. Table 6, for example, asks for "how many cars must they wash", implying the answer should be rounded up to the next integer. The symbolic form derived by the authors' approach doesn't specify such constraint. Even though the final answer **happens** to be an integer, it is unclear how this approach could be generalized if symbolic execution gives a floating number as result.


## Evaluation

1. The authors didn't evaluate their approach on larger open-source LLMs. The authors also didn't compare their results with more advanced closed-source LLMs like gpt-4o (although they compared with gpt-4o mini).

2. Table 1 presented "max score" but didn't explain where this score came from, as this number does not appear in elsewhere in the paper. Furthermore, L122-127 mentioned Qwen2-Math-Instruct 7B whose score is not listed except its score in high-difficulty problems.

3. L285, table 4 compares "their solution" (i.e. Qwen2Math) and "our solution" using an example, but a) it's unclear what our solution is - which setting in table 1 does it correspond to? b) the authors simply listed the solutions by both approaches without comments. Table 4 shows an example where "our solution" performs worse than the baseline by giving an incorrect result, but the authors didn't acknowledge, comment on or explain it.

## Writings

1. L361 claims the dataset and code are provided, but I couldn't find either. Since the checklist is not required anyway, I would consider it as an honest mistake and presume the authors would open-source their dataset and code upon acceptance.

2. L28 wrongly refers to GPT-3.5 as an open-source LLM.
title: title: Review of Math2Sym
main_review: main_review: Needs improvement: 

The idea of using LLM to convert math problems into symbolic forms is not new. Although it is challenging to autoformalize abstract proof based math problems into Isabelle/HOL, the task of converting elementary school word problems into equations which is then delegated to SymPy to be solved is not a challenging research problem but rather one of data curation and careful engineering. Since the class of math problems solvable by the work presented is very limited and elementary, the work lacks major innovation in mathematical reasoning with AI.
title: title: A clever hybrid approach
main_review: main_review: ### Summary

The paper proposes a hybrid approach for solving math word problems (MWPs) by integrating LLMs with symbolic solvers. The Math2Sym framework enhances MWP-solving by converting natural language descriptions into symbolic form using fine-tuned LLMs, and delegating arithmetic and symbolic manipulations to a SymPy-based solver. They also introduce a new dataset, EMSF, which includes problems across various levels of complexity, and demonstrates that the Math2Sym system significantly improves performance in elementary-level math tasks when compared to other LLM-based approaches.

### Strengths

1. The Math2Sym system effectively combines the strengths of LLMs for language comprehension and symbolic solvers for mathematical precision, resulting in a more accurate and interpretable solution process. This division of tasks allows the LLM to focus on problem interpretation and avoids errors in direct computations.
2. The introduction of the EMSF dataset provides a valuable resource for fine-tuning LLMs to solve MWPs. But some details on the dataset are missing, eg: size of dataset, etc.
3. The paper presents strong empirical results, showing that Math2Sym significantly outperforms baselines such as GPT-3.5 and is competitive with GPT-4-mini on elementary math problems. This improvement is particularly notable in few-shot learning tasks and on more difficult problems.

### Weaknesses

1. The system is primarily evaluated on elementary-level problems, which limits its generalizability to more complex math tasks. Although the paper briefly mentions potential scalability, no experiments are conducted on advanced topics such as calculus or higher-dimensional problems, which would be important for broader adoption.
2. The model is evaluated on synthetic and textbook-inspired datasets, but real-world applications (such as noisy data or problems with ambiguous language) are not explored. This limits the paper’s claims about the system’s robustness in practical settings.
3. The paper discusses performance drops with smaller models on advanced datasets, suggesting that the system’s effectiveness is highly dependent on dataset complexity and model size. 

### Questions

1. How does Math2Sym handle ambiguities in problem formulation, such as vague or incomplete descriptions, which are often encountered in real-world problems?
2. Can this system be adapted to solve higher-level mathematical problems beyond the elementary scope, such as calculus or multivariable algebra, and what modifications would be required for this?
3. What are the computational limitations or overheads when using symbolic solvers like SymPy at larger scales? Could this approach still perform efficiently with more complex problems or larger datasets?
4. How robust is Math2Sym in handling noisy or poorly structured input data, which may deviate from textbook-style problems?
title: title: Convert MWP solving into symbolic formats
main_review: main_review: The authors propose Math2Sym, a methodology that utilizes sympy to construct symbolic answer forms from MWP for effective solving such questions. Training 7B models over the constructed EMSF dataset show great improvements in solving Math problems with symbolic formats.

Pros:

- Math2Sym successfully integrates the language understanding strengths of LLMs with the precise reasoning abilities of a symbolic solver, i.e., Sympy. And the author further construct the EMSF dataset which enhances model training by providing diverse and structured math problems of varying difficulty.

Cons:

- I understand there has been page limit for submission, but I hope authors could give some more analysis apart from only one score table.
- Constructing EMSF dataset also requires supervision or help from larger models, e.g., (Mixtral 8x7B or Llama-70B); I wonder if authors can make some experiments over this model sizes to check the dataset’s effect.