Learning Mathematical Rules with Large Language Models

Authors: antoine  gorceix
j p morgan
a i research
bastien  le chenadec
ahmad  rammal
nelson  vadori
manuela  veloso
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-18 16:57:24

Abstract: In this paper, we study the ability of large language models to learn specific mathematical rules such as distributivity or simplifying equations. We present an empirical analysis of their ability to generalize these rules, as well as to reuse them in the context of word problems. For this purpose, we provide a rigorous methodology to build synthetic data incorporating such rules, and perform fine-tuning of large language models on such data. Our experiments show that our model can learn and generalize these rules to some extent, as well as suitably reuse them in the context of word problems.

Main paper: 
 introduction: 
 We focus on a specific aspect of mathematical reasoning, namely the ability of large language models (LLMs) to learn specific abstract mathematical rules and to reuse them while answering word problems, namely questions formulated in natural language that are usually made up of a few sentences describing a scenario that needs to be solved through mathematics. We will also refer to these mathematical rules as "skills". Thus, we study the following research question: Can LLMs learn and generalize specific mathematical rules, and apply them in contexts that have not been seen during training, for example when answering word problems?We will fine-tune models on carefully built synthetic data reflecting the mathematical rules of interest, and we provide a detailed description of our methodology for building such data in section 2 and appendix B. Broadly speaking, we want our training data to be presented similarly to what you would find in a mathematics textbook. For example, focusing on how to rearrange or simplify an equation, or how to use the distributivity property of the addition operator. But without word problems.The data at test time, however, will contain word problems where the model first needs to translate the question in natural language into one or more equations, and then use various rules to solve these equations. This is what we will call bottom-up generalization, namely the model's ability to go from mathematical rules to answering word problems. We also provide an empirical study on the model's ability to generalize when increasing the mathematical complexity of the task. That is, when we increase parameters such as the number of variables, the number of equations, the variable name token length, and so on. We will call this top-down generalization, as we zoom into a given mathematical rule to make it more complex. We comment on the related work in appendix A.Our contributions. We provide a rigorous methodology to create synthetic data containing specific mathematical rules such as manipulating equations (section 2). We show how fine-tuning models on those rules allows them to be reused in the context of word problems, while maintaining performanceIn section 4 we focus on top-down generalization. We consider rules such as distributivity, exponentiation, manipulation of single and pairs of equations as well as solving single steps of Gaussian elimination, all presented in section B.2. In particular, we focus on the model's ability to apply those rules when the variables appearing in the equations are arbitrary strings, and study the impact on generalization performance. For example: Q: Expand this expression: (-5+soccer-dog) * (blue-sky). A: By the distributivity property: -5 * blue+5 * sky +soccer * blue-soccer * sky -dog * blue+dog * sky. The goal is to learn the fundamental nature of the corresponding operators, which do not depend on the variable names.3 Bottom-Up Generalization: Going from Mathematical Rules to Word ProblemsWe consider three classes of word problems. Finding a solution to these problems involves mainly two steps. First, the model needs to correctly translate the question in natural language into one or multiple equations. Second, the model needs to apply one or more mathematical skills to correctly manipulate the equations to answer the question. We choose our examples such that the baseline model is able to perform the first step but struggles on the second. Our goal when performing fine-tuning is as follows: keep the general knowledge required to solve the first step, while providing new tools to also solve the second. We write the solutions to these problems using chain-of-thought, i.e. we break down the reasoning in several steps. We will fine-tune Llama-3 8B Instruct on on a mix of synthetic data described in section B.1 together with data from the Orca dataset Mukherjee et al. (2023) acting as a regularizer in order to avoid catastrophic forgetting and maintain performance on common benchmarks (see section C.4). Additional details are provided in appendix C.Quadratic Polynomials. We consider simple geometric problems involving the calculation of areas for shapes with the same unknown side length. The total area of all shapes is known, while the unknown side length is to be determined. The solution involves two main steps. The first one is to formulate a quadratic equation that models the relationship between the known areas and the unknown side length. This equation is always a quadratic polynomial. The second step is to find the roots of this polynomial to find the unknown side length. Problems are constructed to always yield one positive and one negative root, ensuring no ambiguity in the solution. We generate many such problems by varying the names (pools, fields, etc.) and shapes (square, triangle, parallelogram, rectangle) of the surfaces, the number of such surfaces per shape (in average, 2.5 surfaces per shape per word problem), as well as the values of the areas and side lengths appearing in the problem. We present an example of prompt given to the model in Figure 1 .We provide three examples of prompts and responses to the model (3-shot), and assess the validity of the numerical solution. We evaluate the performance of our fine-tuned model against a baseline across varying levels of difficulty. The difficulty scale ranges from 0 to 100%, in increments of 20%, corresponding to the proportion of problems with non-integer roots. Non-integer roots present a greater challenge as they require more complex calculations compared to integer roots, which the baseline model often guesses via straightforward factorization. To accurately determine non-integer roots, the discriminant of the polynomial must be calculated. We present our results in table 1 .While the baseline model successfully translates the problem into an equation, it encounters difficulties in finding the roots of the quadratic polynomial. Instead of applying the discriminant method, it prematurely attempts a hazardous factorization, often leading to errors. In contrast, the fine-tuned model consistently utilizes the discriminant approach, a result of targeted fine-tuning on a quadratic polynomial solving task. As a consequence, our fine-tuned model is able to outperform the baseline.Figure 1 : Word problem example -quadratic polynomial.Prompt: Jim has a total of 2 swimming pools. The first swimming pool is a square, with an unknown side length x. The second is a rectangle with one side measuring 1 meters and the other being the unknown side length x. The total area covered by these swimming pools is 2 square meters. What is the unknown side length x? Physics Problems Involving Resistors. We consider simple electrical circuits composed of n resistors connected either all in parallel or all in series. The model is tasked with finding the equation governing the circuit and isolating a variable in the equation. We provide 3 few-shot examples of prompts and answers to the model, and then evaluate the correctness of the symbolic solution. We present an example of prompt given to the model in Figure 2 . We compare the performance of a Table 2 shows the results of the models on resistors in series and parallel. The fine-tuned model performs perfectly on resistors in series. On resistors in parallel, the fine-tuned model performs significantly better than the baseline model, but still struggles with more complex configurations despite being trained on these equations.Fruit Baskets. Alice and Bob buy different quantities of fruits, each with a specific price, and both end up paying the same total amount. The goal is to find the price of one fruit based on the prices and quantities of the others. We generate these problems by varying the names of the fruits, the number of fruits, and the relationships between their quantities and prices. In our examples, all the fruits depend on the quantities and price of the first fruit, as illustrated in Figure 3 . This involves setting up an equation, substituting variables, simplifying it, and solving for the unknown. The problem reduces to solving a linear equation involving three variables. We can increase the complexity of the problem by simply increasing the number of fruits that are purchased.We evaluate the model's answer by extracting its symbolic output and comparing it to the correct one. We observed an improvement in the fine-tuned model compared to the Llama-3 8B Instruct Prompt: Alice and Bob went to the grocery store and bought the following items: -bananas: Alice bought qA 1 , and Bob bought qB 1 . The price of a single one is p1.-blueberries: Alice bought qA 2 where qA 2 = 2 × p1, and Bob bought qB 2 where qB 2 = 8 × qA 1 . The price of a single one is p2 where p2 = 5 × qB 1 . Both ended up paying the same total price. Find the price of bananas in terms of qA 1 and qB 1 .baseline. The latter scores 19% versus 35% with the former in the case of two fruits. For three fruits, the baseline scores 0% against 13% for the fine-tuned model. While the baseline model can set up the full equation, it struggles with simplifying or solving it. The fine-tuned model, though not perfect, performs better in finding the correct answer due to its fine-tuning on mathematical tasks such as expression simplification and equation solving.We also considered assigning numerical values to the variables in that problem, requiring the model to perform calculations to find a numerical solution instead of a symbolic one. We illustrate this in figure 9 . We found that the models' performances were very similar to the symbolic case. We refer to the appendix C.3, table 3 for a full overview on the results.On the necessity to align training data with the word problems. Our experiments reveal limitations in the ability to generalize specific mathematical rules to word problems. First, we have to make sure the form of the equations in the training data is the same as the form of the equations in the test data. For instance, the model is unable to perform on the resistors in parallel problem when it is trained to put fractions over a common denominator in its answer. We say that the model is triggered in the wrong modality. Another example with the resistors in parallel is as follows: if in the training data, we put equations in the form "A/B", and in the test data we put them in the form "A * 1/B", the model will not be able to perform. We say that the model is not triggered. This second example illustrates the importance of the choice of the few-shot examples, because the form of the equation will generally match the form in the few-shot examples. 
variable substitution:: 
 • If the simplified expressions are not identical, identify all variables in both expressions.• For a specified number of instances (default is 10), randomly assign numerical values to these variables. • Substitute these values into the simplified expressions. • Check if the numerical expressions are equal for each set of random values.• If they are equal in all instances, return True; otherwise, return False.After verifying that the output and the ground truth are equivalent, we ensure that the output is different from the input to confirm that the model is not merely repeating the input. Depending on the specific rule, we then apply one of the following evaluation tests: Distributivity: Using SymPy expansion function, we check if the model's output is equivalent to the ground truth. 
top-down generalization: increasing the mathematical complexity of the task: 
 We fine-tune Llama-2 7B Chat on all the synthetic data described in section B.2 (and that data only) and evaluate the model's ability to perform the following rules: distributivity, commutativity, division, exponentiation, variable evaluation, remarkable identities, single equation and two equations manipulation. We check in section D.4 that the performance on general knowledge benchmarks remains relatively stable, and present experimental details in section D.6.Solving systems of equations by recursive call of our model. In section D.2, we train the model on individual steps of the Gaussian elimination algorithm and show at test time that we can solve a full system by recursively calling the model. An example is presented in figure 5 .Experiments on various mathematical rules. We analyze the ability of our model to perform the rules of interest under different configuration regarding the training vocabulary. We find that our finetuned Llama-2 model consistently outperforms other models across all considered mathematical rules.The full results are presented in section D.3. In section D.5, we present the model's performance on some word problems, in particular we emphasize its ability to combine skills in figures 15 and 16.Ablation study on the distributivity rule. In section D.1 we focus on the distributivity rule. We train the model on data where the variables appearing in the equations take value in subsets of increasing tokenizer vocabulary sizes. We find that training on larger vocabulary sizes improves the ability of the model to generalize distributivity to unseen variable names as well as to increasing the number of variables, see figure 4 .  
conclusion: 
 We showed how fine-tuning models on some specific mathematical rules allows them to be reused in the context of word problems (bottom-up generalization), and also focused on the ability to generalize specific rules such as distributivity and manipulating equations (top-down generalization). Future research directions could include building a robust and rigorous methodology to create word problem data from a set of mathematical rules. 
 
 Appendix: In recent years, researchers have trained large language models (LLMs) on data of unprecedented size, and studying the emergent capabilities of these models on various tasks has been a central focus Bubeck et al. (2023) . Nevertheless, mathematical reasoning remains a challenge, although the rate of improvement of LLMs on solving mathematical problems has been significant Ahn et al. (2024) . A prize was even launched with the goal of getting AI to perform at gold medal level at the International Mathematical Olympiad XTX Investments (2024), emphasizing the importance of this research topic. In 2024, Numina won the first progress prize. Their recipe involved tool integration as well as fine-tuning DeepSeek Math Shao et al. (2024) , a model built using scalable math pre-training.Properly evaluating LLMs on mathematical reasoning tasks presents significant challenges, among which: i) data contamination: since LLMs are trained on vast amounts of data, it is often not clear that some problems have not been seen during training; ii) assessment of proof correctness: it is not easy to automatically determine whether a sequence of mathematical reasoning steps -and more generally a proof -is correct. On this topic we note the work conducted with proof assistants such as Lean Yang et al. (2023) , for which we can immediately determine whether a proof is true or false as mathematical reasoning is seen as a computer program; iii) the variety of mathematical problems is significant and their complexity/difficulty is not trivial to establish. On this topic we believe that work allowing to suitably categorize problems would be beneficial to the community.ToRA Gou et al. (2024) 2022) solves scientific and mathematical questions in natural language, generates step-by-step solutions using Latex notation, and was trained on data composed of scientific and mathematical information. Yuan et al. (2023) applies rejection sampling fine-tuning (RFT), that uses supervised models to generate and collect correct reasoning paths in order to augment fine-tuning datasets. Zheng et al. (2023) proposes progressive-hint prompting that enables automatic multiple interactions between users and LLMs by using previously generated answers as hints to progressively guide towards correct answers. Wang et al. (2023) proposes plan-and-solve prompting, that consists of two components: first, devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan. Bubeck et al. (2023) present among other things an experimental study of GPT4's mathematical reasoning capabilities. Jelassi et al. (2023) examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training . Lample and Charton (2020) present a deep learning approach for tasks such as symbolic integration and solving differential equations. LeanDojo Yang et al. (2023) explores mathematical reasoning using the Lean proof assistant, among other things they suitably retrieve relevant premises from a vast math library. Mirzadeh et al. (2024) presents an interesting study on how model performance varies when slightly varying problems from the GSM8K benchmark. Chen et al. (2024) studies the impact of mixing different types of data on mathematical reasoning ability. Finally, the recent survey paper Ahn et al. (2024) presents the state of the field on the topic of mathematical reasoning.Our data is a combination of text and mathematical expressions, which are built from a set of types and operations. A type is a fundamental object that can be manipulated (a variable, an integer, a decimal number, etc.). By applying arithmetic operations to these types, we can construct complex mathematical expressions in the form of Abstract Syntax Trees (AST).Specifically we consider the following types for our synthetic data:• integers,• decimals with up to two decimal places,• arbitrary strings, either made from the concatenation of k tokens taken from a subset of the tokenizer's vocabulary1 which we call full vocabulary, or from the concatenation of a latin or greek letter with a digit which we call restricted vocabulary,We overload our programming language's arithmetic operators (+, -, * , /, ˆ) to implement a set of simplification rules. For instance, when adding two integers, we generally simplify the expression by evaluating the sum (though each simplification rule can be turned off). When no simplification applies, a node is added to the AST. This custom data structure allows us to control precise simplification details such as the order of terms, whether to evaluate numerical expressions, etc. In contrast, using a symbolic mathematics library such as Sympy would not allow us to control these details, as some simplifications are performed systematically. We also implement a translation function that converts our custom data structure to Sympy's data structure, and vice versa, to handle more advanced simplification rules externally, striking a balance between control and flexibility.Types only hold their own value (e.g. 4, "x", 12.51), and the negative sign is considered a unary operation, while the other operations are binary. Thus our expressions are ASTs where each leaf is a type, and each internal node is a unary or binary operation. In simpler cases we might consider expressions of the formω 1 ⊙ 1 ω 2 ⊙ 2 • • • ⊙ n-1 ω n ,where each ω i is a type, and each ⊙ i is an arithmetic operation.We are then able to represent our data structure as a string, taking into account operator precedence and associativity. We can thus construct mathematical expressions programatically, and generate prompts for our models. All of our prompts are of the form * start Question * mid Answer * end , where * start , * mid and * end are model-specific strings that separate the question from the answer. For example* start =
, * mid =
 and * end is empty in the case of Llama-2 Chat. Similar string are used in the case of Llama-3 Instruct. In the below, ω i and ⊙ i will denote respectively types and operations, which value can change depending on the context. For each one of the cases below, the number of terms appearing in the expressions is sampled randomly between some pre-specified lower and upper bounds.For the rules below, when relevant we use Sympy to solve equations symbolically and factorize the result.Quadratic polynomials. We want our model to solve quadratic polynomials. ω 1 , ω 2 , ω 3 , are random integers and ω 4 is an arbitrary token.• Question. Solve this quadratic equation:ω 1 × ω 2 4 + ω 2 × ω 4 + ω 3 = 0. • Answer. We first calculate the discriminant ∆ = ω 2 2 -4ω 1 ω 3 . -If ∆ > 0:The equation has two distinct real roots:* ω 4 = -ω2± √ ∆ 2ω1 -If ∆ = 0:The equation has exactly one real root:* ω 4 = -ω2 2ω1 -if ∆ < 0: The equation has two complex conjugate roots. * ω 4 = -ω2±i √ ∆ 2ω1• Example. Question. Solve this quadratic equation: x 2 + x -2 = 0. Answer. To solve this quadratic equation, we first calculate the discriminant ∆: ∆ = ω 2 2 -4ω 1 ω 3 = 1 2 -4(1)(-2) = 9. Since ∆ > 0, the equation has two distinct real roots, calculated using the quadratic formula:x = -ω2± √ ∆ 2a. Substituting the values, we get: root 1:-1+ √ 9 2 = 1, root 2: -1- √ 9 2 = -2.First-order single equation. We consider a single equation that we solve for a certain variable. To ensure simplicity, we focus on first-order equations, which can be solved through straightforward algebraic transformations without the need for advanced techniques. We generate the answer using SymPy. The equation ω A consists of terms on both sides,ω A := n i=1 a i • ω i = n i=1 b i • ω i , for a variable ω k . • Question. Solve for the variable ω k in the equation: ω A . • Answer. ω k = i̸ =k ωi.(bi-ai) a k -b k . • Example. Question. Solve for the variable x in 13 • x • y + 24 • y 2 = 17 • x • z + 12 • y • z. Answer. x = 12•y•(z-2•y) 13•y-17•z .Simplify expression. We want the model to reduce an expressionω A := n i=1 a i •ω i + n i=1 b i•ω i to have it in its canonical form, by grouping terms and simplifying some othersω B := n i=1 (a i +b i )•ω i .We generate the answer using SymPy. Resistors equations. We consider specific equations that appear in the context of electrical circuits, where resistors are connected in parallel or in series. In the first case, we isolate a variable in an equation of the form ωA := ω 1 = ω 2 × n i=3 ω i : • Question. Solve for the variable ω k in ω A . • Answer. ω k = ω1 ω2 -3≤i≤n;i̸ =k ω i • Example. Question. Solve for the variable x in u = v × (w + x). Answer. x = u v -w.In the second case, we isolate a variable in an equation of the form ωB := ω 1 = ω 2 /( n i=3 1/ω i ): • Question. Solve for the variable ω k in ω B . • Answer. ω k = 1/ ω2 ω1 -3≤i≤n;i̸ =k 1/ω i • Example. Question. Solve for the variable x in u = v/(1/w + 1/x). Answer. x = 1/(v/u -1/w).Distributivity. We expand an expression ω A consisting of the product of two brackets ω A = ( Single equation manipulation. We consider two rules. The first consists in computing an affine transformation ω 3 * ω A + ω 4 of an equation ω A of the form ω 1 = ω 2 , namely ω 3 * ω 1 + ω 4 = ω 3 * ω 2 + ω 4 . The second rule consists in "simplifying" an equation ω A of the form ω 1 = ω 2 and performs three steps at once: putting the equation in standard form2 , canceling out terms on both sides, and factorizing the remaining terms. We use the convention that equations are always encapsulated within two semicolons, in order to help the model recognize equations.• Questions. a) Assumptions: E1 ;ω A ;. Compute: Commutativity. We study the commutativity of an operation ⊙ A ∈ { * , ±} in an expressionω 3 * E1 + ω 4 . bω A := ω 1 ⊙ 1 ω 2 ⊙ 2 • • • ⊙ n-1 ω n . If ⊙ A = * , then ⊙ i = * ∀i, and if ⊙ A = ±, then each ⊙ i is either + orwith equal probability. We specify the two types ω i , ω j among n to which we want to apply commutativity, and the commuted expression ⊙ B is the same as ⊙ A but where we switch the positions of ω i , ω j . If one of the latter appears more than once in the expression (i.e. ω i = ω k or ω j = ω k for some k ̸ = i, j), we choose the first from the left.• Question. Apply the commutativity property of ⊙ 1 to ω 1 , ω 2 in ω A .• Answer. By the commutativity property of ⊙ 1 :ω A = ω B .• Example. Question. Apply the commutativity property of * to cat, 5 in 5 * cat * soccer.Answer. We get: cat * 5 * soccer.Division. We consider three properties of the division:ω1⊙1ω2 ω3 = ω1 ω3 ⊙ 1 ω2 ω3 , where ⊙ 1 ∈ {+, -}; ω1 * ω2 ω3 * ω4 = ω1 ω3 * ω2 ω4 ; ω 1 ω 2 ω3 = ω1 ω2 * ω3 .• Question. Use a fundamental property of the division in ω A .• Answer. By a property of the division: ω B .• Example. Question.Use a fundamental property of the division in (-sky * blue)/(tree * -cloud). Answer. By a property of the division: (-sky)/(tree) * (blue)/(-cloud).Exponentiation. We consider five properties of the exponentiation:ω 0 = 1; the definition of exponentiation ω n = ω * • • • * ω n timesif n is a positive integer, the reciprocal of the latter if n is a negative integer; if n, m are signed integers:ω n * ω m = ω n+m ; ω n 1 * ω n 2 = (ω 1 * ω 2 ) n ; ω n ω m = ω n-m .• Question. Use a fundamental property of the exponentiation: (bluesky -3 )/(bluesky 9 ). Answer. By a property of the exponentiation: bluesky -12 .Question. Apply the definition of the exponentiation to: bluesky 3 . Answer. By definition of the exponentiation: bluesky * bluesky * bluesky.Question. Give the value of: bluesky 0 . Answer. By fundamental property of the exponentiation: bluesky 0 = 1.Variable evaluation. We teach the model to substitute k ≥ 0 types in a given equation, based on some assumptions on these types (if k = 0, nothing occurs). Precisely, assume that we have an equationω A of the form ω 1 ⊙ 1 ω 2 ⊙ 2 • • • ⊙ n-1 ω n = ω n+1 ⊙ n+1 ω n+2 ⊙ n+2 • • • ⊙ n+m-1 ω n+m ,and that by assumption ω ni = ω ′ ni for a set of indexes {n i } and some types ω ′ ni . Then, the new equation ω B is the same as ω A but with ω ′ ni replaced by ω ni .• Question. Assumptions:ω n1 = ω ′ n1 , . . . , ω n k = ω ′ n k .Based on the assumptions, evaluate ω A .• Answer. The evaluated expression: ω B .• Example. Question. Assumptions: sky = 2, blueberry = cat. Based on the assumptions, evaluate dog -tree + sky = blueberry. Answer. The evaluated expression: dog -tree + 2 = cat.Remarkable identities. We consider the remarkable identity(ω 1 ⊙ 1 ω 2 ) 2 = ω 2 1 + ω 2 2 ⊙ 1 2ω 1 ω 2 , where ⊙ 1 ∈ {+, -}.• Question. Expand this expression: ω A .• Answer. By the remarkable identity properties: ω B .• Example. Question Expand this expression: (sky -blueberry) 2 . Answer. By the remarkable identity properties: sky 2 + blueberry 2 -2 * sky * blueberry.Combination of two equations. We consider two skills. The first skill consists in computing an affine transformation ω 5 * ω A +ω 6 * ω B of two equations ω A , ω B of respective forms ω 1 = ω 2 and ω 3 = ω 4 .Here, ω 1 , ω 2 , ω 3 , ω 4 are expression types. This yields the equation ω 5 * ω 1 +ω 6 * ω 3 = ω 5 * ω 2 +ω 6 * ω 4 .The second skill consists in being able to determine whether two equations are equivalent. This is done in several steps: first, the two equations are put into standard form. Then, we compute the difference of their left-hand sides. If we get 0 = 0, then we conclude that the two equations are equivalent, otherwise they are not as the left-hand residual is not zero. In the latter case, the residual is provided in factorized form.Remember that Gaussian elimination is a method to solve a system of n linear equations in k steps. Each step transforms the system at step i to a new, simpler system at step i + 1. Our aim is to teach the model to perform any step i → i + 1. For this, we create a system of k equations with n ≥ k variables. The latter are always taken to be of the m-token base type. Then, we generate a matrix M of coefficients of size k × (n + 1) associated with the system, where the coefficients are arbitrary types:M =     a 11 a 12 . . . a 1n | b 1 a 21 a 22 . . . a 2n | b 2 . . . . . . . . . . . . | . . . a k1 a k2 . . . a kn | b k    Let x 1 , . . . , x n be the variables, thus our system of equations at step 0 is:a 11 x 1 + a 12 x 2 + . . . + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + . . . + a 2n x n = b 2 . . . a k1 x 1 + a k2 x 2 + . . . + a kn x n = b kWe then apply a step of Gaussian elimination, which is a method that transforms the coefficient matrix of the system into row-echelon form and then back-substitutes. First, we perform row operations to transform the augmented matrix into row-echelon form. The goal is to create zeros below the diagonal elements. To do this, we first divide the first row L 1 by the coefficient of x 1 and then ∀i ∈ [2, k] we update the expression of the i th row L i , L i → L i -a i • L 1 . This eliminates the x 1 variable from equation i. Then we do the same ∀i ∈[2, k], ∀j ∈ [i, k] we update L j → L j -a j • L i .This eliminates variable i from each equation j. At the end of this first phase, we obtain a new matrix of coefficients:M ′ =     1 a ′ 12 . . . a ′ 1n | b ′ 1 0 1 . . . a ′ 2n | b ′ 2 . . . . . . . . . . . . | . . . 0 0 . . . 1 | b ′ k    Back substitution is then performed to express variables in terms of other variables and coefficients such that the new system is simpler.We store in our data all steps i → i + 1. This gives us a set of pairs (step i , step i+1 ) with step i the state of the system at step i of Gaussian elimination. From each such pair, we construct two kinds of textual data:• One step Gaussian elimination The question contains step i and the answer contains step i+1 .• One "clean" step Gaussian elimination Because our variables and coefficients in the system are essentially m-tokens, working performing Gaussian elimination can yield very large equations after a few steps. Thus, later steps will by construction always contain coefficients that are complex mathematical equations. In order to remediate to this issue, we replace the coefficients appearing in front of the variables in step i by simpler coefficients, namely m-tokens. Concretely, we will replace large coefficients of the form dog-cat * 2 tree * house by a single m-token. We call step clean i the simplification of step i . The question contains step clean i and the answer contains the corresponding step clean i+1 .Systems are built according to this format: ; [; equation 1 ; ; equation 2 ; . . . ; equation n ; ];. The idea behind this convention is to help the model spot equations using the symbol ;. Below we provide an example of Gaussian elimination:• Question. We run our experiments on 4 Nvidia A10 GPUs with 24GB of video memory on an AWS g5.12xl instance, thus fitting one replica of the model on each GPU.We generate 100 examples of quadratic polynomial problems by systematically varying the surface type (e.g., swimming pools, fields), the names of the owners (e.g., Jim, Alice), the number of each surface (square, rectangle, triangle, parallelogram) as well as the known length (for rectangle, triangle, parallelogram). We ensure that the discriminant of all the polynomials are perfect squares. This constraint avoids evaluating the model's ability to compute square roots.We provide an example of prompt and response for our problems (Figure 6 ). Prompt: Jim has a total of 2 swimming pools. The first swimming pool is a square, with an unknown side length x. The second is a rectangle with one side measuring 1 meters and the other being the unknown side length x. The total area covered by these swimming pools is 2 square meters. What is the unknown side length x? Response: The total area is the sum of the area of the square and the rectangle. The area of the square is x 2 , the area of the rectangle is x, and the total area of all fields is 2 square meters. Therefore, the equation to solve is: x 2 + x = 2. To solve the quadratic equation, we first calculate the discriminant ∆: ∆ = b 2 -4ac = 1 2 -4(1)(-2) = 9. Since ∆ > 0, the equation has two distinct real roots, calculated using the quadratic formula: x = -b± √ ∆ 2a. Substituting the values, we get: root 1: -1+ √ 9 2 = 1, root 2:-1-√ 9 2 = -2. Only one root is positive, so the solution is x = 1.In our experiments, the baseline model demonstrated suboptimal performance. To address this, we introduced a 3-shot prompting that explicitly detailed the steps for calculating the roots, leading to a marginal improvement in performance, though it remained relatively low (≈ 10%). While the baseline successfully formulates the quadratic polynomial equation, it fails to consistently apply the correct approach for solving it. Instead of computing the discriminant, the baseline frequently attempts an erroneous factorization, which reduces its overall effectiveness.We fine-tune the baseline model in order to improve its ability to find the roots of the polynomial by computing the discriminant. In the training data, we ensure that the discriminant of the polynomials are always perfect squares, preventing the model from learning how to compute square roots.Let's consider the quadratic polynomial expression ω 1 × ω 2 4 + ω 2 × ω 4 + ω 3 . Initially, the training data included all possible integer values for ω 1 and ω 2 between -10 and 10, and for ω 3 between -100 and 100. However, this range proved too broad, making it difficult for the model to learn effectively. As a result, the fine-tuned model frequently miscalculated the discriminant. To address this issue, we narrowed the parameter distribution, restricting ω 1 and ω 2 to positive integers, and ω 3 to negative values, better aligning the training data with our data at test time.We fine-tuned the model on 100,000 examples with equations akin to those necessary to solve resistor circuits. 50,000 examples correspond to resistors in series, and 50,000 examples correspond to resistors in parallel. We provide an example of prompt and response for a resistor circuit in parallel (Figure 7 ). We also add 25,000 examples from the Orca dataset Mukherjee et al. (2023) as a regularizer during training.For this particular problem we only consider symbolic solutions, thus we do not introduce numerical values for the variables. We evaluate the model's response by extracting the symbolic solution and comparing it to the correct one. Response: Using properties of resistors in series and parallel, and Ohm's law, we can determine that the equation governing the circuit is U = I/(1/R2 + 1/R1). Isolating the unknown, we get the symbolic solution R1 = 1/(I/U -1/R2).The error modes are similar for the baseline and fine-tuned models. The model is always able to find the correct equation, but the fine-tuned model makes less mistakes. These mistakes are mainly due to sign errors or failing to inverse a fraction (i.e. if 1/x = A then x = 1/A).For the fruit baskets examples we trained a model mainly on two mathematical rules: First-order single equation (150,000 examples), and Simplify expression (150,000 examples).We selected the hyperparameters for these rules (such as the number of terms and variables) to ensure that the generated data aligns with the format of equations typically found in the fruit baskets problem. We refer to section B.2 for the full details about these rules. Additionally, we incorporate 150,000 examples from the Orca dataset Mukherjee et al. (2023) as a regularizer during training. The Fruit Baskets problem requires setting up an equation, simplifying it to its canonical form, and solving for a specific variable. By fine-tuning the model on these rules, it can generalize and effectively simplify and solve the equation in Fruit Baskets problems."Numerical" Fruit Baskets. We also considered assigning numerical values to the variables in that problem, requiring the model to perform calculations to find a numerical solution instead of a symbolic one. We illustrate this in figure 9 . We found that the models' performances were similar in the symbolic and numerical cases. This is showcased in table 3. Prompt: Alice and Bob went to the grocery store and bought the following items:-bananas: Alice bought qA 1 , and Bob bought qB 1 . The price of a single one is p1.-blueberries: Alice bought qA 2 where qA 2 = 2 × p1, and Bob bought qB 2 where qB 2 = 8 × qA 1 . The price of a single one is p2 where p2 = 5 × qB 1 . Both ended up paying the same total price. Find the price of bananas in terms of qA 1 and qB 1 .Response: Since both Alice and Bob ended up paying the same total price for the fruits they bought, we can set up the following equation:qA 1 × p1 + qA 2 × p2 = qB 1 × p1 + qB 2 × p2We substitute the known relationships:qA 1 × p1 + 2 × p1 × 5 × qB 1 = qB 1 × p1 + 8 × qA 1 × 5 × qB 1Let x = p1, y = qA 1 , and z = qB 1 . The equation becomes:y × x + 2 × x × 5 × z = z × x + 8 × y × 5 × z Simplifying: y × x + 10 × x × z = z × x + 40 × y × zSolving for x, we get:x = 40 × y × z y + 9 × z Since x = p1, y = qA 1 , and z = qB 1 , we have: Prompt: Alice and Bob went to the grocery store and bought the following items:p1 = 40 × qA 1 × qB 1 qA 1 + 9 × qB 1-bananas: Alice bought qA 1 , and Bob bought qB 1 . The price of a single one is p1.-blueberries: Alice bought qA 2 where qA 2 = 2 × p1, and Bob bought qB 2 where qB 2 = 8 × qA 1 . The price of a single one is p2 where p2 = 5 × qB 1 . Both ended up paying the same total price. Find the price of bananas, given that qA 1 = 2 and qB 1 = 1.Response: Since both Alice and Bob ended up paying the same total price for the fruits they bought, we can set up the following equation:qA 1 × p1 + qA 2 × p2 = qB 1 × p1 + qB 2 × p2We substitute the known relationships:qA 1 × p1 + 2 × p1 × 5 × qB 1 = qB 1 × p1 + 8 × qA 1 × 5 × qB 1Given that qA 1 = 2 and qB 1 = 1, the equation becomes:1 × p1 + 2 × p1 × 5 × 2 = 2 × p1 + 8 × 1 × 5 × 2 Simplifying: p1 + 20 × p1 = 2 × p1 + 80Solving for p1, we get:p1 = 80 19C.4 Benchmark Performance The performance on these benchmarks remains relatively stable after fine-tuning, demonstrating that the newly acquired skills did not disrupt the model's prior knowledge.We fine-tune Llama-2 7B Chat Touvron et al. ( 2023) on all the data described in section B.2 (and that data only) and evaluate the model's ability to perform the following rules: distributivity, commutativity, division, exponentiation, variable evaluation, remarkable identities, single equation and two equations manipulation. The research questions that we want to answer are:• Can the model learn the rules that it has been trained on?• Can the model generalize the rules that it has been trained on?• Can the model use the rules when prompted differently than during training?• Can the model use combinations of the individual rules that it has been trained on?Hyperparameters and compute used during experiments are detailed in section D.6. We will essentially consider two versions of our fine-tuned model: one trained on distributivity data only (section D.1), and one trained on all mathematical rules (sections D.2, D.3, D.5). In section D.4, we evaluate our trained models on general knowledge benchmarks in order to check that performance remains relatively stable.We generate training data of 2 million examples of the distributivity rule as described in section B.2, where each bracket contains at most 5 types ω i : the number of variables is a random integer from 1 to 5. Each type is either an integer (probability 10%) or a k-token (probability 90%), where k ∈ [1, 3].The variables appearing in the equations take value in subsets of increasing tokenizer vocabulary sizes. For this, we split the tokenizer vocabulary of 32,000 tokens into partitions of increasing sizes (1%, 10%, 50%, 75%, 95% and 100%). We evaluate the model both on the complement of its training vocabulary (i.e. on examples that it has never seen during training), as well as on its training vocabulary. For each example, we report a score of 1 if the model prediction matches exactly the expected answer (i.e., the two strings are equal), and 0 otherwise. We present experimental results in figures 4, 10.We find that training on larger vocabulary sizes improves the ability of the model to generalize distributivity to unseen variable names as well as to increasing the number of variables. The model's performance on the training vocabulary is stable accross different vocabulary sizes, and significantly higher than the performance on the complement vocabulary, even for the largest vocabulary size. In terms of generalization, the models reach decent performance on their training vocabulary (consider that distributing 7 times 7 terms leads to 49 terms). In particular, we observe that the model struggles with some particular tokens (e.g. chinese or cyrillic characters). Other than that, it makes mistakes on very large expressions, forgetting chunks of the expression towards the end, or confusing signs.In figure 11 we evaluate the model trained on the full vocabulary on a subset of the vocabulary restricted to latin alphabet (lowercase and uppercase). The model's performance increases in the generalization domain, suggesting some sensibility to certain tokens. System solving is a difficult task for most language models, as it requires the elaboration of a series of reasoning steps. Language models have a finite context window, which constrains the amount of information they can process. Furthermore, numerous variables must be maintained and manipulated over multiple steps. Language models struggle with long-range dependencies, where the relevant information from earlier steps needs to be accurately memorized and applied in later steps. Even state-of-the-art models have difficulty solving systems with more than two variables and equations.To overcome these problems, our idea is to break down system solving into elementary steps and teach our model to correctly transition from each one of the steps to the next (as opposed to training it on the full resolution). The training data contains examples where the system is already resolved (termination condition), and when it is the case, the model recognizes that there is nothing left to do and returns The system is already simplified.The detailed construction of the corresponding data from steps i → i + 1 of the Gaussian elimination is provided in section B.2. We considered systems from 1 to 5 variables. For each step transition i → i + 1, we report a score of 1 if the model predicted string matches exactly the ground truth, and 0 otherwise. We obtain an average score of 88.5 ±0.5 %. The latter was computed over 100 examples and 2 random seeds. We provide a detailed example of system resolution in figure 5 .We fine-tune our model on all the mathematical rules detailed in section B. To evaluate the models, we considered eight different configurations:• 4 train configurations: We use the same hyperparameters as the training dataset: 3 to 5 variables per instance. We consider two vocabulary settings:-Full vocabulary: The full Llama-2 vocabulary. Each term is a concatenation of 1 to 3 Llama-2 tokens.* With integers: 10% of the variables in an expression are integers, while the rest are a concatenation of Llama-2 tokens. * Without integers: All the variables are a concatenation of Llama-2 tokens.-Restricted vocabulary: The vocabulary consists of Latin and Greek letters, one token per variable, and no integers allowed.* With digits: The vocabulary consists of Latin and Greek letters and a concatenation of these letters with numbers from 0 to 9 (e.g., α 0 ). * Without digits: The vocabulary consists of only Latin and Greek letters without any numbers.• 4 test configurations: We increase the number of variables to 6 to 7 variables per instance. We consider two vocabulary settings:-Full vocabulary: The full Llama-2 vocabulary. Each term is a concatenation of 3 to 5 Llama-2 tokens.* With integers: 10% of the variables in an expression are integers, while the rest are a concatenation of Llama-2 tokens. * Without integers: All the variables are a concatenation of Llama-2 tokens.-Restricted vocabulary: Same as the training configurationThe goal of excluding integers in some configurations is to ensure robust evaluation of symbolic reasoning, avoiding cases where models might output correct numerical results without demonstrating an understanding of the underlying properties. For example, in evaluating the division property (1 + 5)/3 = 1/3 + 5/3, we expect models to show understanding of this property rather than simply computing numerical expressions. On the other hand, the restricted vocabulary relying on the Latin and Greek alphabetical letters ensures a fair comparison to baseline models likely unfamiliar with unusual tokens in Llama 2's tokenizer. The baseline models considered are the instruct versions of Llemma Azerbayeva et al. (2024), Llama-2, Llama-3, Mistral, and WizardLM Luo et al. (2023) . Each model generated responses with a max_new_tokens set to 512. This token limit was determined to be sufficient for generating correct answers based on tests with multiple lengths (up to 1500 tokens), 81.5 ± 0.5 89.0 ± 2.0 100.0 ± 0.0 100.0 ± 0.0 92.0 ± 2.0 100.0 ± 0.0 100.0 ± 0.0 96.5 ± 1.5Llama-3 8B Instruct 15.0 ± 4.0 29.5 ± 3.5 16.5 ± 2.5 60.5 ± 1.5 79.0 ± 4.0 35.5 ± 2.5 0.0 ± 0.0 3.0 ± 1.0 WizardMath 7B 4.0 ± 0.0 27.0 ± 4.0 19.5 ± 5.5 58.5 ± 2.5 52.5 ± 2.5 46.0 ± 3.0 0.5 ± 0.5 2.5 ± 0.5 Mistral 7B Instruct 2.5 ± 1.5 18.5 ± 0.5 22.0 ± 0.0 42.5 ± 2.5 52.0 ± 4.0 15.5 ± 4.5 1.0 ± 1.0 1.5 ± 0.5 Llama-2 7B chat 0.0 ± 0.0 4.5 ± 0.5 5.5 ± 0.5 37.5 ± 2.5 15.5 ± 1.5 12.0 ± 6.0 0.0 ± 0.0 1.0 ± 1.0 llemma 7B 0.0 ± 0.0 1.0 ± 1.0 1.5 ± 1.5 36.5 ± 2.0 3.0 ± 1.0 3.0 ± 0.0 0.0 ± 0.0 5.0 ± 1.0 96.0 ± 2.0 99.5 ± 0.5 100.0 ± 0.0 100.0 ± 0.0 95.5 ± 1.5 100.0 ± 0.0 98.5 ± 1.5 97.5 ± 0.5 75.0 ± 2.0 94.5 ± 0.5 100.0 ± 0.0 100.0 ± 0.0 88.0 ± 1.0 100.0 ± 0.0 97.0 ± 1.0 95.5 ± 1.5Llama-3 8B Instruct 6.5 ± 0.5 25.5 ± 4.5 21.5 ± 0.5 79.5 ± 2.5 27.5 ± 0.5 45.0 ± 1.0 2.0 ± 1.0 7.5 ± 1.5 WizardMath 7B4.0 ± 1.0 28.0 ± 2.0 24.5 ± 0.5 85.0 ± 3.0 49.0 ± 1.0 72.0 ± 0.0 1.0 ± 0.0 8.0 ± 1.0 Mistral 7B Instruct 7.0 ± 2.0 19.5 ± 1.5 24.5 ± 2.5 71.0 ± 1.0 42.0 ± 1.0 41.0 ± 7.0 1.5 ± 0.5 4.5 ± 0.5 Llama-2 7B chat 0.0 ± 0.0 7.0 ± 0.0 7.0 ± 1.0 45.0 ± 4.0 22.0 ± 0.0 1.0 ± 1.0 0.0 ± 0.0 20.0 ± 2.0 llemma 7B 0.0 ± 0.0 2.0 ± 1.0 1.5 ± 1.5 44.0 ± 0.0 4.0 ± 2.0 7.5 ± 0.5 0.0 ± 0.0 7.5 ± 0.5 98.0 ± 1.0 99.0 ± 1.0 99.0 ± 0.0 99.5 ± 0.5 99.0 ± 1.0 98.0 ± 2.0 99.0 ± 1.0 99.5 ± 0.5Llama-3 8B Instruct 0.5 ± 0.5 9.5 ± 0.5 2.0 ± 0.0 49.0 ± 0.0 35.0 ± 6.0 21.5 ± 4.5 0.0 ± 0.0 0.0 ± 0.0 WizardMath 7B 6.5 ± 0.5 9.5 ± 0.5 7.0 ± 2.0 65.0 ± 0.5 22.5 ± 4.5 32.5 ± 0.5 0.0 ± 0.0 1.0 ± 0.0 Mistral 7B Instruct 3.0 ± 1.0 8.5 ± 2.5 5.0 ± 2.0 51.0 ± 4.0 5.5 ± 0.5 8.5 ± 3.5 0.0 ± 0.0 0.0 ± 0.0 Llama-2 7B chat 0.0 ± 0.0 4.5 ± 0.5 1.5 ± 1.5 21.0 ± 1.0 0.5 ± 0.5 5.5 ± 0.5 0.0 ± 0.0 0.0 ± 0.0 llemma 7B 0.0 ± 0.0 0.5 ± 0.5 0.0 ± 0.0 22.5 ± 1.5 0.0 ± 0.0 0.5 ± 0.5 0.0 ± 0.0 0.0 ± 0.0 15.0 ± 5.0 96.0 ± 1.0 99.0 ± 1.0 100.0 ± 0.0 96.0 ± 1.0 99.0 ± 1.0 94.5 ± 1.5 65.5 ± 4.5Llama-3 8B Instruct 0.0 ± 0.0 0.5 ± 0.5 0.5 ± 0.5 27.5 ± 1.5 5.0 ± 0.0 16.5 ± 0.5 0.0 ± 0.0 16.5 ± 0.5 WizardMath 7B 0.0 ± 0.0 1.0 ± 1.0 3.0 ± 2.0 56.0 ± 1.0 5.5 ± 3.5 19.5 ± 6.5 0.0 ± 0.0 23.5 ± 2.5 Mistral 7B Instruct 0.0 ± 0.0 0.0 ± 0.0 1.0 ± 0.0 42.5 ± 3.5 0.0 ± 0.0 3.0 ± 2.0 0.0 ± 0.0 15.5 ± 2.5 Llama-2 7B chat 0.0 ± 0.0 0.0 ± 0.0 0.5 ± 0.5 17.0 ± 2.0 0.0 ± 0.0 2.5 ± 1.5 0.0 ± 0.0 35.5 ± 2.5 llemma 7B 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 17.0 ± 2.0 0.0 ± 0.0 0.0 ± 0.0 0.0 ± 0.0 19.5 ± 2.5Table 13 : Evaluation results on exact predictions. For each example, we report a score of 1 if the model prediction matches exactly the expected answer (i.e., the two strings are equal), and 0 otherwise. Catastrophic forgetting is a well-known issue when fine-tuning large language models, where the model's performance on the pre-training task decreases after fine-tuning on a new task. We evaluate our models on 3 established benchmarks before and after fine-tuning. The results are presented in table 15 . While there is a small performance drop after fine-tuning, the model still performs well on the general knowledge tasks.Table 15 : Evaluation on general knowledge tasks. We evaluate on the MMLU Hendrycks et al. (2020) , ARC Clark et al. (2018) and HellaSwag Zellers et al. (2019) datasets for general knowledge and language understanding. The evaluation is performed under the framework of Gao et al. (2023) . D.5 Some Word ProblemsModelHere we discuss cases where our trained model successfully applies the mathematical rules seen during training in new situations. We see that our model is able to extract "abstract mathematical skills" and apply them on practical examples, which is striking.We provide in figures 12 and 13 qualitative examples for the distributivity rule. The main takeway is that despite being trained on the distributivity rule only (with a single prompt template), the model is able to infer the distributivity property on unseen prompts and apply it correctly, whereas the pre-trained model fails to do so. What is more striking is that these unseen prompts are word problems, as opposed to "mathematical" ones seen during training. Figure 12 : The fine-tuned model is able to extract a skill from its training on formal mathematical rules (distributivity) and apply it to a practical example.Instruction: Bob builds toy cars made of different pieces. Each toy is made of 4 wheels, 1 body and 2*z car seats (depending on the models). Bob receives a command for N cars. Give an expression representing how many parts he will need to assemble.Fine-tuned answer: The expression representing how many parts Bob will need to assemble is: N*4 wheels+N*1 body+N*2*z car seats.Pre-trained answer: To determine the number of parts needed to assemble N cars, we need to calculate the total number of parts required for each car and then multiply it by the number of cars requested. The total number of parts required for each car can be calculated as follows: 4 wheels x 1 body x 2 z car seats = 8 parts per car. Therefore, the total number of parts needed to assemble N cars can be expressed as: N x 8 parts per car = 8N. So, Bob will need 8N parts to assemble N cars. Instruction: Alice has to buy new uniforms for her children. One shirt costs $20, one pair of pants costs $30 and one pair of shoes costs $x. Let N be the number of Alice's children. Compute the total cost of these uniforms.Pre-trained answer: To compute the total cost of the uniforms, we need to know the value of x, which represents the cost of one pair of shoes. Let's assume that Alice has n children, so the total cost of the uniforms can be calculated as follows:Cost Instruction: Jimmy owns N houses. For each house he has a key for the front door and one for the garage door. Compute the total number of keys he has.Pre-trained answer: To compute the total number of keys Jimmy has, we need to multiply the number of houses he owns by the number of keys he has for each house. Since Jimmy owns N houses, he has a total of N keys for the front doors and N keys for the garage doors. Therefore, the total number of keys Jimmy has is: N × N = N 2 . So, Jimmy has N 2 keys in total.Fine-tuned answer: The total number of keys Jimmy has is 2 * N . Instruction: Expand (3+x)*(y+5). Then replace x by z in the resulting expression.Pre-trained answer: Sure! To expand the expression (3+x)*(y+5), we need to follow the order of operations (PEMDAS): 1. Parentheses: inside the parentheses we have (3+x) and (y+5). 2. Exponents (none in this case). 3. Multiplication: multiply (3+x) and (y+5). So the result is: (3+x)*(y+5) = 3x + 5y + x + 5. Now, if we replace x by z, we get: (3z + 5y + z + 5). Fine-tuned (full) answer: By the distributivity property: 3*y+3*5+z*y+z*5. Instruction: Expand the expression (a^5+cat)*(a^2). Fine-tuned (full) answer: By the distributivity property: a^7+cat*a^2. Fine-tuned (distributivity) answer: By the distributivity property: a^5*a^2+cat*a^2.Pre-trained answer: To expand the expression '(a^5 + cat)*(a^2)', we need to follow the order of operations (PEMDAS): 1. Parentheses: inside the parentheses we have 'a^5 + cat' 2. Exponents: 'a^5' = 'a * a * a * a * a * a' = 'a^6' 3. Multiplication: 'a^6 * a^2' = 'a * a * a * a * a * a * a' = 'a^7' So, the expanded expression is 'a^7'.To evaluate and compare the performance of our fine-tuned model against baseline models, we developed a streamlined pipeline. The primary objective of this pipeline is to assess the correctness of answers generated by models in response to mathematical prompts. Our approach heavily relies on SymPy for handling symbolic mathematics. The pipeline is summarized in figure 17 . First we provide an overview of the main elements.Figure 17 : Pipeline for evaluation Extract Relevant Mathematical Expressions. Models such as LLaMA, Mistral, WizardLM, and Llemma do not produce outputs in a standardized format. Therefore, isolating mathematical formulas from their often noisy outputs is generally challenging. The ambiguity and variability of notation and conventions in mathematical expressions, and the handling of nested parenthesis make it impossible to write comprehensive regular expression patterns. Therefore, we developed a custom algorithm (Figure 18 ) that parses strings and identifies expressions involving symbols like +, -, /, * , =, ÷, ×, ∧. These expressions are then converted to be compatible with SymPy.Adapting to SymPy. In our evaluations, we rely on SymPy for symbolic mathematics. However, the extensive vocabulary of Llama-2 includes some unusual strings that SymPy cannot process. We address this issue by mapping problematic tokens to non-problematic ones.Evaluation metrics. To evaluate an output's correctness, we create custom metrics for each mathematical rule. We perform a common check common across all rules using SymPy, namely that the model's output and the ground truth answer are equivalent.At the level of the generated dataset, all useful information for later evaluation is saved. When generating a test dataset, the output should be a dictionary for each instance. The dictionaries typically have the following structure:• dict["prompt"]: The prompt of the instance, used to prompt the different models.Example: Instance = " * start Expand this expression: {original_expression}. * end By the distributivity property: {distributive_expression}." Prompt = " * start Expand this expression: {original_expression}. * end "• dict["original_expression"]: The original expression of the instance.• dict["answer"]: The answer (distributed, commuted, solved equation, etc.) of the instance. Both the original expression and the answer are important for evaluating the correctness of a given response.• • dict["prompt_type"]: The type of the instance (distributivity, commutativity, single equation, etc.)The baseline models considered are the instruct versions of Llemma, Llama-2, Llama-3, Mistral, and WizardLM. Prompting these models involves several key steps: Models are loaded using the AutoModelForCausalLM and AutoTokenizer classes from the Hugging Face transformers library.Prompts are customized based on the specific requirements of each model. For instance, the prompt format for Llama-2 differs from that of WizardLM, with specific patterns adjusted using regular expressions provided on the Hugging Face page. For example, the prompt for WizardMath 7B is:"Below is an instruction that describes a task. Write a response that appropriately completes the request.###Instruction: cleaned_prompt ### Response:"Prompts are processed in batches to optimize performance and resource utilization. The generated responses are then cleaned to remove instruction tokens and other artifacts based on the specific model's requirements, ensuring a consistent format for evaluation.Models such as LLaMA, Mistral, WizardLM, and Llemma do not produce outputs in a standardized format. Therefore, isolating mathematical formulas from their generally noisy outputs is very challenging. The ambiguity and variability of notation and conventions in mathematical expressions, and the handling of nested parenthesis make it impossible to write comprehensive regex patterns. Therefore, we developed a custom algorithm that parses strings and identifies expressions involving symbols like +, -, /, * , =, ÷, ×, ∧. These expressions are then converted to be compatible with SymPy.Our algorithm works as follows: we start by splitting a string on whitespace and examining each word and its neighbors. If the previous word ends with a mathematical sign, or if the current word begins with one, then the current word is part of a mathematical expression. Otherwise, if the current word ends with a mathematical sign, or if the next word begins with one, then the current word starts a new expression. Otherwise, we check if the current word contains any mathematical symbols. If it does and SymPy can process it, it is considered a standalone expression.There are corner cases, such as extracting is-2+3 from The answer is -2 + 3. Here, we use domain knowledge to remove outlier terms like is. Additional processing steps handle issues like treating punctuations, and fixing the multiplication in expressions like ab to become a*b using dataset knowledge.The decision tree 18 gives an overview on the algorithm's reasoning. In our evaluations, we rely on SymPy for symbolic mathematics. However, the extensive vocabulary of Llama-2 includes many unusual strings that SymPy cannot process. To address this issue, we created a mapping between tokens that SymPy cannot process and tokens that SymPy can handle. This mapping ensures a clean bijection with no overlapping, preserving the structure of the original expression while making it readable by SymPy, thus enabling effective evaluation of models using the full vocabulary of Llama-2.We evaluate the model's mathematical outputs according to the rule that we're testing. The initial evaluation involves checking if the model's output and the ground truth answer are equivalent.To achieve this, we implemented a function using SymPy to determine the equivalence of two mathematical expressions. The process is as follows:1. Symbolic Simplification:• Convert the input expressions into SymPy • Compare the simplified expressions. If they are identical, return True.We gave Llama-3 the following prompt: Apply the distributive property of multiplication over addition to the expression l * (U + s + Z).The model generated the following response to the task:Using the distributive property of multiplication we have: l * (U +s+Z) = l * (U +s)+l * (Z)The model applied the distributive property partially but did not fully expand the terms. The correct answer (ground truth) is:l * (U + s + Z) = l * U + l * s + l * ZFirst, we extract the model's mathematical output from its response: l * (U + s) + l * (Z).Both the model's output and the ground truth are converted into symbolic expressions using the SymPy library. Finally, we verify if the model correctly applied the distributive property. SymPy will make the necessary simplifications, and is invariant to order, ensuring that the two expressions will be equivalent only if both are fully expanded. We compare the sympified correct expression with the sympified model's output. The expanded form (l * U + l * s + l * Z) differs from the model's output (l * (U + s) + l * (Z)), indicating that the distributive property was not fully applied. Therefore we label it as False.Factorization: We count the number of terms in the original expression and we check that this number decreases in the output of the model while maintaining equivalence with the original expression.We consider the full vocabulary minus a set of tokens such as spaces, arithmetic symbols {+, -, * , /, =}, etc.that is, in the form ω1 -ω2 = 0. 



Reviews: review: title: title: Exploring LLMs' Ability to Learn and Apply Mathematical Rules.
main_review: main_review: The authors focus on whether LLMs can learn and generalize specific mathematical rules, applying them to unseen scenarios like solving word problems. They fine-tune the LLaMA model on a synthetic dataset containing various mathematical rules and demonstrate that the model can generalize these rules to real-world problems, though its generalization is limited by the form of equations and vocabulary size in the training data.

Pros:

- Creating synthetic data that includes specific mathematical rules (e.g., manipulating equations, distributive property, exponentiation, etc.) may offer a valuable resource for training large language models, enabling them to learn and apply these rules.
- Experimental results show that the model performs exceptionally well on various mathematical tasks, proving the effectiveness of this approach.

Cons:

- I think the authors may give some future exploration into scaling up such synthesis with less effort.
title: title: A good paper highlighting the potential and the challenges of using LLMs for mathematical reasoning tasks.
main_review: main_review: This paper examines the ability of large language models (LLMs) to learn and apply mathematical rules, specifically focusing on rules like distributivity and simplifying equations within the context of word problems. A rigorous methodology was developed for building synthetic data that mimics the style of a mathematics textbook without word problems. The models are then fine-tuned on this synthetic data. The study highlights that while models can learn and generalize these mathematical rules to some extent, they face challenges when applied to word problems that require converting natural language into mathematical expressions and solving them.

Strengths:

1.	The creation of synthetic data that reflects mathematical textbook problems provides a controlled environment to test the models' ability to learn specific mathematical rules effectively.

2.	The study addresses a practical challenge in AI education technology—improving LLMs' ability to handle word problems, which are a fundamental component of mathematical learning and assessment.

3.	The paper provides a thorough analysis of the models' performance on synthetic and real-world style data, offering insights into their capabilities and limitations in mathematical reasoning.

Weaknesses:

1.	The models struggle with the application of learned mathematical rules to new problems, particularly in translating natural language problems into mathematical equations—a critical step that is not sufficiently addressed.

2.	While the use of synthetic data is beneficial for controlled experiments, it may not capture the complexity and variety of natural language and numerical relationships found in real-world data, potentially limiting the models' practical applicability.

3.	There is a risk that fine-tuning on highly specific tasks might lead models to overfit to the synthetic training data, reducing their effectiveness on more generalized tasks or unseen data types.
title: title: Rigorous paper with interesting findings.
main_review: main_review: In the present paper authors explore the ability of LLMs to learn, generalize, and apply abstract mathematical rules, such as distributivity or equation simplification. Through various experiments, the authors demonstrate that LLMs can generalize mathematical rules to some extent and reuse them in solving word problems. They also show that models perform better when the structure of training data closely matches the test data. The paper contains detail analysis using various LLMs.
title: title: Review of paper 16: Learning Mathematical Rules with Large Language Models
main_review: main_review: **Summary of contributions:** 
 
The paper focuses on large language models’ ability to acquire knowledge on mathematical rules through fine tuning to solve equations of mathematical problems represented through text.

**Relevance and Clarity:**

Highly relevant to the guiding theme of the proposed workshop and explicitly presents results in a manner directly applicable to the intersection of NLP and mathematical reasoning.

**Originality** - The authors provide an innovative methodology for creating synthetic data with specific mathematical rules. Specially the following strategy adopted by the authors appear to be appealing:

*Sympy vs. Custom Approach:* Instead of using a symbolic mathematics library like Sympy which does automatic simplifications systematically, the authors stick with their custom approach to have that fine-grained control while implementing the simplification rules in the synthetic data. 

**Strengths**

1. *Generalization:* 

The authors’ endeavor to teach the model the fundamental nature of the mathematical operators i.e. to apply mathematical rules such as distributivity even when the variables appearing in the equations are arbitrary strings. 

2. *Translation Function:* 

The authors also talk about implementing a translation function which converts their custom data structure into Sympy’s data structure (and vice versa). This should enable the model to handle more advanced simplification rules externally when needed.

**Weakness** 

The authors reported that their fine-tuned Llama-2 7B Chat exhibited decreased performances on the pre-training task across all the three established benchmarks which might pose as an obstacle for adopting this fine-tuning methodology. A better clarity on this issue with possible mitigation to ward off this kind of forgetting in the model after fine-tuning could have been presented within the main body of the paper. This also opens a potential future direction for the study to explore fine-tuning methodologies which mitigate catastrophic forgetting in large language models while learning mathematical rules.

**Significance** 

This paper does a good job in adopting a quality methodology to produce synthetic data for teaching mathematical rules to large language models and has the potential to contribute towards designing benchmarks to evaluate mathematical reasoning abilities of large language models.