FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning

## Abstract

Abstract The application of formulas is a fundamental ability of humans when addressing numerical reasoning problems. However, existing numerical reasoning datasets seldom indicate explicitly the formulas employed during the reasoning steps. To bridge this gap, we construct a dataset for formula-based numerical reasoning called FormulaReasoning, which consists of 5,420 reasoning-based questions.
 We employ it to conduct evaluations of LLMs with size ranging from 7B to over 100B parameters utilizing zero-shot and few-shot chain-of-thought methods, and we further explore using retrieval-augmented LLMs provided with an external formula database associated with our dataset. We also experiment with supervised methods where we divide the reasoning process into formula generation, parameter extraction, and numerical calculation, and perform data augmentation. Our empirical findings underscore the significant potential for improvement in existing models when applied to our challenging, formula-driven FormulaReasoning.

## 1 Introduction

Numerical reasoning constitutes one of the significant forms within natural language reasoning (Frieder et al., ). The study of numerical reasoning has seen substantial progress in recent years, largely driven by the development of LLMs (OpenAI, ; Touvron et al., ; Li et al., ) and specialized datasets (Wang et al., ; Dua et al., ; Amini et al., ; Cobbe et al., ). Current datasets for numerical reasoning typically include simple, commonsense numerical questions that do not reflect the complexity of real-world problems.
These datasets have not fully addressed the interpretability issue in numerical reasoning, as they often rely on implicit commonsense knowledge without explicit guidance knowledge during the reasoning process.
This issue becomes particularly evident when LLMs meet hallucination (Frieder et al., ; Bang et al., ).
Consequently, one might naturally ask “What knowledge could be used to guide numerical reasoning process?”. Formulas exactly represent such knowledge that has been largely overlooked in previous research but is frequently utilized in real-life applications.

Figure: Figure 1: An example taken from FormulaReasoning. Numerical values (including units) given in the question and obtained from intermediate steps are highlighted in red and purple, respectively. Formulas and their elements are in blue.
Refer to caption: x1.png

Take a question from the GSM8K (Cobbe et al., ) as an example: “A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?”. This example only requires the use of implicit commonsense mathematical knowledge to solve without domain-specific formula. However, in our FormulaReasoning dataset, we require domain-specific formulas to guide the numerical reasoning process, such as the formula used to calculate the heat absorption of an object.

Recently, Liu et al., constructed two formula-based datasets, Math23K-F and MAWPS-F. However, the formulas in these datasets primarily consist of commonsense formulas (such as total_amount = unit_amount $\times$ total_number), and only 33.5% and 38.4% of the questions in these datasets, respectively, require the use of formulas.

To fill this gap, we constructed a dataset for numerical reasoning that requires the use of formulas called FormulaReasoning.
We annotated formulas for each question in FormulaReasoning.
An example of FormulaReasoning is shown in Figure 1.(^1^11Please note that FormulaReasoning is originally in Chinese. For the convenience of understanding, we translated Chinese into English in all the examples presented in this paper.) The formula-based feature makes FormulaReasoning a more challenging dataset for developing systems that can tackle real-world numerical reasoning problems. Indeed, in fields such as mathematics and physics, formulas serve as an important vessel for representing domain knowledge. However, existing datasets scarcely consider explicit incorporation of formulas into numerical reasoning.

Table: Table 1: Statistics of Math23-F, MAWPS-F, GSM8K, MATH and our FormulaReasoning.

We collected questions requiring formula-based numerical reasoning from Chinese junior high school physics examinations. With the combined efforts of manual annotation and assistance from LLMs, we annotated each question with an explanation text, a final answer, and a set of relevant formulas (including formula structures, parameter names, symbols, numerical values, and units) and built a consolidated formula database. The formula database functions as an external knowledge base, which can be used to evaluate retrieval-based/augmented systems. In Table 1, we compare FormulaReasoning with two existing formula-based datasets and the well-known GSM8K and MATH (Hendrycks et al., ). In comparison to Math23K-F and MAWPS-F, FormulaReasoning contains a larger number of formulas (272), whereas the other two datasets contain 51 and 18 formulas. Additionally, all questions in FormulaReasoning require the use of formulas. The higher average number of reasoning steps (2.37 vs. 1.16/1.01) implies that FormulaReasoning is more challenging and better suited for evaluating existing models as a multi-step formula-based reasoning task.

We used FormulaReasoning to evaluate LLMs ranging from 7B to $>$100B parameters, as well as fine-tuned models such as Qwen-1.8B (Bai et al., ) and ChatGLM3-6B (Zeng et al., ) with a proposed Chain-of-Thought supervised fine-tuned method and a data augmentation method. We also trained an encoder for formula retrieval and experimented with retrieval-augmented generative models. Our empirical findings show that the best existing models only achieve an accuracy of around 84%, lagging behind an accuracy 92% of humans, indicating that there is still significant room for exploration in formula-based numerical reasoning.

Our contributions are summarized as follows:

- •
We construct a formula-based numerical reasoning dataset FormulaReasoning, with fine-grained annotations for each question. As a formular knowledge-guided numerical reasoning dataset, it can be applied to tasks involving trustworthy and verifiable reasoning.
- •
We conduct evaluations on LLMs of various sizes, supervised fine-tuned models, and retrieval-augmented generative models. The experimental results establish a strong baseline for future research and also indicate that the task remains unresolved.

The dataset is available on Zenodo(^2^22[https://zenodo.org/doi/10.5281/zenodo.11408109](https://zenodo.org/doi/10.5281/zenodo.11408109)) under the CC BY 4.0 License and our code is available on GitHub(^3^33[https://github.com/nju-websoft/FormulaReasoning](https://github.com/nju-websoft/FormulaReasoning)) under the Apache License 2.0.

## 2 Related Work

### 2.1 Numerical Reasoning Datasets

Numerical reasoning is one of the fundamental capabilities of natural language reasoning. The study of numerical reasoning in natural language has existed for several years. Numerous datasets, such as DROP (Dua et al., ), GSM8K (Cobbe et al., ), TSQA (Li et al., ) and MATH (Hendrycks et al., ), have introduced natural language numerical reasoning.
Another line of research focusing on numerical reasoning in natural language is math word problem (MWP). MWP tasks typically provide a short passage (i.e., a question) and require the generation of an arithmetic expression that can compute an answer. Representative datasets include MAWPS (Koncel-Kedziorski et al., ), Math23K (Wang et al., ), MathQA (Amini et al., ), etc.
Several works focus on specialized domains where some of the questions in their datasets require numerical reasoning. Examples include GeoSQA (Huang et al., ), which focuses on the geography domain, the STEM (Drori et al., ) dataset and the ScienceQA (Lu et al., ) which covers multiple disciplines in science and technology. The distinguishing feature of our FormulaReasoning is that the numerical reasoning questions within these datasets lack explicitly labeled formulas.

The recently introduced datasets (Liu et al., ) Math23K-F and MAWPS-F require formulas for only 33.5% and 38.4% of the questions, respectively, and the formulas within these datasets are all simple commonsense formulas (e.g., total_cost = unit_cost $\times$ total_number). By contrast, our FormulaReasoning dataset collects questions from junior high school physics examinations, with every question accompanied by formulas. In addition, we also annotated a formula database for FormulaReasoning that can serve as an external knowledge base, used to assess retrieval-augmented systems.

### 2.2 Numerical Reasoning Methods

The methods for solving numerical reasoning have evolved from statistical approaches (Hosseini et al., ; Kushman et al., ) to those based on rules and templates (Shi et al., ; Wang et al., ) and further to methods based on deep learning models (Gupta et al., ; Chen et al., ; Kim et al., ; Li et al., ). In the past two years, with the rapid development of LLMs, LLMs have demonstrated strong capabilities in resolving numerical reasoning questions. Consequently, several methods aimed at enhancing the reasoning abilities of LLMs have been proposed, including the notable Chain of Thoughts (CoTs) method (Wei et al., ), along with many subsequent variant approaches (Kojima et al., ; Wang et al., ; Zhou et al., ; Li et al., ).

We established representative existing methods as baselines for FormulaReasoning, including zero/few-shot CoTs prompting methods to LLMs ranging from 7B to over 100B parameters. We trained a specialized formula retriever for retrieving formulas and explored retrieval-enhanced numerical reasoning. We also divided the reasoning process into formula generation, parameter extraction, and numerical calculation, and used data augmentation to enhance fine-tuned models with fewer than 7B parameters.

## 3 Dataset Construction

We collected raw questions from Chinese junior high school physics examinations from 2015 to the present. We had a total of five postgraduate volunteer students, and they all hold a bachelor’s degree in science and engineering. We then annotated the reasoning steps and corresponding formulas for each question. This process involved a combination of manual annotation and the assistance of LLMs to improve the efficiency of annotation. Each question is associated with an explanation of the reasoning steps in natural language with a symbolic representation of the reasoning steps using formulas, including the values and units for all the parameters within the formulas. Finally, we compiled all the formulas and we merged those expressing the same meaning to create a formula database.
We describe this process to construct FormulaReasoning in detail below.

### 3.1 Preprocessing

We crawled 18,433 junior high school physics examination questions in China from 2015 to the present from public sources, including only those with free-text answers and excluding multiple-choice and true/false questions. Each raw question contains a question text and an explanation text that includes the reasoning steps. We eliminated questions requiring diagrams.

Subsequently, we filtered the questions by assessing the presence of numerical values within the explanation and confirming that the final answer was numerical. Utilizing a regular expression-based approach, we extracted the final numerical answer, including its unit, from the explanation. We found that for 487 questions, the regular expressions did not return results, so we manually annotated the positions of their answers in the text explanations.
Following the preprocessing phase, we compiled an initial dataset comprising 6,306 questions.

**Table 2: Original explanation and explanation with normalized formulas (highlighted in blue).**
| Original explanation. |
| --- |
| The change in water temperature is 60 - 20 = 40 °C. Therefore, the heat absorbed by the water is Q_{absorbed}=50 kg $\times$ 4.2 $\times 10^{3}$ J/(kg·°C) $\times$ 40 °C = 8.4 $\times 10^{6}$ J. Given that the total electrical energy consumed in the heating process is $1\times 10^{7}$ J, the thermal efficiency of the water heater can be calculated using the formula for the efficiency of a heat engine: $\eta$ = Q_{absorbed}}/W_{total}$\times 100$% = ($8.4\times 10^{6}$ J)/($1.0\times 10^{7}$ J)$\times 100$% = $84$%. Answer: If it is known that the total electrical energy consumed during the heating process is $1\times 10^{7}$, the thermal efficiency of the water heater is $84$%. |
| Explanation with normalized formulas. |
| 1. Calculating the temperature increase in water: [Degree of water temperature increase] = [Final temperature] - [Initial temperature] = 60 ℃ - 20 ℃ = 40 ℃. The degree of water temperature increase = 40 ℃. |
| 2. Calculating the heat absorbed by water: [Heat absorbed by water] = [Mass of water] $\times$ [Specific heat capacity of water] $\times$ [Degree of water temperature increase] = 50 kg $\times$ 4.2 $\times$ $10^{3}$ J/(kg·℃) $\times$ 40 ℃ = 8400000 J. The heat absorbed by water = 8400000 J. |
| 3. The thermal efficiency of the water heater can be obtained from: [Thermal efficiency of the water heater] = [Heat absorbed by water] / [Total electrical energy consumed] $\times$ 100% = 8400000 J / ($1\times 10^{7}$ J) * 100% = 84%. The thermal efficiency of the water heater = 84%. |
| Answer = 84% |

### 3.2 Formula Normalization

We found that the reasoning steps (i.e. the explanation) in the obtained raw dataset lacked a normalized format and were expressed quite casually. Some formulas mixed parameter names (e.g., “mass of water”) and symbols (e.g., “$m_{water}$”), while others simply provided calculations in numerical form without parameter names or symbols.
In order to ensure that all explanations adopted a normalized form of formulas, we normalized the formula annotations in the explanations. An example can be found in Table 2.
In this process, we need to identify the formulas used within the original explanations and to correct any formatting issues. Manually undertaking such tasks would require significant effort.
However, since the process is not open-ended, but rather structured and verifiable, we could automatically, e.g., *using a LLM*, extract formulas from the explanations, calculate each step, and compare the result with the given answer to ensure the accuracy of this normalization process.

Specifically, to enhance the efficiency of the annotation, we adopted a coarse-to-fine annotation approach with the help of a LLM(^4^44During dataset construction, we accessed Qwen-max via API ([https://help.aliyun.com/zh/dashscope/developer-reference/quick-start](https://help.aliyun.com/zh/dashscope/developer-reference/quick-start)). Qwen-max is a LLM with over 100B parameters and a strong capability in Chinese.).
We first prompted the LLM in a few-shot manner to generate accurate explanations of the reasoning process. Then, we used few-shot prompts to guide the LLM in correcting minor errors within the normalized explanations, including formatting errors in formula annotations and inaccuracies in the parameters used during computations. Both prompts can be found in Appendix A.1.1. Next, we will provide a detailed description of this process.

Initially, we introduced the question along with its original explanation and the corresponding answer to guide the LLM through few-shot prompting to revise the original explanation.
We observed that the ability of the LLM to revise explanations towards normalized explanations remained satisfactory.
To assess the correctness of the revised explanations, we extracted formulas from these explanations and then computed the answer using the numbat tool(^5^55[https://numbat.dev](https://numbat.dev). Numbat is designed for scientific computations with support for physical units.).
In addition to providing explanations, we also required the LLM to present the values, symbols, and units of each parameter in the formulas in the form of a table. An example is shown in Figure 1.

At this stage, we checked the correctness of the formula format in the explanations by automatic rules, including whether there were omissions in parameter names, parameter symbols, or corresponding units, and these issues were all correctable.
Therefore, if our program detected that the LLM had not successfully generated an accurate normalized explanation, we used few-shot prompting to identify and correct these specific errors. More details can be found in Appendix A.1.1.
We observed that the questions which remained incorrect despite multiple attempts by the LLM were of notably poor quality, including missing important reasoning steps, unclear question formulation, and so on. Some examples of these questions can be found in Appendix A.1.2. These questions were removed from our dataset. Following this step, our dataset contains a remaining total of 5,420 questions.

### 3.3 Formula Database Construction

Table: Table 3: Changes in the number of formulas after each merging step.

Our next step was to construct a unified formula database for the entire dataset. Given that parameters in the same formula can be expressed differently across various problem contexts, for instance, the two formulas “[weight of water] = [mass of water] * [gravitational acceleration]” and “[weight] = [mass] * [gravitational acceleration]” both calculate the weight of an object, we need to merge these formulas into a single representation.

We divided the construction process of the formula database into three steps: 1) Merge the formulas through symbolic rules. 2) Merge the formulas through semantic-based method. 3) Manual review and error correction.
In Table 3, we present the initial number of formulas and the remaining number of formulas after each step.

##### Symbolic rules based merging.

In this step, we merged formulas through symbolic rules. Specifically, this was achieved by comparing the structure of the formulas and the symbols.
Take the following as an example of judging whether two formulas have the same structure: the formulas “$f_{1}:~{}a_{1}=(b_{1}+c_{1})/d_{1}$”, “$f_{2}:a_{2}=(b_{2}+c_{2})/d_{2}$” and “$f_{3}:b_{1}=a_{1}*d_{1}-c_{1}$” have the same structure because $f_{2}$ can be derived from $f_{1}$ by renaming parameters, and $f_{3}$ can be obtained from $f_{1}$ by transformation.
Moreover, in physics, certain physical quantities are conventionally represented by specific symbols. For example, the mass of an object is often denoted by “$m$” and the density of an object is frequently represented by the symbol “$\rho$”. Subscripts are then used to distinguish which specific object a physical quantity refers to, such as “$\rho_{water}$” for the density of water.
For any two formulas, we first computed all the transformations of each formula to obtain a set of all its variants. Then, we compared the formula structures in the two sets to determine if two formulas were structurally equivalent.
If they shared the same structure, we then compared whether their symbols, with subscripts removed, were identical. If they were, we considered these two formulas to be mergeable. When merging, we retained the parameter with the shorter length from the two.
After merging based on symbolic rules, we reduced the number of formulas in the formula database from 12,906 to 1,163.

##### Semantic-based merging.

In the symbolic rules based merging process, the semantic information of the parameter names was neglected. This led us to perform merges grounded on the semantics of the parameter names.
For instance, two formulas that were not merged during the symbolic fusion stage, “[density] = [mass] / [volume]” and “[density of water ] = [mass of water] / [volume of water]”, can actually be merged. We would carry out the merging of these two formulas based on the semantic information of the parameter names (for example, ”density” and ”density of water” are semantically similar).
Specifically, for formulas with identical structures, we tokenized each pair of corresponding parameters to create two sets of words(^6^66We used jieba: [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba).).
When the two sets overlapped, the parameters were considered to have semantic connection, and the formulas became candidates for merging. Utilizing this approach, we identified a set of pairs of potentially mergeable formulas and then consulted the LLM for a thorough evaluation of each pair. The prompts can be found in Appendix A.1.3. After this step, the number of formulas in the formula database was reduced to 439.

##### Manual review and error correction.

Upon completing the aforementioned merging process, we manually inspected the correctness of the results, rectified instances where errors occurred during merging, and manually merged formulas that were overlooked by the LLM. In this process, there were two human volunteers cross-validating the results of manual review and annotation. Finally, we obtained a formula database consisting of 272 formulas.

## 4 Experiments Setup

In this section, we explore several methods for handling the questions within FormulaReasoning, including prompting LLMs using zero-shot and few-shot chain-of-thought (CoT, Wei et al., ; Kojima et al., ), and training a formula retriever to retrieve formulas to be incorporated into LLM prompts. Additionally, we employed two approaches to enhancing the reasoning abilities of fine-tuned models with fewer than 7B parameters. The first approach involved dividing the reasoning process into distinct steps: formula generation, parameter extraction, and numerical calculation. The second approach leveraged data augmentation to improve the models’ reasoning ability.

### 4.1 Dataset Split

We divided FormulaReasoning into into subsets for training, id (in-distribution) test, and ood (out-of-distribution) test, comprising 4,608, 421 and 391 questions, respectively. We required that all formulas in the id test must appear in the training set, whereas in the ood test, each question involves at least one formula that has not been seen in the training set. This division is designed to evaluate the generalizability of fine-tuned models on formulas that they have not previously encountered.

### 4.2 Evaluated Methods

#### 4.2.1 Human Performance

We recruited 108 students from a high school, with each student being assigned 7–8 questions. Each student was given 40 minutes to complete these questions. These questions were used as part of their in-class exercises, and at the end, each student received a gift. The final statistics were collected to evaluate human performance, which was consented by all the students.

#### 4.2.2 LLMs

Following Kojima et al., , we incorporated the phrase “Let’s think step by step” into the zero-shot prompt to guide LLMs in generating the reasoning steps. For the few-shot setting, we randomly sampled five questions from the training set to serve as examples for in-context learning. Each example includes the question text and the reasoning steps (i.e., the explanation). Examples of the prompts can be found in Appendix A.2.2.

We conducted experiments on GPT-4o, GPT-4-turbo, GPT-3.5-turbo, GLM-4-plus, GLM-4-flash (GLM et al., ), and Qwen-max. We also evaluated on Qwen2.5-7B/14B (Yang et al., ) and Llama3.1-8B (Meta, ).

#### 4.2.3 Formula Retriever

We trained a formula retriever on the training set. Specifically, we encoded each question using the Chinese-BERT-wwm-base (Devlin et al., ; Cui et al., ) model to obtain the CLS vector of the question. Each formula in the formula database was represented by a randomly initialized vector. During training, we calculated the cosine score between the question vector and the formula vector. The retriever was then trained with in-batch negatives and contrastive learning loss (Gao et al., ). Subsequently, for each question in the id test, we retrieved the top five formulas with the highest scores and included them in the prompt to observe the change in the performance of the LLM when provided with relevant formulas. More details can be found in Appendix A.2.3.

#### 4.2.4 Supervised Fine-tuned Models

We found that directly prompting models possessing fewer than 7B parameters failed to produce satisfactory outcomes (for example, ChatGLM3-6B attained merely 8.99 points in a zero-shot setting). Therefore, we conducted supervised fine-tuning of models with fewer than 7B parameters, yet discerned that, dissimilar to larger models (such as GLM-4-plus), smaller models did not exhibit proficient performance in numerical extraction and calculation. In order to augment the reasoning capabilities of smaller models, we explored two approaches for improvement. We conducted experiments on Qwen-1.8B (Bai et al., ) and ChatGLM3-6B (Zeng et al., ).

##### Chain-of-Thought Supervised Fine-Tuning (CoT-SFT)

We decomposed the reasoning process into several steps. First, we instructed the model to generate the formulas required to solve the question. Subsequently, the parameter names within the formulas were extracted, allowing the model to retrieve the corresponding values and units from the context. Next, the formulas and the associated parameter values were provided to a calculator to obtain the final result. This approach relieved the model from numerical calculation, allowing it to concentrate on the reasoning aspect.

##### Data Augmentation (DA)

We augmented the training dataset with the assistance of larger models. Firstly, we utilized a few-shot approach to prompt a LLM (Qwen-max) to generate new question-answer pairs. The correctness of the computation process generated by the LLM was meticulously verified using a calculator. Subsequently, the formulas generated by the model were extracted and normalized. More details could be found in Appendix A.2.1.

### 4.3 Metric

We utilized numbat to evaluate the predictions generated by the model against the gold-standard answers. A prediction is deemed correct if the relative error (prediction - gold) / gold is less than 1%. We employed accuracy, which is the proportion of questions answered correctly, as our metric.

### 4.4 Implementation Details

We accessed to GPT-4o (gpt-4o-2024-08-06 version), GPT-4-turbo (gpt-4-1106-preview version), GPT-3.5-turbo (gpt-3.5-turbo-1106 version)(^7^77https://platform.openai.com/docs), GLM-4-plus, GLM-4-flash(^8^88https://open.bigmodel.cn/), Qwen-max and Qwen2.5-7B/14B(^9^99https://help.aliyun.com/zh/dashscope/developer-reference/quick-start) through API calls with the default hyper-parameters. For Llama3.1, we conducted experiments on NVIDIA V100-32G GPUs. These LLMs generated using nucleus sampling with top_p=0.8. Models that require fine-tuning were experimented on NVIDIA V100 GPUs with Huggingface Transformers and Pytorch 2.0. For Qwen-1.8B, we used a learning rate of 1e-5 and a batch size of 32, and tested the model after training for 10 epochs. For ChatGLM3-6B, we fine-tuned with LoRA (Hu et al., ) with r=8, alpha=32 and learning rate of 5e-5, batch size of 1. The max input length and output length are both set to 512. We utilized nucleus sampling with top_p=0.8 for generation. In the case of CoT-SFT, which directly outputted formulas along with corresponding parameter values and units, if the generation output contained formatting errors, we allowed the small model to retry up to 5 times until a correctly formatted output was generated. Training Qwen-1.8B, ChatGLM3-6B models required 12 and 24 hours respectively.

## 5 Experiments Results

### 5.1 Human Performance

In FormulaReasoning, humans achieved impressive performance, with a score of 93.49 on the id test, 90.47 on the ood test, and an average score of 92.03.

### 5.2 Results of LLMs

Table: Table 4: Results of LLMs with zero-shot and few-shot prompting.

The evaluation results on LLMs are shown in Table 4. GLM-4-plus exhibited the best performance in both zero-shot and few-shot settings, surpassing the second-ranked GPT-4o by an average of 7.88 points in zero-shot setting and 9.36 in few-shot setting.
Among models with size not exceeding 20B, Qwen2.5-14B demonstrated commendable performance in both zero-shot and few-shot settings. The subpar performance of Llama3.1 might be due to its pre-training data being primarily in English.
After incorporating few-shot examples, GPT-4-turbo, GPT-3.5-turbo, GLM-4-plus and Qwen2.5 demonstrated performance improvements, ranging from 0.25 to 20.18. However, similar performance changes were not observed on other LLMs. Surprisingly, the open-source Qwen2.5-14B model outperformed the closed-source Qwen-max model(^10^1010We have not yet found clear information indicating whether the closed-source Qwen-max is also based on version 2.5.).

Human performance surpassed the performance of the flagship model GLM-4-plus with zero-shot setting and few-shot setting by margins of 9.27 and 7.79 points, respectively. Such results demonstrated that there remained a substantial gap between the current capabilities of state-of-the-art LLMs and human performance. This was even more pronounced when considering smaller-scale models. These findings underscored the challenging nature of FormulaReasoning as an unresolved dataset, and that there was significant room for improvement in LLMs as they struggled to match human levels of reasoning.

We also compared the chain of thought (CoT) and program of thought (PoT, Chen et al., ) methods, with the results presented in Appendix A.2.4. The results indicated that CoT and PoT demonstrated varying performances between different models and under different settings.

### 5.3 Results of LLMs with Formula Retriever

Table: Table 5: Results of LLMs with Formula Retriever on the id test.

The results of LLMs utilizing the formula retriever are shown in Table 5.
We found that the impact on performance varied among different LLMs when incorporating retrieved formulas into prompts. We observed a positive enhancement on Qwen2.5-7B, with score increments of 10.92 and 4.04 with zero-shot and few-shot, respectively, on the id test.
However, we found that the performance was essentially on par on the GLM-4-flash. Specifically, we found that the top 5 retrieved formulas often included irrelevant ones, as the number of formulas required varies for each problem. The presence of these extraneous formulas affected the model’s performance, indicating that there is considerable room for further research in retrieving from a formula database.

### 5.4 Results of Supervised Fine-tuned Models

Table: Table 6: Results of supervised fine-tuned models on FormulaReasoning.

Table 6 shows the results for the supervised fine-tuned models, with and without CoT-SFT and DA, which were detailed in Section 4.2.4.
In most settings, both models achieved higher scores on the id test than the ood test, yet they still exhibited considerable performance on the ood test. This indicates that 1) the ood formulas indeed challenged model performance and 2) the models still demonstrated a certain level of generalizability.
We hope that the division of id test and ood test will be helpful for assessing the generalization ability of fine-tuned models in future work.

It was noteworthy that with CoT-SFT, Qwen-1.8B and ChatGLM3-6B, with a mere parameter count of 1.8B and 6B, respectively, achieved performance comparable to GPT-4o (though such a comparison may not be entirely fair). This indicated that the incorporation of CoT-SFT and the use of calculators could significantly enhance the reasoning capabilities of small models. Our findings revealed that focusing on reasoning with CoT while delegating numerical calculation to a calculator could enhance the performance of small models, given their limited calculating capability. The assistance of LLMs for data augmentation could also enhance smaller models’ reasoning capability. This finding provides valuable insights for future deployment of numerical reasoning systems with small models.

### 5.5 Case Study and Error Analysis

Figure: (a) An error case caused by wrong formulas.
Refer to caption: x2.png

We sampled 50 error cases from the id test (few-shot setting) of GPT-3.5-turbo and manually categorized the types and proportions of errors. We divided the error types into two main categories: formula errors and calculation errors. Formula errors encompass inappropriate formulas and omitted formulas, while calculation errors primarily involve inaccuracies in numerical calculation and unit errors. We found that 38% of errors were caused by incorrect formulas, while the remaining 62% were attributable to calculation errors.
We provide one example for each of the two types of errors listed in Figure 2. It could be observed that FormulaReasoning poses challenges to existing models in terms of formula application and numerical calculation (including unit calculation and arithmetic calculation).

## 6 Conclusion and Limitations

We introduced FormulaReasoning, a dataset for formula-based numerical reasoning. We annotated the reasoning steps with formulas for each question with both manual and LLM-assisted efforts. Furthermore, we constructed a formula database after merging formulas with similar meanings, serving as an external knowledge base for subsequent retrieval-based/augmented approaches.
We evaluated FormulaReasoning across various sizes of LLMs, supervised fine-tuned models, and retrieval-augmented LLMs, demonstrating its challenging nature as an unresolved task. Our findings indicate substantial room for improvement of existing models on formula-based numerical reasoning, thus motivating future research efforts.

In the future work, we plan to utilize the formula knowledge from FormulaReasoning to improve the numerical reasoning capabilities of LLMs. Possible approaches include enhancing reasoning abilities through knowledge-driven methods, preference learning methods based on formula feedback.

One limitation of this work is that our evaluation results reported in the paper were obtained from the original Chinese version of FormulaReasoning. We have employed a combination of LLM-based translation and manual review to release an English version of FormulaReasoning. Currently, we provide a preview English version in our GitHub repository, and we will release the official English version of FormulaReasoning after completing the manual review process.
Another limitation is that, our dataset is limited to the domain of physics. Although junior high school physics is not overly complex and can be understood by most people which would benefit evaluation efforts, it is still possible to explore formula-based question answering data in other domains.

## Appendix A Appendix

### A.1 Dataset Construction

#### A.1.1 Prompts in Formula Normalization

The process of formula normalization is delineated into three distinct stages: the generation of natural language explanations, the extraction of the associated parameters from the explanations, and the subsequent error correction phase. The initial two stages are illustrated in Figures 3 and 4. The third stage is further splited into three specific error categories, each addressed by a dedicated prompt: input errors, where the parameters mentioned in the explanation are absent from the question; calculation errors, which occur when the calculator reports an error during the computation process; and output errors, where the final computed answer is incorrect. We provide an example here focusing on prompts for correcting calculation errors, while prompts for the other two error types can be found in our code submission. The prompts designed to correct calculation errors are depicted in Figure 5. The entire normalization procedure employs a 6-shot prompting, an instance of which is provided herein for illustrative purposes.

#### A.1.2 Examples of Deleted Questions

The questions which remained incorrect despite multiple attempts by the LLM were of notably poor quality, including missing important reasoning steps, wrong reference answer, and so on. Here is an example of these questions in Figure 6.

#### A.1.3 Semantic-based Merging for Formula Database Construction

Semantic-based merging primarily employs the LLM to comprehend formulas, ascertain if two formulas are semantically equivalent, and subsequently determine whether they can be merged into a single formula. The prompt for this procedure is illustrated in Figure 7. This approach ensures that the nuanced meanings embedded within formulas are accurately captured and evaluated for potential merging, thereby enhancing the quality of formula database.

### A.2 Experiments

#### A.2.1 Data Augmentation (DA) for FormulaReasoning

There have been several studies utilizing large language models (LLMs) for data augmentation (Ding et al., ). The data generated in these related works (Zheng et al., ; Whitehouse et al., ) primarily focus on daily conversations or sentiment analysis and do not require rigorous numerical calculations. Some research on data augmentation involving numerical calculations (Shum et al., ) employs LLMs to generate solutions to questions to aid in training, rather than creating complete questions. In contrast to these approaches, our work generates complete questions that involve numerical calculations (particularly formula calculations), along with automatic improvement and selection to ensure data quality.

In order to enhance the capabilities of models, we use LLM to generate more data for fine-tuning. We divide the process of data generation into the following several steps.

First, we randomly generated 17,000 prompts. Each prompt was obtained by stacking five question-answer pairs sampled form training set. At the end of the prompt, LLM was required to generate the sixth question-answer pair.
Second, we normalized the generated formulas. Except for the absence of manual review, the remaining steps were consistent with those in Section 3.2. At last, we unitized the calculator to check whether the calculation process in the data generated by the LLM is correct, and discarded the generated data with incorrect calculation processes.
After the above steps, we finally retained more than 2500 questions.

We found that mixing the newly generated data into the original training set did not always bring positive improvement, perhaps because the newly generated data has not undergone manual review. We found that randomly selecting a small portion of the newly generated data can enable the model to have performance improvement. We set several different mixing ratios selected from $\{5\%,10\%,15\%,20\%,2\%,30\%,35\%,40\%\}$. We fine-tuned each model using the augmented data set. After training for a fixed number of steps (150k and 200k), we selected the checkpoints with the smallest loss among models of different mixing ratios.

#### A.2.2 Zero-shot and Few-shot Prompts

Zero-shot and few-shot prompts are shown in Figure 8.

#### A.2.3 Formula Retriever

Let the number of formulas in the formula database be $N$. During training, we randomly initialized a matrix $\mathbf{F}\in\mathbb{R}^{N\times d}$, where $d$ is the hidden size and the $i$-th row in $\mathbf{F}$ represented the initial representation of the $i$-th formula in formula database. We denoted a batch of questions with a batch size of $B$ as $Q=\{q_{1},q_{2},...,q_{B}\}$. The indices of the gold-standard formulas corresponding to these $B$ questions were denoted as $L=\{l_{1},l_{2},\cdots,l_{B}\}$ (i.e. the label of $q_{i}$ is $l_{i}$, where $1\leq i\leq B$).

BERT was utilized to encode each question,

$$ $\mathbf{h}^{i}_{cls},\mathbf{h}^{i}_{1},\cdots=\mathtt{BERT}(q_{i}),1\leq i \leq B.$ (1) $$

Subsequently, we took the CLS vector $\mathbf{h}^{i}_{cls}$ as the representation for the $i$-th question.

We utilized in-batch negatives and contrastive learning loss,

$$ $\mathcal{L}=-\frac{1}{B}\sum_{1\leq i\leq B}\log\frac{\exp(\cos(\mathbf{h}^{i} _{cls},\mathbf{F}_{l_{i}}))}{\sum_{1\leq j\leq B}\exp(\cos(\mathbf{h}^{i}_{cls },\mathbf{F}_{l_{j}}))}.$ (2) $$

Each question might correspond to multiple correct formulas, and we ensured that the same question did not appear twice in the same batch when loading the data. Based on the implementation of Chinese-BERT-wwm-base, we tested the retrieval performance on the id test set and found that Recall@5 reached 97.69%.

Models were evaluated with top-5 retrieved formulas. Prompts can be found in Appendix A.2.5. We utilized zero-shot CoTs.

#### A.2.4 Comparison of CoT and PoT Prompts

Table: Table 7: Results of LLMs with zero-shot and few-shot chain of thought (CoT) and program of thought (PoT).

Results are shown in Table 7. In the PoT approach, we utilized a Python interpreter to execute the code and obtain the final results.
We found that the performance comparison between CoT and PoT varies across models. GPT-4o consistently demonstrated superior performance with PoT across all settings, achieving improvements of 2.58 points on average in the zero-shot setting and 7.14 points on average in the few-shot setting. In contrast, GLM-4-plus showed an average decline of 1.36 points in the zero-shot setting but showed an average improvement of 1.60 points in the few-shot setting. The finding might be related to the code capabilities of the models.

#### A.2.5 Prompts for LLMs with Formula Retriever

We added the formulas before each question in the few-shot setting. For the examples sampled from the training set, gold-standard formulas were added before each question. For the final question from the test set in both zero-shot and few-shot prompts, we included the top 5 retrieved formulas. The prompts are shown in Figure 9.

Figure: Figure 3: Prompt of the formula normalization stage 1.
Refer to caption: x4.png

Figure: Figure 4: Prompt of the formula normalization stage 2.
Refer to caption: x5.png

Figure: Figure 5: Prompt of the formula normalization stage 3: error correction for “calculation error”.
Refer to caption: x6.png

Figure: Figure 6: An example of deleted questions.
Refer to caption: x7.png

Figure: Figure 7: Prompt for semantic-based merging.
Refer to caption: x8.png

Figure: (a) Zero-shot prompt for LLMs.
Refer to caption: x9.png

Figure: (a) Few-shot prompt for LLMs with formula retriever.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. What is the use of a *unified formula database*? What if we don't merge the formulas? What happens?
2. How does performance change if there are more variables in the formula? For example z = x+y (2 variables) and z = a*b + c/d (4 variables). Do models struggle if there are many variables in the formula?
soundness: 3
strengths: * *Novel dataset addressing a critical gap*: The FormulaReasoning dataset fills a clear need for benchmarks that evaluate formula-based reasoning, going beyond simple arithmetic to test domain-specific knowledge applications.
* *Rigorous dataset construction process*: The multi-step process for collecting, annotating, and normalizing the questions and formulas is well-described and appears to be carefully designed.
* *Comprehensive evaluations*: The experiments cover a wide range of model types and sizes, providing a thorough picture of current capabilities. The results highlight the challenges of formula-based reasoning for current models and demonstrate the potential of techniques like Chain-of-Thought supervised fine-tuning for improving performance.
summary: This paper presents a novel question-answering dataset called FormulaReasoning for evaluating formula-based numerical reasoning capabilities of language models. The dataset addresses a critical gap in numerical reasoning benchmarks by explicitly incorporating formulas and domain-specific knowledge. The paper thoroughly describes the dataset construction process and presents baseline results using various state-of-the-art language models and fine-tuning approaches. The FormulaReasoning dataset contains 5,420 questions covering 272 formulas.
weaknesses: 1. The idea of *Formula Retriever* does not make sense. Why will the representation of the question and the required formula be similar? For a given question, let's say z = x/y is the right formula to use.  The representation of z = x*y and z = x/y will be very similar. Unless this is handled (maybe using special tokens), the model will always struggle to choose the right formula.
2. While there is a brief error analysis, a more in-depth examination of the types of mistakes made by different models could provide additional insights.
3. While the focus on physics is well-motivated, discussing how the approach might generalize to other formula-heavy domains would strengthen the paper.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Could you evaluate smarter models on your benchmark?
2. What are the identified formulas useful for, exactly?
3. What do you see as the default use case of this dataset in the next year?
soundness: 3
strengths: The dataset construction process is well-documented and thorough. I've looked into the supplementary files at https://anonymous.4open.science/r/FormulaReasoning, and I commend the authors for enabling the reviewers to inspect the dataset easily.

This is, to the best of my knowledge, the first dataset with such an extensive categorization of the mathematical formulas used in every problem.

Although the test split is too small, the overall dataset itself large enough to reliably detect small improvements in accuracy.
summary: The key contributions of the paper are 
- a dataset consisting of 5420 reasoning-based questions from Chinese junior high school physics examinations, where each question requires the use of formulas to solve.
- a formula database containing 272 unique formulas used in the above questions, that were assembled by a meticulous process of symbolic,  LLM-based, and manual filtering.

The authors also evaluate a range of LLMs on a subset of their dataset.
weaknesses: The paper says, verbatim:
> Such results demonstrated that there remained a substantial gap between the current capabilities of state-of-the-art LLMs and human performance. This was even more pronounced when considering smaller-scale models. These findings underscored _the challenging nature of FormulaReasoning as an unresolved dataset_, and that there was significant room for improvement in LLMs as they struggled to match human levels of reasoning.

I prompted _o1-preview_ with the problems from `id_test.json` https://anonymous.4open.science/r/FormulaReasoning/id_test.json in the supplementary material. The dataset seems to be in no particular order, so to avoid cherry-picking, I took the first 20 problems in the dataset. The prompt was "Translate the problem to English, then solve it. question: [problem in Chinese]".

The score was 19/20. Looking into the only one it missed:
```
  "如图所示，是平底热水壶，其质量为0.8kg，内底面积为180cm^2．某次用该热水壶装1.2L水放在水平桌面上，测得水深10cm，初温25℃。[ρ_水=1.0*10^3kg/m^3，C_水=4.2*10^3J/（kg℃），g=10N/kg]加热前水对壶底的压力多大？",
```
translates to:
```
As shown in the figure, this is a flat-bottomed kettle with a mass of 0.8kg and an inner bottom area of ​​180cm^2. Once, the kettle was filled with 1.2L of water and placed on a horizontal table. The water depth was measured to be 10cm and the initial temperature was 25℃. [ρ_water=1.0*10^3kg/m^3, C_water=4.2*10^3J/(kg℃), g=10N/kg] How much pressure does the water exert on the bottom of the kettle before heating?
```
The ground truth answer is given as 18 N, while the model gives the answer in units of Pa. However, the dataset lacks figures, so it's unclear if all necessary information is provided. Additionally, the correct answer seems odd, as the question asks for pressure, but N is a unit of force. This discrepancy could be due to a misunderstanding of the original Chinese statement by both myself and _o1-preview_.


I suspect a better prompting + a cleaned dataset would make even _o1-mini_ score very close to 100% when run on the entire dataset.


Summary: 
- I strongly disagree with the characterization of  “the challenging nature of FormulaReasoning as an unresolved dataset”. The dataset is very easy for today's LLMs and is not useful as a benchmark.
- The paper does not present a clear vision for a use of this dataset apart from benchmarking LLMs;
- Apart from the _o1_ series, the paper lacks evaluations for Claude-3.5-Sonnet and Llama-3-405b;
- While the dataset filtering and formula identification methods are quite involved, the overall utility of this dataset remains unclear. Specifically, the purpose of the identified formulas is not well-defined. A "formula database" derived from high school physics questions may be an interesting artifact for educational research, but its relevance to this conference is less apparent.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: The example in Figure 1 is presented in English, despite references to Chinese-specific techniques (e.g., using Chinese-BERT-wwm-base as the retriever). Clarification on the dataset's current language and any future plans for language expansion would be helpful.
soundness: 3
strengths: - The data collection process is meticulous (e.g., formula normalization and the construction of formula databases), resulting in a high-quality reasoning dataset with annotated formulas.
- The experiments provide a solid baseline for the benchmark, covering a range of LLM approaches effectively.
summary: The paper presents FormulaReasoning, a new numerical reasoning dataset comprising 5,420 questions with reasoning paths and corresponding formulas. The dataset's raw questions are sourced from Chinese junior high school physics exams, then filtered and annotated through a collaborative process involving human annotators and LLMs. Experiments benchmark various LLMs and prompting methods, including zero-shot, few-shot, and retrieval-augmented approaches.
weaknesses: - The motivation for creating this dataset lacks clarity. In the introduction, the authors state that “Current datasets … do not reflect the complexity of real-world problems.” However, it appears that formulas are neither a necessary nor a sufficient condition for solving complex numerical problems. Similarly, on line 41, the dataset is described as requiring “domain-specific formulas to guide the numerical reasoning process,” yet this new requirement does not necessarily demonstrate that formulas add value. From my perspective, annotated formulas are particularly valuable for reducing hallucinations and enhancing interpretability, especially when LLMs are applied in scientific domains where this matters more. The authors may wish to clarify the significance of their dataset for the broader community.
- The metric does not include formula validation. According to Section 4.3, the metric only measures the accuracy of the final result. It raises the question of why annotated formulas are introduced if LLMs can produce correct answers without explicitly displaying formulas.
- The dataset’s coverage is limited. It consists solely of physics questions (5,420 samples) with 272 unique formulas. As a retrieval database or a testbed for LLM formula knowledge, expanding to more domains and increasing the dataset size would enhance its applicability.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: - Did you scan MMLU and GPQA diamond for formula based reasoning? I suspect these datasets must have formula based reasoning similar to your dataset rather than GSM8K, Math23K-F, or MAWPS-F.
- Did you try testing GPT-o1's capabilities on your benchmark? It would be valuable to see how much improvement that results in. 
- Can you provide more examples where the top-performing model fails? I believe it's important to understand when and why these models fail. Is it mostly calculation or formula mistakes?
soundness: 3
strengths: FormulaReasoning introduces a unique dataset that has a particular focus on formula-based reasonings. The authors test a wide-range of LLMs on this dataset and compare to the performance of high school students on these problems.

The paper clearly explains the methodology on how the dataset was collected and how the benchmarking was done. Furthermore, it is very well written and easy to follow.
summary: The paper introduces a dataset of 5,420 questions to test formula-based reasoning of LLMs. The main contribution of this dataset is that

1. All questions require use of formulas.
2. Dataset contains a larger number of formulas (272) compared to existing datasets such as Math23K-F and MAWPS-F.

The questions are parsed from China high school physics examinations so I believe they rather test the ability to apply physics formulas rather than mathematics. The authors apply some manual and LLM based preprocessing steps in order to get a clean dataset where ever question is accompanied by a formula. 

The authors test existing LLMs on this benchmark and demonstrate that LLMs still perform worse than humans on this task (%84 vs %92).
weaknesses: The paper's main technical contribution is that it introduces a *formula-based* reasoning dataset, however, I believe that there are existing datasets such as GPQA (https://arxiv.org/abs/2311.12022) and MMLU (https://arxiv.org/abs/2009.03300) that has questions that require the use of formulas. This renders the contribution of this paper relatively limited.

The analysis of where the existing models fail is also limited - authors show two examples of failures but do not further elaborate on failure modes or statistics. In addition, I think presenting the failures of the strongest model rather than GPT-3.5 would be more informative.

The reason I gave a 3 is that the existing benchmarks already require formula-based reasoning - so the main/only benefit of this paper seems to me that the formulas are explicitly written out. However, I don't see how this helps with evaluation of the models. It could be helpful if these formulas were novel ones that do not appear in model's training data, in which case, the formula could be included in the prompt to test model's ability to include it (or the model could be asked to retrieve it in a "RAG" style). However, most of these formulas are still mainstream ones that the model would see in its training. With this in mind, it seems to me that this dataset is more of an additive set to existing benchmark and lack technical novelty required for ICLR publications.