Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes (2024)

Abstract: Numerical reasoning is an essential ability for NLP systems to handle numeric information. Recent research indicates that fine-tuning a small-scale model to learn generating reasoning processes alongside answers can significantly enhance performance. However, current methods have the limitation that most methods generate reasoning processes with large language models (LLMs), which are “unreliable” since such processes could contain information unrelated to the answer. To address this limitation, we introduce Enhancing NumeriCal reasOning with Reliable procEsses (ENCORE), which derives the reliable reasoning process by decomposing the answer formula, ensuring which fully supports the answer. Nevertheless, models could lack enough data to learn the reasoning process generation adequately, since our method generates only one single reasoning process for one formula. To overcome this difficulty, we present a series of pre-training tasks to help models learn the reasoning process generation with synthesized data. The experiments show that ENCORE yields improvement on all five experimental datasets with an average of 1.8%, proving the effectiveness of our method1.

heading: 1 Introduction
text: Numerical reasoning is an essential ability for NLP systems to handle arithmetic questions in real scenarios, which refers to generating answers to numerical questions with given evidence (Geva et al., 2020). The evidence is to support reasoning since the question could not furnish all necessary information, as seen in passages and tables in numerical data (Zhu et al., 2021b,a; Chen et al., 2021). The answer generated encompasses various elements, including values (Dua et al., 2019), programs (Chen et al., 2021), and formulas (Zhu et al., 2021a)2.
∗Corresponding author. 1Our code is released in link. 2For the sake of conciseness in this paper, we collectively
refer to these elements as formulas.
Currently, although LLMs have demonstrated great performance on the numerical reasoning (Chen et al., 2022a; Gao et al., 2022), we argue that it is still valuable to study and employ the small-scale model (e.g., BARTLARGE (Lewis et al., 2020)) since their low computational efficiency and decent performance, which still have application value in real scenarios. Previous research has demonstrated that teaching small-scale models to generate reasoning processes during fine-tuning can make the prediction more accurate and explainable (Cobbe et al., 2021; Ho et al., 2023; Magister et al., 2023). For example, SCOTT (Wang et al., 2023) employs LLMs to generate reasoning processes based on questions and answers, which are used to fine-tune small-scale models. However, the reasoning processes of the current methods could be unreliable since most methods employ LLMs to generate the processes, where such processes could contain information that does not support the answer (e.g., bold words in the left part of Figure 1).
10812
To address this issue, we present a novel numerical reasoning method called Enhancing NumeriCal reasOning with Reliable procEsses (ENCORE). Our method decomposes the operands and operators from the answers as the reasoning process, which we concatenate with the answer as the output of the model. The generated reasoning process of ENCORE is reliable since the process entirely derives from decomposing the answer, ensuring that the process does not contain answer-unrelated information and fully supports the answer, as shown in the right part of Figure 1. However, our method could lack enough data to enable the model to learn the reasoning process generation adequately, as for one answer, our method generates only one reasoning process, while methods based on LLMs can generate multiple processes (Ho et al., 2023). To overcome this difficulty, we present a series of pre-training tasks to help models learn the process generation with synthesized data.
To evaluate the effectiveness of ENCORE, we adopt it on five mainstream numerical reasoning datasets with various settings. Compared with baseline models, ENCORE brings performance improvement on all datasets and leads an average boost of 1.8%, showing the effectiveness and generalization of ENCORE. Moreover, in comparison to fine-tuning with the reasoning process generated by gpt-3.5-turbo, our method is superior by about 10%, which proves that the reasoning process generated by ENCORE has higher quality.
Our contributions can be summarized as follows: • To ensure that generated reasoning processes
fully support answers, we propose ENCORE, which generates reliable reasoning processes by decomposing answer formulas. • To alleviate the difficulty of the insufficient data scale of ENCORE, we introduce a series of pretraining tasks, which enhance the generation of the reasoning process with synthesized data. • To prove the effectiveness of ENCORE, we evaluate it with five mainstream numerical reasoning datasets, which yield performance improvement on all experimental datasets with an average boost of 1.8% compared with baselines.

heading: 2 Background
text: The numerical reasoning task is to generate single or multiple formulas as answers based on the given question as well as the textual and tabular evidence. About the input, textual evidence contains several
paragraphs, and tabular evidence consists of one or more tables, where the first row and column are called table headers, which describe information about the cells of the corresponding rows or columns. About the output, one formula consists of operators and operands (e.g., 2 + 1 × 3). The operands refer to the values manipulated or processed in the formula (e.g., 2, 1, 3). The operators are arithmetic symbols or functions that are to be performed on the operands (e.g., +, ×).
However, in practical applications, answers could not be annotated with formulas. We also try to apply our method to such data without formulas. For the questions with simple calculations (e.g., DROP (Dua et al., 2019)), we can directly generate the formulas following the previous work3. For the more complex calculations (e.g., GSM8K (Cobbe et al., 2021)), we employ LLMs with in-context learning, which can generate answers with a few samples without fine-tuning, to prove the effectiveness of the reasoning process we designed.

heading: 3 Methodology
text: In this section, we present our numerical reasoning method called ENCORE. First, we introduce the pipeline of ENCORE, which generates reliable reasoning processes fully supporting the answers (§ 3.1). Then, we propose three pre-training tasks based on ENCORE to enhance the generation performance of the reasoning process (§ 3.2).

heading: 3.1 ENCORE
text: In this section, we introduce ENCORE, which enhances numerical reasoning by generating reliable reasoning processes decomposed from answers. The illustration of ENCORE is shown in Figure 2.

heading: 3.1.1 Retrieve
text: Because evidence irrelevant to the question could mislead the model, causing performance degradation, we employ a retriever to retrieve the questionrelevant evidence. We concatenate each text paragraph and table column with questions, then feed it into a binary classification model to generate correlation confidence. The classification model is trained with the ground evidence annotated by the dataset or the headers in the located formulas obtained in § 3.1.2. Then, we sort each text paragraph and table column with the correlation confidence
3https://github.com/allenai/ allennlp-reading-comprehension
of the model output. We select the top-k evidence as the retrieval result, where k is determined by the input length limit, and then we concatenate such evidence with the question as the model input.

heading: 3.1.2 Locate
text: This step is designed to reduce the difficulty of value memory and table understanding by changing the value format in answers. In prior work, it has been observed that current models struggle to accurately retain complex floating-point values present in the evidence (Thawani et al., 2021). Besides, the table understanding ability of most models is limited, as linearized input disrupts the structural information of the table (Jin et al., 2022). To alleviate the challenge of extracting numerical values from tables, we propose substituting values in the answer by locating their respective headers in the table, which we call the located formula. For instance, as illustrated in Figure 2, instead of directly using the value “1.1”, we use “{Col1, Row1}” corresponding to its cell headers in the table. Consequently, the model only needs to recall the headers associated with relevant cells, lowering the difficulty of specific value memory and table understanding, thereby enhancing the reasoning performance.
We use string matching to locate the cell corresponding to each value in the formula, which could not handle the cells with the same value. An example of our labeling method is shown in Appendix B However, how to detect the question-related entities in the evidence is a long-studied problem,
which has been discussed in detail by the previous works (Liu et al., 2021; Kumar et al., 2023; Wu et al., 2023). Therefore, to focus on the main topic of this paper, we will discuss how to merge the detecting methods with ENCORE in the future.

heading: 3.1.3 Decompose
text: The designed motivation for this step is to reduce the complexity of reasoning through multi-step generation. Current methods, which ask models to generate formulas in one step, often lead to challenges in establishing a clear correspondence between the answer and the input information. For example, most operands in the formula of the answer are typically extracted from evidence, while the majority of operators are determined by the semantics of the question (e.g., “ratio” in the question leads to the division operator). Moreover, some formulas are too complex to generate correctly in one step. To address these issues, we design a multi-step generation process for the model to achieve the numerical reasoning results. We decompose the formula into operators and operands, which are used to ask the model to generate before located formulas. By first generating the more straightforward operators and operands and then generating the complete formula, we can reduce the difficulty in generating formulas, thereby enhancing accuracy.

heading: 3.1.4 Fine-tune
text: After constructing the located formulas and corresponding operators and operands, we take them with answers as output and use the question and
the retrieved evidence as input to fine-tune the seq2seq model. During the construction of inputs and outputs, we use tags like the form “<A>” to distinguish different parts. We also add other information in the output sequence to meet the requirements of different datasets, such as value scales (e.g., “billion” and “percentage” in TAT-QA) and spans (e.g., span-type answers in DROP).

heading: 3.2 Pre-Training
text: With the reasoning process generated by ENCORE, the model could still struggle to learn how to produce such processes because of the limit of the training data scale. To aid the model in learning to generate the reasoning process, in this section, we introduce three pre-training tasks. We synthesize questions, answers, and reasoning processes based on different templates, then pre-train the model with all these data as the multi-task training. The template and example of each pre-training task are shown in Appendix A.
We primarily design pre-training tasks for tabular evidence rather than textual evidence, for two reasons: (1) Most current pre-trained language models are trained on textual data, ensuring their proficiency in generating text-related reasoning processes. (2) Direct linearization of the tabular evidence during input disrupts the structural information, making it challenging for the model to generate table-related reasoning processes.
Table Location Prediction is designed to help the model better locate operands in the formula by learning the correspondence between the cell and the corresponding table headers. Given the row and column headers of one cell, the model should predict the value of this cell.
Table Calculation Prediction is designed to enhance operator generation. About the tabular evidence, many calculation formulas involve all values in one column as the operands, such as the average or total value of a column. We help models learn the generation of these formulas with the given column and the calculation type.
Hierarchical Table Prediction is designed for models to perform better operand extraction by comprehending the hierarchical table structure with multi-level headers (Zhao et al., 2022), where the whole table can be seen as several sub-tables. For this task, models should predict the name of the first level of each given table header.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Datasets We apply ENCORE on five datasets with various settings: FinQA (Chen et al., 2021), ConvFinQA (Chen et al., 2022b), TAT-QA (Zhu et al., 2021a), MathQA (Amini et al., 2019) and DROP (Dua et al., 2019), which cover different types of evidence and answers. More detailed information can be found in Appendix C.
Metrics For MathQA, FinQA, and ConvFinQA, we employ execution accuracy as our evaluation metrics (Chen et al., 2021). About DROP and TATQA, we evaluate methods with the exact match (EM) (Zhu et al., 2021a). For TAT-QA, we additionally use Arithmetic EM to represent the EM on numerical reasoning questions. The definition of these metrics can be found in Appendix D.
Baselines We adopt BERTBASE (Devlin et al., 2019) as our baseline retrieval model. We use BARTLARGE (Lewis et al., 2020) and T53B (Raffel et al., 2020) as our baseline seq2seq models.
Settings All experimental models are implemented with Huggingface transformers (Wolf et al., 2020) and Fairseq (Ott et al., 2019). We adopt the pre-training tasks to TAT-QA, FinQA, and ConvFinQA with 121, 732 synthesized examples since they contain tabular evidence. More detailed settings are shown in Appendix E.

heading: 4.2 Main Results
text: The main experiment results are summarized in Table 1, where the detailed results on each dataset are shown in Appendix F. ENCORE brings performance improvement on all experiment datasets with all used baseline models and achieves SOTA or near-SOTA results on most datasets, which proves the efficiency and generalizability of our method. Compared to BARTLARGE, our method exhibits more obvious improvements on T53B, suggesting that larger-scale models can more effectively learn the generation of reasoning processes and apply the associated capabilities to answer generation. However, our method exhibits an obvious discrepancy with the current SOTA on DROP. This is attributed to the low quality of the synthesized formulas, where the synthesized results could be incorrect, which subsequently misleads the model into erroneous reasoning processes, resulting in poor generation performance.
It is noteworthy that, in comparison to datasets that solely utilize textual evidence (e.g., MathQA, DROP), our method exhibits a more significant improvement on datasets with both textual and tabular evidence (e.g., TAT-QA, FinQA). This is because our designed located formula addresses the challenge of cell location, where textual evidence does not involve this challenge. Besides, our designed pre-training tasks mainly focus on tabular evidence, so the improvement of textual evidence is less obvious when compared to tabular evidence.

heading: 4.3 Ablation Studies
text: In this section, we perform ablation studies to further evaluate the performance of ENCORE. We use TAT-QA as our study dataset since it covers various types of evidence and answers, which can comprehensively reflect the performance of the model.

heading: 4.3.1 Reasoning Process Studies
text: To verify that each designed part of the reasoning process in ENCORE is effective, we perform ablation experiments on each part separately, which is shown in Table 2. We can see that each part of the reasoning process contributes the performance improvement, which proves the effectiveness of the
reasoning process designed by our method. According to the arithmetic EM, we can see that extracting operands has the most apparent impact on model performance. This is because the model regards the values in the answer as part of the formula structure, lacking the awareness of extracting values from evidence. The effect of the located formula is also apparent, which proves that it is hard for models to map the table headers to the corresponding cell value. The improvement of introducing operators is not significant since the formula operator is similar to that in the answer.
To verify the impact of generation orders of different parts of the reasoning process, we also adopt the ablations of the reasoning process format under two settings: generate operands first and operators first. Generating operands first leads to 74.1% on EM, and generating operators first is 73.6% in our experiment, showing that generating operands first is better, which is the order we used in ENCORE.

heading: 4.3.2 Pre-Training Studies
text: To prove that all designed pre-training tasks are effective, we conduct ablation experiments on them. Table 3 shows the experiment results of ENCORE with different pre-training settings.
From Table 3, we can observe that: (1) the ablation of each pre-training task leads to a drop in performance, which proves the effectiveness of all pre-training tasks; (2) the performance increment of Arithmetic EM is much higher than EM of all types of questions, which proves that the pretraining does improve model performance by improving numerical reasoning capabilities; (3) Table Calculation Prediction task leads to the most significant improvement for the model, proving that the ability to handle operator generation of the baseline is weak, while our method effectively improves the ability to handle such a reasoning process.

heading: 4.4 Analysis
text: Does ENCORE generate better reasoning processes than using LLMs? To compare the quality of the reasoning processes generated by ENCORE and by LLMs, we fine-tune models using reasoning processes produced by both methods. We employ gpt-3.5-turbo to generate reasoning processes given the question and the answer with zero-shot inference. The performance of different process sources is shown in Table 5. From the table, we can see that the model fine-tuned with the reasoning process generated by using ENCORE markedly outperforms using gpt-3.5-turbo. Therefore, the reasoning process synthesized by our method can better help small models generate correct results.
Does ENCORE work well on various answer formats? To prove our method can handle data from multiple scenes at the same time, we adapt ENCORE to the unified setting by merging multiple datasets. We transfer the answer formats of MathQA, FinQA, and numerical reasoning questions of TAT-QA into the domain-specific language format (Amini et al., 2019) to unify the answer format (e.g., 2+ 1× 3→ add(2,multiple(1, 3))). Then, we train the models in this unified setting and evaluate them on the unified and divided datasets respectively to evaluate the performance.
The experimental results are shown in Table 6, from which we can observe that: (1) compared with single training, the unified setting achieves much better performance since more training examples make the model learn more numerical reasoning knowledge; (2) ENCORE can further improve the performance under the unified setting by 1.5% compared with the baselines, demonstrating the generalizability under different answer types.
Is the reasoning process format of ENCORE still effective for in-context learning? Although ENCORE brings great performance improvement, it cannot handle the questions annotating answers without formulas (e.g., GSM8K). Considering the brilliant in-context learning ability of the current LLMs, we conduct experiments to verify whether the reasoning process format of ENCORE can still improve the performance without fine-tuning.
We compare our method with two prompt methods: generate directly and with Chain-of-Thought (CoT) (Wei et al., 2022), where CoT asks LLMs to generate answers with the prompt “think it step by step” using 8-shot prompt following Fu et al. (2023). The detailed prompts we used are shown in Appendix G. We evaluate ENCORE on the arithmetic subset of TAT-QA, FinQA under the 3-shot setting and GSM8K with 8-shot since the questions of GSM8K are harder than the above two datasets. As shown in Table 7, compared with CoT, ENCORE brings an average performance improvement of 8.9% on all datasets and LLMs, which shows that our method is still effective under the in-context learning setting.
What is the performance of ENCORE on different answer types? We categorize the predictions based on answer types and sources, which are shown in Table 4. About the performance of different question types, compared with the baseline model, ENCORE improves the performance of arithmetic questions with 4.9%, showing that our method does improve the numerical reasoning ability. Furthermore, our method also shows enhancements for other types of answers, indicating that the reasoning process generation can elevate the reasoning for various answer types to some extent. About the results of different evidence sources, ENCORE increases the performance of table-source and hybrid-source questions, showing that generating located formulas indeed lowers the difficulty of the table understanding.
However, ENCORE suffers from performance degradation on the text-source and span-type, which are mainly span extraction questions. There are two reasons for this result. Firstly, there are no fixed rules for annotating span-type answers, which leads to performance fluctuations during the prediction. Besides, our method focuses on improving the numerical reasoning ability, and the additionally generated information (e.g., operands, operators) could reduce the span extraction ability.
What are the main error types of ENCORE? To better understand how our method improves the numerical reasoning performance of models and to better observe the direction of future improvement, we study the current error distribution on numerical reasoning questions. We categorize the error cases into three types: (1) operand denotes that the extracted operators is incorrect; (2) operator means that the model makes mistakes in the operator generation; (3) other is the error other than the above two categories. We randomly select 256 numerical questions and then analyze them manually, which is shown in Figure 3.
From Figure 3, we can observe that: (1) with ENCORE, the model makes fewer mistakes on all error types, showing that our method can significantly improve the model performance on both operator generation and operand extraction; (2) the most significant error drop is in the operand error
type since the operand extraction and the located formula make the model not need to memorize specific values, lowering the difficulty of reasoning; (3) the operand error types still account for the main part of the bad cases with ENCORE, which requires follow-up work to continue to improve the operand extraction ability of models.

heading: 4.5 Case Study
text: To better understand how ENCORE improves the numerical reasoning ability, we show an example case of TAT-QA in Figure 4, which requires locating the cell value based on the column name and the row name. The baseline model generates a wrong number 754, which does not exist in the table, showing that the baseline method makes it hard to detect the question-related value in the table.
With ENCORE, the model correctly corresponds the header names in the question to {col10, row2}, and then extracts the corresponding value 774 by locating the header without model reasoning. That is because our method obviates the need for the model to memorize specific values and reduces the complexity of table understanding, thereby decreasing the difficulty of the operand extraction.

heading: 5 Related Work
text: Numerical Reasoning Numerical reasoning is a widely researched topic to handle questions about documents with rich numerical information. Previous works have found that it is hard for the model to generate the numerical answers directly and explore generating reasoning processes for enhancing interpretability and extra supervision for answer generation (Ling et al., 2017). Based on this finding, many researchers have designed model structures to generate reasoning processes implicitly (Ling et al., 2017; Zhu et al., 2021a; Lei et al., 2022; Shao et al., 2022; Zhang and Moshfeghi, 2022). Considering the powerful few-shot inference ability of LLMs without fine-tuning, Wei et al. (2022) presents the chain-of-thought to generate the reasoning process by LLMs themselves, attracting much attention (Wang et al., 2022; Ye and Durrett, 2022; Kojima et al., 2022; Chen et al., 2022a).
Although LLMs have demonstrated impressive performance in the numerical reasoning task, their substantial computational overhead hinders their deployment in practical applications. To address this issue, we explore the use of small-scale models with low computation for numerical reasoning, enhancing their reasoning capabilities by training them to generate reasoning processes.
Answering with Reasoning Processes Previous research has indicated that concurrently generating reasoning processes while producing answers can significantly enhance the accuracy of the responses (Wei et al., 2022; Chu et al., 2023). Subsequent studies have revealed that LLMs exhibit varying performance when generating different types of reasoning processes (Chen et al., 2022a; Gao et al., 2022; Ziqi and Lu, 2023). Apart from LLMs, researchers also find that fine-tuning small-scale models using reasoning processes generated by LLMs can also improve performance (Cobbe et al., 2021; Ho et al., 2023; Wang et al., 2023; Magister et al., 2023). Considering the lower computational overhead and acceptable performance of small-scale models, such works remain worthy of research.
However, the aforementioned methods confront the challenge wherein the reasoning process generated by LLMs could not fully support the answers. To address this limitation, our method directly obtains the reliable reasoning process by decomposing operators and operands from the answers. Our method enhances the quality of the generated reasoning process, thereby improving the accuracy of the generated answers.

heading: 6 Conclusion
text: We propose a novel numerical reasoning method called ENCORE to address the limitations of the reasoning process generation. Compared with previous methods, ENCORE can guarantee to generate reliable reasoning processes that fully support the answer and aim models to learn to generate the process with pre-training. According to experiments, ENCORE brings significant performance improvement over all five experimental datasets, leading to an average improvement of 1.8% compared with baselines, which shows the effectiveness and generalization of our method. Meanwhile, our method is superior by about 10% in comparison to the reasoning process generated by gpt-3.5-turbo, which proves the higher quality of the reasoning processes generated by ENCORE.
Limitations
ENCORE has two limitations, including: 1. About the operand extraction, we directly check whether each operand appears in the evidence, which could lead mistake. For future work, we will adapt better grounding methods, enhancing the extraction accuracy. 2. It is required that the training data be labeled with formulas, demanding high label overhead. In future work, we will employ LLMs to synthesize formulas, thereby reducing label cost.
Ethics Statement
All datasets and models used in this paper are publicly available, and our usage follows their licenses and terms.

heading: Acknowledgment
text: We thank all anonymous reviewers for their constructive comments. We gratefully acknowledge the support of the National Natural Science Foundation of China (NSFC) via grant 62236004, 62206078 and 62441603 and the support of Du Xiaoman (Beijing) Science Technology Co., Ltd.

heading: A Template and Example of Pre-Training
text: The templates and examples of our pre-training method are shown in Table 8.

heading: B Example of Automatically Labeling
text: Take Table 9, the question “I want to know the balance sum from 2018 to 2020” and the labeling answer “(113, 246 − 18, 967) + (120, 523 − 19, 786) + (125, 843− 21, 355)” as an example.
The grammar of the answer formula format is shown in Table 10, where we omit the specific descriptions of Operator and Operand. Following this grammar, the given answer can be parsed as “( Operand Operator Operand ) Operator ( Operand Operator Operand ) Operator ( Operand Operator Operand )”. We replace all Operand with x1, x2, ..., and keep all Operator as the original symbols. Then, we can get the formula structure “(x1 − x2) + (x3 − x4) + (x5 − x6)”.
About the operands, we can directly extract the values corresponding to the operands. In this example, they are “113, 246, 18, 967, 120, 523, 19, 786, 125, 843, and 21, 355”, which are also the extracted values. After that, we employ the string match method and locate their positions in the table and get {Col2, Row1}, {Col1, Row1}, {Col2, Row2}, {Col1, Row2}, {Col2, Row3}, {Col1, Row3}, which are the extracted entities. Although the method above is sample yet effective, how to detect the question-related entities in the evidence completely correct should be carefully studied (e.g., MITQA (Kumar et al., 2023), TACR (Wu et al., 2023)). Considering the complexity of this issue, we leave how to solve it as future work.
Correspondingly, the ENCORE output of this example is “<V> {Col2, Row1} | {Col1, Row1} | {Col2, Row2} | {Col1, Row2} | {Col2, Row3} | {Col1, Row3} | <D> (x1 − x2) + (x3 − x4) + (x5 − x6) | <A> ({Col2, Row1} − {Col1, Row1}) + ({Col2, Row2} − {Col1, Row2}) + ({Col2, Row3} − {Col1, Row3}) ”.

heading: C Experiment Datasets
text: In this section, we discuss the detailed information on the datasets of the experiments. The settings of these datasets are shown in Table 11
FinQA A financial HybridQA dataset while it only contains numerical reasoning questions, just like MathQA. The operations of FinQA are fewer
and more straightforward than MathQA, which only includes less than ten elementary operations. Following the settings of TAT-QA, we also locate the program values’ positions as extracted entities. FinQA contains 8,281 examples and is released as training (6,251), validation (883), and test (1,147) following a 75%/10%/15% split.
ConvFinQA A context-dependent version of FinQA, which decomposes the complex numerical reasoning questions into multiple steps. During the validation, it evaluates the model predictions of all steps.
TAT-QA A financial HybridQA dataset containing textual as well as tabular evidence and four types of answers including span, multi-span, count, and arithmetic. We follow the origin derivation format of the dataset and extract all values in the derivation, then locate the positions of the values in the table as extracted entities. TAT-QA consists of 16,552 question-answer pairs and is split into the training set (80%), development set (10%), and test set (10%).
MathQA A numerical reasoning dataset containing DSL-format questions involves more than one hundred scientific operations. Because the maximum step number of one answer is 66, which is too long for the model to generate, we only train the model on the example with the number of answer program steps less than ten that have covered the 93% dataset questions. MathQA consists of 37k problems, split into (80/12/8)% training/dev/test sets. We do not add the values to the inputs of MathQA since we do not design the alias for the text.
DROP A reading comprehension dataset contains three types of answers, including spans, date and number, where number questions only require + and - operations. Considering that it does not annotate the calculation process, we first extract all numbers in the evidence, calculate the similarity between the question and the words around every number, then select the legal calculation formula with maximum similarity. DROP includes a total of 96,567 question-answer pairs and is partitioned into training (80%), development (10%), and test (10%) sets.
GSM8K The GSM8K is a collection of 8,500 meticulously created and linguistically varied math word problems suitable for grade school levels, crafted by professional problem writers. This dataset is divided into two sections: 7,500 problems
Method Dev Test
EM F1 EM F1
Discriminative Methods
NumNet+ (Ran et al., 2019) 81.1 84.4 81.5 84.8 QDGAT (Chen et al., 2020) 84.1 87.1 84.5 87.6
Generative Methods
for training and 1,000 for testing. Each problem is designed to be solved in 2 to 8 steps, predominantly involving a series of basic arithmetic operations (addition, subtraction, multiplication, division) to attain the ultimate solution. These problems are intended to be solvable by middle school students and are ideal for enhancing multi-step mathematical reasoning skills.

heading: D Evaluation Metrics
text: Exact Match measures the percentage of predictions that match the ground truth answers. Usually, two arithmetic answers are considered equal if their four decimal places are equal, following the rule of rounding function.
Numeracy-Focused F1 Score measures the average token-level overlap between the predictions and the ground truth answers, which can reduce false negative labeling. When an answer has multiple spans, the numeracy-focused F1 performs a one-to-one alignment greedily based on the bagof-word overlap on the set spans to ensure every current span can get the highest F1 value, then compute micro-average F1 over each span.
Program Accuracy measures the accuracy of operands and operators between the predicted programs and the golden programs.
Execution Accuracy measures the accuracy of the execution result of the predicted programs.

heading: E Hyper-Parameter Settings
text: Our model is implemented with PyTorch (Paszke et al., 2019), transformers (Wolf et al., 2020) and Fairseq (Ott et al., 2019). About the retrieval model, we set the learning rate as 2e-5 and the dropout as
0.1. We select three negative examples for every positive instance, set batch size as 16, and max epoch as 20, which takes 10 hours for training. For every question, we retrieve the top 5 as candidate evidence. About the generation models of PLMs, we consider it a Seq2Seq task with label-smoothed cross-entropy loss. We set the learning rate as 1e5, dropout as 0.1, and weight decay as 0.05. We use max tokens as 8192 during every step, update the model parameter every four updates, and warm up with 5000 updates. We set the max epoch as 1 for pre-training, 100 for BARTLARGE, and 20 for T53B, save the model every ten epochs during finetuning and use early-stop checkpoints. Any other hyper-parameters following the default settings of the package. To lower the difficulty of table understanding, we mark the numerical order of each column in the table. About the LLMs generation models, we set top_p as 0.95 and temperature as 0.
We employ one NVIDIA A100 40G GPU as our experiment device. The retrieval model takes around 6 hours for training. The PLMs generation model takes around 1 hour for pre-training, 12 hours for BARTLARGE fine-tuning, and 48 hours for T53B fine-tuning. The LLMs generation model takes about 1.5 hours to infer for 1k examples.

heading: F Detailed Experiment Results
text: In this section, we show the detailed results of each experiment dataset in Table 12, Table 13, Table 14 and Table 15.

heading: G Prompts of ENCORE with LLMs
text: In this section, we present the prompts we used with LLMs in § 4.4 in Table 16, Table 17 and Table 18.

heading: Future Work
text: Considering the complexity of this issue, we leave how to solve it as future work. Correspondingly, the ENCORE output of this example is “<V> {Col2, Row1} | {Col1, Row1} | {Col2, Row2} | {Col1, Row2} | {Col2, Row3} | {Col1, Row3} | <D> (x1 − x2) + (x3 − x4) + (x5 − x6) | <A> ({Col2, Row1} − {Col1, Row1}) + ({Col2, Row2} − {Col1, Row2}) + ({Col2, Row3} − {Col1, Row3}) ”