Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems
Authors: Marek Kadlčík
Michal Štefánik
Ondrej Sotolar
Vlastimil Martinek
Venue: EMNLP 2023 Main (2023)
Keywords: arithmetic reasoning
multistep reasoning
dataset
generation
TL;DR: We create Calc-X datasets collection allowing to train Calcformer generative language models to offload the reasoning steps requiring arithmetic computation to precise symbolic systems such as a calculator.

Abstract: Despite outstanding performance in many tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. We address this deficiency by creating Calc-X, a collection of datasets that demonstrates the appropriate use of a calculator in reasoning chains. Calc-X is suitable for teaching language models to offload computations to a symbolic system.
We survey and unify several existing chain-of-thought datasets into a proposed format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new Calc-X collection to train open-source calculator-using models and show that these models approximately double the accuracy of generating correct results compared to vanilla language model baselines.

CALC-X and CALCFORMERS: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems
Marek Kadlčík 0* and Michal Štefánik 8* and Ondřej Sotolár and Vlastimil Martinek*
Faculty of Informatics, Masaryk University, Czech Republic
"The Central European Institute of Technology, Masaryk University, Czech Republic
(kadlcik, stw.ssik.,,sseeeeee.eee www.olassoooararaooo.ooo 445261) eemail wwwni.cunz.......mm.............
Abstract
How many apples...?
Despite outstanding performance in many tasks, language models are notoriously inclined to make factual errors tasks requiring arith- metic computation. We address this deficiency by creating CALC-X, collection of datasets demonstrates the appropriate use of a cal- culator in reasoning chains. CALC-X is suit- able for teaching language models to offload computations to a symbolic system. We sur- vey and unify several existing chain-of-thought datasets into a proposed format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new CALC-X collection to train open-source calculator-using models call CALCFORMERS and show that these models approximately dou- ble the accuracy of generating correct results compared to vanilla language model baselines. We make all CALC-X datasets, source code and CAlCFoRMERS models publicly available.'
You had 4 apples, and received 2. Now you have <gadget> 4+2 </gadget> <output> 6 </output> apples.
Figure Generation process CALCFORMER models: By generating the closing </gadget> tag, model calls an external tool. The following tokens are inserted into the model's context by the tool. Finally, the model continues generation using all of the previous tokens.
based approaches (Section 2) with compromises in the quality and reproducibilityy
To support future research in developing open- source tool-assisted language models, we curate a CAL.c-X collection of over 300,000 samples for mathematical reasoning. CALC-X transforms sev- eral existing datasets into unified format that can be used to train and evaluate LMs for the correct use of calculator. We survey existing chain-of- thought (CoT) datasets for arithmetical reasoning (Section 2) and pick subset suitable for integration into a consistent collection. To enable efficient inte- gration of LMs with independent tools, we propose a unified format of a fully parseable HTML--lie markup language (Section 3). For each dataset, we describe the curation process of its calculator- augmented (CALC-X) version (Section 4). Finally, show that training on full mixture of CALC- X datasets enables LMs use a symbolic system during inference and largely improves the accuracy on held-out math problems (Section 5). We make all our building tools, datasets and models publicly available (Appendix A.4).
1 Introduction
While the language models (LMs) demonstrate out- standing efficiency in working with unstructured language data, they struggle with problems that require exact computations (Patel et al., 2021b). On the other hand, symbolic systems, such as a calculator, can perform arithmetics without errors. Thus, combining the strengths of both neural and symbolic systems can yield significant benefits in tackling tasks that require arithmetics (Schick et al., 2023; Gao et al., 2023).
Given a sufficient amount of supervised data, the interaction with symbolic systems can be learned. However, obtaining texts demonstrating the interac- tion in relevant situations and in a consistent struc- ture is non-trivial. Consequently, a substantial ef- fort of most related work addresses the data scarcity problem through semi-supervised learning, heuris- ties, prompting or few-shot, and reinforcement-
Related Work
Math Datasets GSM8K (Cobbe et al., 2021) con- tains grade-school math problems with human- written CoT explanations and explicit annotation of formulas. ASDiv (Miao et al., 2020), SVAMP (Pa-
"Equal contribution 'https: http//gttttttttte/e/ http:///gghhuuttcmoeeeeeeeeee com/ppromppeess/cllccc

---

tel et al., 2021a), MAWPS (Koncel-Kedziorski al., 2016) and larger Chinese Ape210k (Zhao et al., 2020) contain problems of similar complex- ity. but the solutions are written as nested expres- sions. AQuA-RAT (Ling et al., 2017) contains multiple-choice problems with selected answers and free-text rationales. MathQA (Amini et al., 2019) is subset of AQuA-RAT with additional an- notation: nested expressions that lead to answer, similar those in Ape210k or ASDiv. MATH and AMPS (Hendrycks et al., 2021) consist of more challenging problems and contain CoT solutions formatted in I Specifically, MATH is a set of high-school math competition problems, and AMPS is large-scale pre-training dataset, partly scraped and partly synthetically generated. Math- ematics Dataset (Saxton et al., 2019) is another generated dataset but containing only final answers.
3 CALC-X Interaction Format
We propose a semi-structured format for CoT data to provide both the flexibility of unstruc- tured text and the precision of structured formats. The HTML-based structure of interactions is com- patible with existing parsers, such as Beautiful- Soup (Richardson, 2007). This allows fast execu- tion of parsing in the interaction with tools within the generation but also allows future work to eas- ily transfer CAL.-X collection into other desired interaction formats.
Our format, displayed in Figure 2, uses three tags: gadget, output, and result. Tag gadget is in- tended for inputs or "queries" to an external system. Tag oulpul wraps the response of the external sys- tem to the query. The tag result wraps the final result of the thought chain.
main contribution of much of the previous work in building tool-using models addresses the problem of data scarcity. Komeili et al. (2022), WebGPT (Nakano et al., 2021), and LaMDA (Thoppilan et al., 2022) let crowd workers annotate tool calls to train models to use web search, calculator, and machine translation sys- tem. PAL (Gao et al., 2023) applies prompt engi- neering to make an LM use a Python interpreter without training. Toolformer's approach (Schick et al., 2023) is to prompt an LM to insert gadget tags ("API calls") into CoT datasets and filter out irrelevant ones using the trained model's perplexity. In evaluation, Toolformer simplifies the problem to only one tool call per example and supports gener- ation with several heuristic rules.
Tool-using LMs
After buying bread and candy bar, you have 32-3-2= <gadget (de"ccllccaator3s3==222gdgeee <output>>2<<outtou>>>
$27. You spend 27/3= <gadget d
<output>></output> dollars on the turkey. You have 27-9= <gadget ide"callullatoo"22==////gog <output>1&</output>>>>> $18 left. The final result is 18. <resul 1>18</result>
An example of target text from chain-of- thought dataset encoded in our proposed format. Our format is designed to allow the interaction of LMs with multiple external systems, such as a calculator.
Figure 2:
Creation of CALC-X Collection
Out of the datasets reviewed in Section 2, we cre- ate the first version of CALC-X collection from these datasets: GSM8K, AQuA-RAT, MathQA, Ape210k, MAWPS, SVAMP and ASDiv. Our se- lection considers the datasets' size, primary focus on arithmetics, and parseability of the tool calls.
TaLM (Parisi et al., 2022) extends a training dataset by self-training They start with a small set of fully annotated data, including the annotations of tool calls, and then iteratively generate CoTs with tool calls for a larger dataset with incomplete annotation. The examples added to the training set are chosen heuristically without guarantee that the entire reasoning chain is correct.
The resulting CALC-X collection is designed simplify the correct usability of the whole collec- tion in both training and evaluation while persist- ing the maximum of datasets' original information. Most importantly, the process includes (1) the uni- fication of key attributes (i.e. inputs, labels and correct results) over all datasets, and (2) the clim- ination of data leakages between different (train/- val/test) data splits throughout the whole collection.
We note that none of the referenced work pub- licly releases the resulting models. In combination with the many training and inference heuristics, it is largely difficult to reproduce and build upon the proposed methods. However, the availability of an extensive, standardized collection of tool- assisted datasets like the one presented by CALC-X will allow future work to substantially simplify the methods needed for creating assisted models.
We perform the second step based on lexical overlap between pairs of samples' input texts from different splits across all datasets. We consider a

---

pair of train and test examples a leak if the Jaccard similarity of their 1-gram and 2-gram representa- tions is over 0.5. This results in data splits com- posed of subsets of the original datasets, but thanks to this step, the whole CALC-X collection can be used to perform both validation and tests over all datasets when all datasets are also used in training.
Nested expression: (2 8) + (2 - 8) * (50x
Linear chain: - 8 -6
5e 100 1/2
(1/2) + 3 7/2
(-6) * (7/2) = -21
(-6) (-21) = -27
The remainder of this section describes the con- version process of each dataset currently included in the CALC-X collection.
Figure 4: Linearization of nested expression
4.1 GSM8K
quence of steps and remove examples whose end result does not numerically match the original re- sult saved in the data. We also discard all exam- ples with the original result written in the form of such as ((1/2) because of the ambiguity between implicit multiplication and mixed fractions, which are both present the data in the same form. In total, more than 97% the examples in each split passed all checks and were kept in the dataset.
GSM8K (Cobbe al., 2021) is CoT dataset with over 8,000 examples containing arithmetical ex- pressions that can be evaluated using a calculator. The syntax is not standard but can be easily parsed.
"Natalia sold 48/2 = ((48/2 24)) 24 clips in May. Natalia sold 48+24 = ((48 + 72)) 72 clips altogether in April and May. #### 72"
Finally, the linearized examples be di- rectly transformed into our unified format. While Ape210K is much larger than GSM8K, the ex- ported chains do not contain any comments in natu- ral language, and the English prompts are machine- translated.
Figure 3: The syntax used in the GSM8K dataset.
In GSM8K, the calculations are explicitly anno- tated, and removing the tags from chain-of-thought results in natural language sentences. The final result is single number that is also explicitly an- notated at the end of the solution.
Analysis of overlaps shows that around 60% of both validation and test examples present dupli- cates near-duplicates to the Ape210K's training split. In CALC-X, we remove these examples from validation and test splits with around 1700 remain- ing in each.
We parse the formulas using regular expressions, evaluate them using the sYMPy library (Meurer et al., 2017), and verify that all outputs are numerically close to the values in the data. The conversion into our unified format is a direct one-to-one mapping.
Our analysis shows that the original validation and test splits of GSM8K do not contain duplicates and are not contained in training split of other datasets.
4.3 AQuA-RAT
AQuA-RAT (Ling et al., 2017) is a dataset of 100K math problems. The annotations consist of 1) mul- tiple choices, 2) the correct choice, and 3) an infor- mal, free-text rationale that explains the selected choice. The answer is usually single number but can also be pair of numbers (coordinates), include unit, ratio, "None of the options," and others.
4.2 Ape210K
Ape210K (Zhao al., 2020) is dataset of over 200K math problems involving simple arithmetics. The questions are written Chinese, and the solu- tions are represented as nested arithmetical expres- sions and single numerical result to which they evaluate. We automatically translate the questions to English using Google Translate and linearize the nested expressions into sequence of simple expressions using depth-first traversal of the ex- pression tree. Figure 4 illustrates the process of linearization.
The rationale is in free-text format and gener- ally not parseable with formal grammar. In some cases, calculations are written in words, such "ten pages per day means seventy pages per week." We approach this in a best-effort manner and use regular expressions to find equations in the form of expression number. We remove all the non- symbolic characters (mainly all textual characters) from both sides of such-identified equations and evaluate the left-hand side using SYMPY calculator.
Furthermore, we discard all examples that can- not be parsed. Then, we evaluate the linear se-

---

Finally, we compare the calculator output with the right-hand equation side, and if the result of the calculator matches, we insert tagged calculator call into the rationale. This results in 1.6 calculator calls on average per single reasoning chain.
of AQuA-RAT. Hence, we completely remove the validation and test splits of MathQA CALC-X and omit evaluations on MathQA in our results.
4.5 MAWPS
Our error analysis shows that the annotators are usually consistent in their rationale structure. The described parsing heuristic works well for the anno- tations that consistently use "=" (equals symbol) in their chain. However, we usually do not inject any calculator calls for many others that do not follow the equation-following structure. Thus, for appli- cations with high priority of recall in the injected gadget calls, we propose to further filter our dataset to the samples with at least three calculator calls. Analysis of data leaks shows around 2% of the training split are near-duplicates of around 30% and 25% of test and validation AQuA-RAT sam- ples, respectively. In CALC-X collection, we remove these samples from the train split.
MAWPS (Koncel-Kedziorske et al., 2016) is col- lection of around 5000 elementary school-level problems from online websites. The solution each problem is annotated as an equation with single variable x. Solving the equation for x gives the answer to the problem. We isolate x from the equations by manual annotation and then linearize the corresponding expression into sequence of cal- culations to convert the data into our unified format. We do not train our models on any MAWPS data to ensure fair comparison with previous work. Around 70% of MAWPS's s train samples are near-duplicates of its train split, test split, or ASDiv- test split. We remove these samples from CALC- X's train collection.
4.4 MathQA
4.6 ASDiv-A and SVAMP
MathQA (Amini et al., 2019) is a subset (37K) of AQuA-RAT with further annotations. Human annotators have corrected errors inside the AQuA- RAT rationales and annotated the solution with nested expression that leads to the correct answer.
ASDiv (Miao et al., 2020) is an arithmetics benchmark with problems of similar difficulty MAWPS, ASDiv-A picks around 1,200 samples with number as a solution and a nested expres- sion evaluating to the correct result. SVAMP (Pa- tel et al., 202 comprises 1,000 math problems derived from ASDiv, overcoming some of its defi- ciencies.
We parse the nested expressions and linearize them using a similar procedure as for Ape2 Less than 0.3% of examples were removed due to parsing evaluation problems. We also replace all function calls (such as circle_area) with corre- sponding clementary operations that can be exc- cuted with SYMPY calculator.
Whole ASDiv-A and SVAMP datasets were di- rectly convertible to common format. With official train-test split, we use both for evaluation only.
Next, we keep the examples only if their ex- pression evaluation result is in =5% range of the selected correct choice in the data, which results in a loss of around 30% of the data. We note that the mismatch of the computed results with annotated options is not consistent with the authors' claim² that the expressions in the dataset are guaranteed evaluate the selected option, but is consistent with observations by Parisi al. (2022). After in- spection, we attribute most of these errors to the inconsistency in the original MathQA dataset. In our published variant of the dataset, we remove all examples in which the expression does not evaluate to value numerically close to the selected option. Evaluation of data leakages shows that all sam- ples of MathQA originate from the training split Fmath-qa. qa.......... tiub. wo/matoo/ooooo/o QA, accessed 20/110/2022
Experiments
To explore the potential of our newly curated data collection, we train models of identical architecture and parametrization in two configurations:
Train on the original datasets: use all of the selected datasets (see Section 4) to train a baseline: generative model that produces an associated output reasoning chain on a given input sequence. All training samples removed from CALC-X are also removed from baseline data for fair comparison.
2. Train CALCFORMERS on the CALC-X datasets: train the model for an identical objective, but on the corresponding CALC-X datasets, to demonstrate the interaction with a symbolic

---

GSMSK AQuA-RAT Ape210K
MAWPS
SVAMP
ASDiv-A
GPT-3
175B
19.9"
10.0"
14.0"
TOOLFORMER
6.7B
44.0"
29,4"
40,4"
T5-L
700M
17.0+3,4
26.646.6
19.0+188
11.2#28
18.2#23
29.7
CALCFORMER-TSA4
700M
34.2++4.
27.2+6.6
53.1+2.3
39.4+4411
34.4+3.0
55.6+2.8
TS-XL
3B
19.2#3.6
33.5+7.2
20.8
16.0+3.1
24.7=2..6
37.0=2.7
CALCTORMER T5-X
39.62444
33.5+6.9
53.8+2.3
49.0=4.3
44.7=3.1
66.8+2.6
FLAN-XL
3B
24.2#3.8
20.8+6.1
22.942.0
15.8+3.1
24.0=2.6
38.7+2.7
CALCFORMER CLANMEERR-----LL- FLAN-XL
39.4::44
31.2+666
53.4
47.1+4.3
46.7+3.1
72.5+2.5
Table 1: Percentage of correct results on test sets of listed datasets, for (i) models of previous work, (ii) CALCFORMER models trained on our CALC-X collection and evaluated with access to SYMPY calculator, and (iii) language models generation. Confidence intervals computed in bootstrapped evaluation (sample size n=500, repeats r=1 1,000)) bold denotes significantly best results for a combination of base model + dataset. Values marked with are self-reported results from Schick et al. (2023).
trained on the original datasets, evaluated in standard sequence-to-sequeet
system, In inference, whenever the model gen- erates the enclosing gadget tags, the model's generated text extended with the output of the SYMPY calculator (see Figure I).
We can see that on the AQuA-RAT dataset, CAL- CFORMERS perform comparably with baselines in the case of two of three base models. We find that this is due the low average number of tool calls inside the AQuA-RAS training split, leading to in- consistent usage of the calculator on AQuA-RAT test questions.
In both settings, we fine-tune T5 foundation mod- els of Raffel et al. (2020), and Chung et al. (2022) in 700-million and 3-billion parameters' versions, using teacher forcing and cross-entropy loss, com- monly applied on gruuence-t--squuee Transform- ers (Bahdanau al., 2016; Vaswani et al., 2017). We use greedy decoding inference.
Conclusion
This paper introduces a CALC-X dataset collection, transforming over 300,000 samples of arithmetic reasoning datasets into a unificd chain-of of-thuugth format with explicit annotation of interaction with a calculator. CALC-X enables integration of simple symbolic system in the reasoning chains of lan- guage models via traditional supervised learning, easily allowing the models offload mathematical computation an external tool.
We evaluate both systems by numerically com- paring the value of the final result extracted from the generated answer with the ground truth result. the case of AQuA-RAT, where the answer is one the options, we compare the predictions against all options and pick the one with the lowest Leven- shtein distance as chosen by the model.
We support the correct use of the CALC-X collee- tion for both training and evaluation by unifying the format of all included datasets and eliminating the datasets' mutual data leakages, making CALC-X a convenient default for any future research address- ing models' arithmetic reasoning.
In the case of the CALCFORMER models, the an- swer is enclosed in the <RESULT> tags that we use to extract the answer. For the baseline models, we extract the answer as the sequence following the key phrase "The final result is"; this format is pre- sented in all the baselines' training samples. Our training setup is further detailed Appendix A.
Finally, we demonstrate the potential of CALC-X by utilizing the whole unified collection in training and adjusting the models' inference process for the use of the calculator. As result, our calculator- assisted models reach an accuracy that approxi- mately doubles the accuracy of the traditional gen- eration and outperforms existing previous work. We make the CALC-X collection and newly-created caleulator-supported CAI.FFOrmER models publicly available to facilitate further research in the fast- paced area of tool-using language models.
Results Table compares the performance of the conventional generative models and calculator- supported models trained CALC-X datasets. CAL- CFORMERS surpass the accuracy of the baseline mod- els 2-3 times, with the exception of the AQuA-RAT dataset. Nevertheless, the overall improvement of the calculator calculator-using models in reaching the correct answer is 99.6% on average across all datasets.

---

Limitations
Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai. Mirac Suzgun, Xinyun Chen, Aakanksha Chowd- hery, Alex Castro-Ros, Marie Pellat, Kevin Robin- son, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Fed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling Instruction- Finetuned Language Models, arXiv e-prints, page arXiv:2210.11416.
We acknowledge the limitations of our heuristic for injecting annotations into AQuA-RAT rationales. We suggest future authors experiment with utilizing a sequence-to-sequenee language model similarly to the method by Schick et al. (2023), which might yield higher recall.
Further, we note that in some CALC-X datasets, the format of reasoning chains differs from others in that does not contain verbal explanations sur- rounding the computational parts. We believe that using a language model here to write explanatory comments in natural language between the com- putation steps is a promising path if a consistent format of CoT, including both arithmetics and free- text reasoning, is desired.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/21 14168.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. Pal: Program-aides language models.
We also note the limitations of simple calcula- tor in advanced mathematical reasoning. While the models' extension with a calculator eircumvents an important bottleneck, difficult mathematical tasks might require more general symbolic systems to obtain satisfactory results.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. CoRR. abs/2103.03874.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022, Internet-augmented dialogue generation. In Proceed- ings of the 60th Anmual Meeting of the Association for Computational Linguistics Volume 1: Long Pa- pers), pages -8788, Dublin, Ireland. Association for Computational Linguistics.
Acknowledgments
This work was supported by the Martina Roe- selová Memorial Fellowship granted by the 1OCB Tech Foundation. Computational re- sources were provided by the --INARA CZ project (ID:90254), supported by the Ministry of Ed- ucation, Youth and Sports of the Czech Re- public. Further, we acknowledge the Centre for Biomedical Image Analysis at Masaryk Uni- versity supported by MEYS CR (LM2203050 and c2.02.1.01/0.0/0..00 00000000000001880440000000040000 046/0016045 Czech- BioImaging) for their support with training the models evaluated within this paper. The work has also received funding from the Czech Science Foun- dation (project number 19-27828X).
Rik Koncel-Kedziorski Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, pages 1152-1157 San Diego California. Association for Computational Linguistics.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. CoRR, abx/1 705.04146
Aaron Meurer, Christopher P. Smith, Mateusz Pa- procki, Ondfcj Čertík, Sergey B. Kirpichev, Matthew Rocklin, Amit Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johans- son, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fer- nando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. 2017. Sympy: symbolic computing in python. PeerJ Computer Science, 3te103.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Ketziorski, Yejin Choi, and Hannanch Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. CoRR, 133119
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate, ArXiv: 440.0433.7
Shen-yun Miao, Chao-Chun Liang, and Keh-Yil Vih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In P Proceedings of the 58th Annual Meeting of the Association for
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi

---

Computational Linguistics, pages 975-984, Online, Association for Computational Linguistics.
Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR, abs/2201.082329
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang. Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. Webgpt: Browser- assisted question question-answentg answering with human feedback. CoRR, abs/2112.0000 12.0332..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Proc. of the 3 31 NIPS conference, volume 30 of NIPS '17. pages 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template- nichee-rchhhhh dataset of math word problems. CoRR. abs/2009. 1506..
Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models.
Arkil Patel, S. Bhattamishra, and Navin Goyal. 202 Are nlp models really able solve simple math word problems? In North American Chapter of the Associ- ation for Computational Linguistics.
A Training details
A.1 Model Settings
We trained the models using the HuggingFFace Transformers library. We applied standard sequence-to-sequence training with cross-entropy loss on all tokens. We optimized the model with Adam W optimizer and effective batch size of 32. We used learning rate of 5e-5 with 1000 warmup steps, and a linear Ir decay 0 in 400000 steps. The models were trained in bf16 precision. All models were trained on mixture of all datasets, either in CALC-X or in original format, with data upsampling to balance different dataset sizes.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021b. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(146):1-67.
A.2 Training progress
Leonard Richardson. 2007. Beautiful soup.
During training, monitored the percentage of validation predictions that have a correct final result. We compute the performance on each dataset sepa- rately and average them together, which we use for early stopping and selecting the best checkpoint after training.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathemati- cal reasoning abilities of neural models. CoRR, abs/1904.01557.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.
A comparison of the models on the aggregate metric during training is illustrated in Figure 5. The detailed (per-dataset) metrics of the best model can be seen in Figure 6.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze -ze Cheng, Alicia Jin, Taylor Bos, Leslie Baker. Yu Du, YaGuang Li. Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu. Zhifeng Chen, Adam Roberts. Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman- John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Agüira y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and
A.3 Hardware
To train each of our models, we used a single NVIDIA A A100 80GB GPU, 40GB of RAM, and 4 CPU cores. We have trained two models with 700M parameters and four models with 3B param- eters, with total training wall time of around 21 days.
A.4 Datasets, Models, and Code
We make our code, CALC-X datasets, and models publicly available.
Code:

---

validation results all datasets
Flon-dL
T5+L
Calc-15-11
tate-Tbel
532
100% 153x
2338
2T0k
100%
20%
Figure 5: Percentage of validation predictions with rect result during training. Baseline 15 XL run is miss- ing from because technical difficulties requiring re- runs from intermediate checkpoints.
validation results
0.
02
20k
4B
15k
wqunrrat gandk
age
Figure 6: Percentage of validation predictions with correct result of single run Calcformer-Flan on each dataset during training.
Datasets
huggiingfaee.co/ddttttttts//// wtw.//aatsstttt//////////ccaaaaa
huggingface.co//ll
Please note that more datasets might be added to the CALC-X in the future; see the project repository for an up-to-date list and ready-to-ur examples of the full CALC-X collection.
Calculator-supported Models
huggingface..c/MMMMMMire http:///e...caaaaaa/////aaaaeaaa
huggingface.com htt.f//w.....uuueeeec//aooommmar
huggingt beggingface.comm fttp.e///oo.////caaaaeee-----e--
See the corresponding model cards for usage.


Reviews:
[{"review_id": "M81QhqtrkG", "reviewer": "Reviewer_ZPay", "summary": "", "questions": "A: The improvement on datasets like GSM is not large. What are the typical types of error made by the fine-tuned T5 models? \n\nB: Why did you decide to have the model call tools using the HTML tag format? Could this impact the performance? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work; we appreciate your questions and feedback.\n\nWe agree with your comment on limited applicability beyond datasets that are well-structured to begin with. However, we still identify some possibilities for how future work can benefit from *both* free-form traces and structured Calc-X datasets to create calculator-assisted models:\n\n1. Pretrain on free-form data and then finetune on Calc-X data to teach the model to use tools.\n2. Train on a mixture of free-form and Calc-X data and inform the model which output format is expected via a prompt. During inference, always prompt the model to use Calc-X format to encourage tool-using.\n3. Utilize LLM to rewrite free-form answers into our format by injecting the tags, similar to Schick et al. [1].\n\n**Question 1A:**\nWe would argue that improvements in models' performance on the GSM dataset *are* substantial - from 17.0 to 34.2, from 19.2 to 39.6, and from 24.2 to 39.4. In other words, Calc-X models are around twice as likely to arrive at a correct final result than baselines in the GSM dataset.\n\nThe typical errors for the baseline models can be divided into two broad categories: (1) in reasoning and (2) in arithmetics. Calc-X models eliminate arithmetic errors as they outsource the computation to a reliable symbolic system. However, they still make errors in reasoning - such as drawing incorrect conclusions in multi-step reasoning or applying irrelevant formulas.\n\n**Question 1B:**\nWe choose the HTML-like tag format for its efficient parseability with existing tools, making the resulting models practical for integration in more complex inference scenarios, such as in conversational systems. \nIn related settings, previous work utilizing T5 models [2] found that a specific choice of interaction format does not play a substantial role in performance; this makes sense to us as the \"learning to interact correctly\" task is semantically much simpler than the downstream fine-tuning task at hand. \n\nIn line with this assumption, we also observe that models learn to generate valid (i.e., parseable) HTML tags early in the training process, and syntactical errors related to HTML formatting are practically non-existent in the outputs of fine-tuned models. Speculatively, HTML format could help models train faster because HTML syntax structure might appear in the pretraining corpus, but we do not support this hypothesis with quantitative evidence.\n\n\n\n[1] Schick et al.; [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761)\n\n[2] Raffel et al.; [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/abs/1910.10683); Section 2.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The dataset with unified format for arithmetic tool use is valuable, and the strong results show it clearly improves the math reasoning ability of smaller LMs. \n- The author have made the dataset and fine-tuned model available, which can facilitate progress in building this capability into LLMs more reliably.\n- Clear documentation of the data creation process as well as the released artifacts. ", "reasons_to_reject": "The proposed approach for data format conversion may have limited applicability beyond datasets with simple and well-structured arithmetic operations. As demonstrated by the difficulties with the AQuA-RAT dataset, the proposed processing strategy does not generalize well to more free-form reasoning traces. ", "questions_to_authors": "A: The improvement on datasets like GSM is not large. What are the typical types of error made by the fine-tuned T5 models? \n\nB: Why did you decide to have the model call tools using the HTML tag format? Could this impact the performance? ", "paper_topic_and_main_contributions": "The paper contributes a dataset that demonstrates use of calculator tool in mathematical reasoning chains. The dataset consists of more than 300K examples created using GSM8K, Ape-210K, AQuA-RAT, and MathQA, all of which contain mathematical reasoning chains involving calculations. To create the examples, the authors parse the three datasets into a unified HTML-style format, where calculations are expressed as <gadget></gadget> and <output></output>. \n\nFine-tuning T5 models on this data format instead of the original data leads to clear improvements in % of correct results, although there is no improvement for AQuA-RAT, which appears to also require intermediate computation results.\n\nThe authors have already released the model and dataset at anonymous endpoints on Huggingface, with clear documentation. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I can think of.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is clear and well-presented."}, "extra_text": {}}, {"review_id": "IWm8AK12tu", "reviewer": "Reviewer_yZbJ", "summary": "", "questions": "Based on my understanding of section 4, it sounds like a few thousand examples from Ape210k, AQuA-RAT, and MathQA were removed because their format couldn’t be parsed by your data annotation method. I think it would be very interesting to see how many of those removed examples can be solved correctly by models trained on the Calc-X format. Do you have results for anything along those lines?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for valuable feedback on our work.\n\nWhile recent previous works demonstrate the benefit of using a calculator during inference, contrary to these, we make all our datasets, code, and trained models publicly available, making it easier for other researchers to train and work with tool-using models. As we identify in the Related Work section, without standardized supervised datasets, much of the energy in previous work was invested in overcoming the data scarcity; they develop prompting methods with heuristic filtering to generate data [1], utilize self-training [2], or spend resources on proprietary annotation collection [3], all of which are slower, more complex, or more costly methods than training from a supervised dataset.\n\n\n**Question: Performance on dropped-out data**\n\n* **Clarification of data removal:**\n\n    First, we would like to clarify how many data instances are omitted and why.\n    \n    * In MathQA, only less than 0.3% of examples were removed due to parsing/syntactical problems. The rest (cca 30%) was removed because of inconsistency in the data, which we observed in agreement with other prior works [2, 4] that inspected the annotation in the MathQA dataset.\n    * In Ape210k, less than 3% of the data was removed, with *ambiguity* being the prevalent cause, not a parser insufficiency. We manually inspected the cause and found that mixed fractions and implicit multiplication were written the same way, making it impossible to distinguish between them algorithmically. \n    * In GSM8K and AQuA-RAT, we did not remove any instances, but we recommend filtering out AQuA-RAT instances with a low number of gadget calls in Section 4.3 and in Limitations.\n\n* **Evaluation on dropped-out data:**\n\n    Evaluating models on dropped-out data is costly because it requires manual human re-annotation - i.e., solving the math problems. However, we performed a small-scale evaluation on the dropped-out samples and report our results below.\n    \n    We selected 20 random instances in both MathQA and Ape210k (from the test split) that were dropped during the conversion process, solved them manually, and then manually compared the solutions to outputs from Calc-Flan-XL and Baseline-Flan-XL models.\n    For transparency, we make the manually collected evaluations within this analysis available as notebooks in our anonymized repository [5].\n    \n    * From 20 **MathQA**-dropped instances, 1 question was ill-formed with no valid solution. Calc-Flan-XL agreed with our solution in 7 out of the remaining 19 questions, and Baseline-Flan-XL agreed on 0 (none) of the 19 instances. While lower success rates compared to the measures in Table 1 for both models suggest that non-parseable samples are systematically more difficult, we also see that calculator-aided models are more robust in this data segment.\n    \n    * From 20 **Ape210k**-dropped instances, 9 contained ambiguity between implicit multiplication and mixed fraction in the question itself (and not just the solution), so we disqualified them from the evaluation. From the remaining 11 problems, 8 were solved correctly by Calc-Flan-XL, which seems consistent with the performance measured in Table 1 on kept instances. Baseline-Flan-XL solved none of the remaining 11 problems correctly.\n\n\n[1] Schick et al.; [Toolformer: Language Models Can Teach Themselves to Use Tools](https://arxiv.org/abs/2302.04761).\n\n[2] Aaron Parisi et al.; [Talm: Tool augmented language models](https://arxiv.org/abs/2205.12255).\n\n[3] Thoppilan et al.; [LaMDA: Language Models for Dialog Applications](https://arxiv.org/abs/2201.08239).\n\n[4] Shen-Yun Miao et al.; [A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers](https://arxiv.org/abs/2106.15772).\n\n[5] https://github.com/emnlp2023sub/gadgets/tree/master/notebooks", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The datasets proposed in this work, as well as the format, may be quite useful for future research into tool-use in LLMs\n\nThe results from this study are strong evidence of the benefit of tool use for LLMs for arithmetic reasoning.", "reasons_to_reject": "Prior works have clearly demonstrated the benefit of calculator use in large language models [1], so it’s a little unclear just how much novelty is derived from the results of this study. However, this work considers different datasets from previous works, and provides a more concentrated study on arithmetic reasoning within chain-of-thought.\n\n[1] Schick et al. Toolformer: Language Models Can Teach Themselves to Use Tools ", "questions_to_authors": "Based on my understanding of section 4, it sounds like a few thousand examples from Ape210k, AQuA-RAT, and MathQA were removed because their format couldn’t be parsed by your data annotation method. I think it would be very interesting to see how many of those removed examples can be solved correctly by models trained on the Calc-X format. Do you have results for anything along those lines?", "paper_topic_and_main_contributions": "This work focuses on the ability of large language models to use a calculator as a tool. To study this ability, the authors find 4 arithmetic reasoning datasets that have chain-of-thought annotations and convert them into a format (Calc-X) that allows the language model to utilize a calculator in addition to chain-of-thought. The authors consider 3 model variations (T5-large, T5-XL, and FLAN-XL), training each model on the original chain-of-thought dataset as well as their novel Calc-X format. This work demonstrates that through a unified tool-use format, language models can significantly improve accuracy on arithmetic problems.\n\nOverall, I think this paper provides a clear and focused study on arithmetic reasoning within chain-of-thought prompting, and demonstrates the benefits quite well.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ajCSnBIYLm", "reviewer": "Reviewer_jPuE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work and providing us with the feedback. We try to address your outlined Weakness points below.\n\n* **Reproducibility:** Given the space constraints, we try to keep the flow of the main article mostly high-level, but please note that we performed many extra steps to make our work as reproducible as possible, already for the anonymized submission; Complementary to the training details in Appendix A, on the first page we reference a [Huggingface hub](https://huggingface.co/emnlp2023) with both our datasets and models, including instructions for their use. Additionally, in Appendix A, we reference [anonymized GitHub repository](https://github.com/emnlp2023sub/gadgets) with instructions for the exact reproduction of the results reported in Table 1 as well as the training scripts with all parameters intentionally fixed.\n\n* **Data filtering:** We believe there is a miscommunication about the training data of baseline models. Data instanced filtered in the conversion process were removed for *both* calculator-using models *and* baselines. This can be verified with our anonymized GitHub repository [1]. However, we confirm that the paper does not explicitly state this fact, and we *will* include it in the camera-ready version.\n\n* **Claimed superior accuracy:** Our claim about superior accuracy achieved with Calc-X datasets is supported by the results reported in Table 1 - a comparison of *Calc-* models with the corresponding baselines shows that the improvement is two-fold or higher in a majority of cases. However, please note that we do not attribute the improvement to our format directly but primarily to the fact that the **models trained on Calc-X format can use (and do use) a calculator during inference**, eliminating the source of arithmetic errors.\n\n* **Suitability for the Resources track of EMNLP:** Please note that additionally to the proposed data transformations, important contributions of our work are also in a proposition and reproducible implementation of the tool-using model, as well as the empirical verification of efficiency of all the presented components with a variety of datasets and base models.\n\nPlease let us know if you still have concerns about any of our discussed claims (reproducibility, data filtering, superior results of Calc-X models) and our proposed improvement for the camera-ready. If the discussion dispersed some of these, we would appreciate it if you give one more consideration to our assigned *Soundness* rating.\n\n-----\n\n[1] GitHub repository: https://github.com/emnlp2023sub/gadgets. Relevant parts: lines 46-49 in file *examples/train\\_calc\\_balanced\\_baseline.py* and definition of *dataset_to_keys* variable in file *examples/baseline\\_utils.py*. Note that the baseline's training implementation loads *our* version of the datasets from HuggingFace, which *are already filtered* and contain the data in both ours and the original formatting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The common formatting and data normalization idea of the paper is surely relevant for the scientific community", "reasons_to_reject": "- The experiments are not that meaningful and not easily reproducible given the informations provided in the paper:\nthe comparisons of scores between datasets that end up being different (some examples are dropped by the conversion process) are nonsense to me.\n- The authors conclude that they get superior accuracy with their recoded dataset but this claim is not supported by the content\n- Although there is a ressources and evaluation track at EMNLP, I feel this paper would be more appopriate in a ressources and evaluation conference like LREC.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This short paper addresses the problem of data normalization in chain of thought (CoT) problems.\nThere currently exists several datasets for CoT and acquiring data for these task is time consuming and expensive.\nFurthermore the current data sets are heterogeneous and encoded in different formats, entailing a situation where data preprocessing becomes a real issue.\n\nThe authors introduce an HTML/XML format (dubbed Calc-X) for encoding the existing data sets which is easy to process with existing libraries.\nThey specifically describe the conversion processes for datasets GSM8K,Ape210K,AQuA-RAT and MathQA\n\nThere is a short experimental section relying on fine tuning models from the T5 family on the task by comparing performances on the original datasets and on the newly encoded datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a dataset designed to showcase the use of a calculator tool in mathematical reasoning chains. By leveraging four existing datasets, namely GSM8K, Ape-210K, AQuA-RAT, and MathQA, the authors have created over 300K examples. These datasets were parsed into a unified HTML-style format, termed Calc-X, where calculations are represented using specific tags. When T5 models are fine-tuned using this format, there's a noticeable improvement in the accuracy of results, except for the AQuA-RAT dataset. The authors have made the dataset and the fine-tuned model available on Huggingface, accompanied by comprehensive documentation. The paper emphasizes the potential of a unified tool-use format in enhancing the accuracy of language models in arithmetic problems.

The paper's contribution of a dataset with a unified format for arithmetic tool use is invaluable. The results indicate a clear enhancement in the mathematical reasoning abilities of language models. The authors have made both the dataset and the fine-tuned model accessible to the public, which can further stimulate advancements in this domain. The documentation detailing the data creation process and the released resources is commendably clear. The datasets and format proposed in this work could be instrumental for future research on tool-use in large language models. The study's outcomes serve as robust evidence supporting the advantages of tool use for arithmetic reasoning in large language models.

The paper's proposed data format conversion strategy might have limited scope, especially for datasets that involve complex and less structured arithmetic operations. This limitation is evident from the challenges encountered with the AQuA-RAT dataset. While the paper emphasizes the benefits of using calculators in large language models, similar findings have been previously reported, raising questions about the novelty of this study.  A reviewer suggests that the paper might be better suited for a resources and evaluation conference rather than an NLP conference.
Decision: Accept-Main