ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
Authors: Zhibin Gou
Zhihong Shao
Yeyun Gong
yelong shen
Yujiu Yang
Minlie Huang
Nan Duan
Weizhu Chen
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Mathematical Reasoning
Tool Learning
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: Tailor language models for interacting with tools to solve complex mathematical reasoning problems

Abstract: Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics. In this paper, we propose ToRA a series of Tool-integrated Reasoning Agents designed to solve challenging mathematical problems by seamlessly integrating natural language reasoning with the utilization of external tools (e.g., computation libraries and symbolic solvers), thereby amalgamating the analytical prowess of language and the computational efficiency of tools. To train ToRA, we curate interactive tool-use trajectories on mathematical datasets, apply imitation learning on the annotations, and propose output space shaping to further refine models' reasoning behavior. As a result, ToRA models significantly outperform open-source models on 10 mathematical reasoning datasets across all scales with 13%-19% absolute improvements on average. Notably, ToRA-7B reaches 44.6% on the competition-level dataset MATH, surpassing the best open-source model WizardMath-70B by 22% absolute. ToRA-34B is also the first open-source model that achieves an accuracy exceeding 50% on MATH, which significantly outperforms GPT-4's CoT result, and is competitive with GPT-4 solving problems with programs. Additionally, we conduct a comprehensive analysis of the benefits and remaining challenges of tool interaction for mathematical reasoning, providing valuable insights for future research.

Published as a conference paper at ICLR 2024
ToRA: A TOOL-INTEGRATED REASONING AGENT FOR MATHEMATICAL PROBLEM SOLVING
Zhihong Shao Yeyun Gonga Yelong Shen Yujiu Yang"! , Minlie Huang" Nan Duan", Weizhu Chen TTinggua University Miicroooft Research "Microsoft Azure AI 1gzb22, szh1 9) @ma i 1s s innhuuu en (yegong, yeshe, nanduan, wzchen @microsoft.com com
Zhibin Goul.2;
ABSTRACT
Large language models have made significant progress in various language tasks, yet they still struggle with complex mathematics. this paper, we propose ToRA, series of Tool-integrated Reasoning Agents designed solve challenging math- ematical problems by seamlessly integrating natural language reasoning with the utilization of external tools (c.g., computation libraries and symbolic solvers), thereby amalgamating the analytical prowess of language and the computational efficiency of tools. To train ToRA, we curate interactive tool-use trajectories on mathematical datasets, apply imitation learning on the annotations, and propose output space shaping to further refine models' reasoning behavior. As result, ToRA models significantly outperform open-source models on 10 mathematical reasoning datasets across all scales with 13%- 19% absolute improvements on av- erage. Notably, ToRA-7B reaches 44.6% the competition-level dataset MATH, surpassing the best soencsourcee model WizardMath- by 22% absolute. ToRA- CODE-34B is also the first open-source model that achieves accuracy exceeding 50% on MATH, which significantly outperforms GPT-4's CoT result, and is com- petitive with GPT-4 solving problems with programs. Additionally, comprehensive amalysis of the benefits and remaining challenges of tool interaction for mathematical reasoning, providing valuable insights for future research []
conduct
GPT-4-aade
ChatGPI-Dode
WizhttMMath
GPT4
ChatePT
SFT
ToRA
MATH
GSM8k
100
50
46.1
40.7
80
Accuracy (%)
63.9
Accuracy
51.9
$1.1
57.8
30
40
20
14.4
21.3
18.7
0
7B
13R
70B
7B
13B
70B
Figure 1: Comparing ToRA with baselines LLaMA-2 base models from 7B to 70B. ToRA models exhibit remarkable improvements over previous state-of-the-e approaches across all scales. In particular, ToRA-70B notably outperforms GPT-4's CoT result on MATH and attains comparable results to GPT-4 solving problems with code.
'Equal contribution. See Contributions section for details. Work done during internship at Microsoft. "Correspondigg authors. Code and models available httppp///ggithbb..oo/nrtrrrrreee

---

Published as a conference paper at ICLR 2024
Problem: Suppose that the of the squares of two complex numbers and y and the sum of their cubes 10. List possible values for x + y, separated by commas.
Let's use 'Sympy to calculate and print possible values + y.
T1
from sympy import symbols, Eq. solve, simplity
We are that
import math
yd = 7 and x² yd
def possible possible_vvlless]]
10. We can factor xo
def solution():
y =symbols(' y')
y" to get (x y)(x*
x symbole(X)) =
eq1 = =Eq(x"22 2_"x 7)
xy yf).
x"2++""==7
eq2 =q(x""3 + 10)
a1
x x"3+""3=10
solutions solve((eg1 (aaee(((e11ee22) eee2). y))
Thus, we can 10 = (x)) It follows that x + must divide 10.
surns = [simplify(sc([[] + 5]] for sol in
x (-1 math.
solutions]
y (-1- math. math.sqr((17))/2
return sums
print(possible values())
# Get all possible values for x-y
The only possibilities are that x y 1,2,
result [x + y, yy
[-5, 5, 1, 4,
01
>>> Synta cannot assign to expression.
Removing duplicates, the possible values for
x+ are 1boxed(-5, 1)
12
Rationale
Program
(c) Tool-integrated Reasoning (Format used by ToRA)
Ourpu
(a) Rationale-based
(b) Program-based
Figure 2: Examples of three reasoning formats for mathematical reasoning: (a) Rationale-bast
methods (e.g., CoT prompting) generate step-by-step natural language rationales, (b) Program-based methods (e.g.. PAL prompting) solve tasks with program synthesis, and (c) our proposed Tool- integrated Reasoning format interleaves rationales with program-based tool use. For brevity, present a simple example of single-round tool interaction, where the model creates rationale r1 for analysis, writes program a1 call an external solver, obtains the execution output 01. and then generates rationale r2 to finalize the answer.
INTRODUCTION
Large language models (LLMs), such as GPT-4 (OpenAl| 2023) and PaLM-2 (Anil et al. 2023). have demonstrated remarkable progress in a wide range of language tasks, particularly in the longstanding challenge mathematical reasoning (Feigenbaum et al. 1963 Hosseini et al.] (014). However, open-source models, such as LLaMA-2 (Touvron et 2023a and Falcon (Penedo et al. 2023).
still struggle with advanced mathematical reasoning tasks.
Existing works improve mathematical performance of language models either with step-by-step natural language reasoning (Wei et al. 2022) as illustrated Fig (a), or by synthesizing and executing programs to obtain the answers (Gao et al., 2022 Chen et al. 2022), depicted in Fig (b). Both approaches exhibit complementary advantages. Natural language is suitable for semantic analysis, planning, and abstract reasoning (e.g., commonsense reasoning). but struggles with precise computation, symbolic manipulation, and algorithmic processing. Conversely, programs excel in rigorous operations, and can outsource intricate calculations specialized tools like equation solvers.
To leverage the benefits of both natural language reasoning and program-based tool use, we train open-source models such LLaMA-2 to reason in way where natural language reasoning is interleaved with program-based tool use synergistically (as depicted in Fig (c)), thereby largely reducing the gap with closed-source models like GPT-4 mathematical reasoning. Specifically, we first design the interleaving format of reasoning. curate corresponding interactive tool-use trajectories for mathematical problems from the popular GSM8k (Cobbe et al. (02T)) and MATH (Hendrycks) et al. 2021) dataset, and then apply imitation learning on the high-quality annotations, leading to better performance than any existing open-source model. Furthermore, since the curated data is far from exhausting all valid trajectories for problem, relying solely on imitation learning restricts model's 's output space, hindering the flexibility in exploring plausible trajectories during testing. To improve the diversity of plausible reasoning steps and mitigate improper tool-use behavior, we apply output space shaping which additionally trains the models on both self- sampled valid trajectories and invalid ones that have been corrected by teacher model (e.g., 34B model can serve the teacher
2

---

Published as a conference paper at ICLR 2024
1
Imitation Learning
Tool-integrated Reasoning
ToRA-Corpus
Fine-tune
Valid Trajectories
Preblem
OHEpat
Rillande
Output Space Shaping
Valid Trajectories
Fine-tune
Problem
Ouppu
M
ToRA
M
Output Sampling
Teacher Correction
Figure 3: Training ToRA contains two steps. Imitation Learning: Prompt LLMs like GPT-4 to generate Tool-integrated Reasoning trajectories (TORA-CORPUS) and use this corpus to fine-tune model M; @ Output Space Shaping: Sample diverse tool-use trajectories with M, keep the valid ones, correct the invalid ones with a teacher model M', and retrain M on the union of sampled valid trajectories, corrected ones, and the initial PORA-CORPUS to obtain TORA.
for 7B model). Ouiput space shaping significantly boosts reasoning, allowing open-source models to attain an accuracy exceeding 50% on the competition-level MATH dataset for the first time.
We evaluate the resulting suite of Tool-integrated Reasoning Agents (ToRA) A) ranging from 7B to 70B on diverse mathematical reasoning datasets. As shown Fig | ToRA A series significantly outperform open-source models across all scales. Notably, the competition level MATH dataset, ToRA-7B outperforms the previous SoTA WizardMath-701 (Luo et al. 2023) by 22% absolute. TORA-CODE-341 beats GPT-4's CoT result (Bubeck al. (023) by 8.3% absolute (50.8% vs. 42.5%), ) and is competitive with GPT-4 solving problems with code (GPT-4-Code, 51.8%). addition, analyze the benefits and remaining challenges of tool interaction for mathematical reasoning, providing valuable insights for future work.
TORA: TOOL-INTEGRATED AGENTS FOR MATHEMATICAL REASONING
2.1 OVERVIEW
ToRA series solve challenging mathematical problems by leveraging both natural language reasoning and program-based basead tool use. shown in (c), given a mathematical problem g, TORA reasons with natural language, producing When reaching a point where program- based tool use is more appropriate for the subsequent task, c.g., equation solving, ToRA generates a program aj for tool use following natural language guidance T1- The exccution output о₁ will be fed to ToRA for subsequent processing including tool use adjustments, sub-tasks solving, or answer finalization. We repeat the process until the model places its answer within "\boxed ()", The resulting trajectory is denoted as t Tj..0o_. an On P where Fn contains the answer.
Fig presents the training pipeline of ToRA. We first collect interactive tool-use trajectories popular mathematical datasets. We then apply imitation learning on the resulting annotations, as well output space shaping to further refine models' reasoning behavior.
2.2 COLLECTING INTERACTIVE TOOL-USE TRAJECTORIES
Existing mathematical reasoning datasets primarily contain annotations in either natural language code, posing a challenge for training tool-integrated agents due to the absence of interactive tool-use annotations. To address this, utilize GPT-4 to synthesize high-quality trajectories on the GSM8k and MATH training sets. We select GSM8k and MATH as they exhibit diverse reasoning patterns, spanning multiple domains and difficulty levels.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Inference of Tool- Inte-Integr Reasoning
Require: problem 9. model G, prompt external tools E, stop condition Stop(-), maximum iteration rounds
I: To
Trajectory Initialization
for 1 to
3: r: > @ 0 7:-1)
> Rationale Generation (Eq. >> Stopping Criteria
4: Stop(r.) then
5:
return T:-1 rr end if
6:
7:
04 ~ Po(lpp 4 m
Program Generation (Eq. 2
8:
0: t 8(as)
> Tool Execution
9:
T @r:
> Trajectory Update (Eq.
10: end for
return 7n
Prompt Curation We compose instructions along with diverse few-shot examples, utilizing an in- terleaved format as depicted in Fig (c). These examples showcase interactive tool usage trajectories, incorporating descriptive variable names and combined program outputs. Please refer to Appendix E for the assembled prompts.
Inference Procedure We follow Algorithm and feed GPT-4 (g) with the composed prompt p to generate a tool-use trajectory T for each question from the training set. The trajectory is initialized as an empty string To, for each interaction round i, we first generate a rationale:
Ti ~ P.(.
+ T1-1)
where + means concatenation. If r; includes an answer within "\boxed| 1" (i.e., the stopping condition Stop(r;)), we cease generation, otherwise the model continues write program for tool use:
Pp((Ip @ T:-1 4 r;)
In line with Gou et al. (2023), if the model triggers the code execution stop words like ".* put", we supply it with the corresponding execution message and output o; by calling tools with o; t Ea,,). facilitating the generation of subsequent steps. Then, we update the trajectory by concatenating with the newly generated rationale T;, program aj, and output o;:
Tj f- T- ① rú 04
We repeat the above interaction process until we reach the maximum rounds n.
Trajectory Sampling We set 3 and perform inference using GPT-4 with greedy decoding, retaining trajectories that yield correct answers. For questions where GPT-4 fails with greedy decoding, apply nucleus sampling with sample size of 10 and keep up to 4 valid trajectories per question. Ultimately, we successfully annotate trajectories for 98.2% of GSMSk questions and 83.1% of MATH questions. After filtering out invalid trajectories with tool-use errors or wrong answers, we obtain 16k annotations which constitute our dataset ToR A A-CORPUS. Table compprrs ToRA- CORPUS with recently proposed mathematical reasoning datasets, while Table 6 in the Appendix displays MATH annotation accuracy details.
2.3 TRAINING
Imitation Learning We apply imitation learning on TORA-CORPUS by minimizing negative log-likelihood loss on the trajectory 7 conditioned on the problem
M = arg min M
log P Pererer4tttsssll..e (1......))))
(4)
2.T
where M is the resulting model. After imitation learning, Algorithm| by setting prompt to empty p = "" for inference. Imitation learning leads to state-of-the. art mathematical reasoning performance despite the small scale of ToR TORA-CORPUS.
simply apply the same procedure in
4

---

Published as a conference paper at ICLR 2024
Table 1: Compared with mathematical reasoning datasets, TORA-CORPUS uniquely combines
natural language rationales with questions from the original training set of MATH and GSM8k.
tool usage. Note that TORA-CORPUS only employ
Methods
#Annotation
Tool
Interleaving
LLM Used
Source
RFT Yuan al.
2023
100k
X
LLaMA-2
GSM8K
Open-Platypus Lee ct al 2023
25k
X
GPT-
11 datascts with MATH
WizardMath (Lio et al. 2025
>96k
x
ChatGPT
MATH GSM8K
Lila (Mishra èr
134k
(PPTT)
20 datasets with MATH & GSM8K
Mathinstruct
2023
260k
((PoT)
GPT-4
14 datasets with MATH & GSM8K
TORA-CORPUS (ours)
16k
GPT-4
MATH GSM8K
Output Space Shaping For each question, TORA- A-CORPUS mostly demonstrates only one valid interactive tool-use trajectory, which may restrict a model's output space, rendering it inflexible in exploring plausible trajectories during testing. We therefore propose oulput space shaping in order to encourage the diversity of plausible reasoning steps and reduce improper tool-use behavior.
To explore diverse valid trajectories, we apply nucleus sampling to imitation learning models M to sample 64 trajectories per training question q, following the inference procedure in Section 2.2 We retain valid trajectories with correct answers and no tool-use emrrr.. As many samples duplicates, to further improve diversity and in an attempt to correct models' improper behavior, we seek to leverage invalid trajectories as well. We observe that trajectories with wrong answers mostly incorrect halfway (Li et al. (002)) and the preceding reasoning is still plausible; in other words, we can obtain valid trajectories by correcting the subsequent portions. Specifically, wrong trajectory 7, when written in text, can be represented as sequence of lines separated by line breaks, i.c., == mm where is the total number of lines in F. We enumerate possible preceding portions of wrong trajectories, i.e., F/: 5] 11...jj. and leverage a teacher model M to complete the subsequent steps with greedy decoding: 7 (-lg F|: ji) where we abuse the notation P.r denote the interactive tool use process following Section 2.2 Finally, corrected trajectories as well as valid trajectory samples will be used for model training, thereby shaping the output space.
In our experiments, we always use CodeLLaMMA334R trained on ToRA-COM CORPUS the teacher model, and apply sampling with the CodeLLaMA series (ranging from 7B to 34B, with imitation learning). We obtain total of 233k distinct valid trajectory samples and 69k corrected ones. From this combined dataset, we randomly select up to 4 trajectories per GSM8k and MATH problem, merge them with ToRA-COR and then train all ToRA models on the resulting 69k annotations.
3 EXPERIMENTS
3.1 IMPLEMENTATION DETAILS
We fine-tuned LLaMA-2 Touvron el al. 2023b) and CodeLLaMA (Rozière et al. 2023) series (ranging from 7B to 70B) using TORA-CORPUS with output space shaping, yielding the ToRA and ToRA-CODE series respectively. We used a learning rate of 2c-5 by default except that we used le-5 for the 34B and 70B models. We set the global batch size to 128 and used linear scheduler with 3% warm-up period for 3 epochs. We trained all models with DeepSpeed ZeRO Stage3 (Rajbhandari al. 2021) and Flash-Attention 2 (Daol (202)). We used greedy decoding for all results, with the maximum sequence length set to 2,048 and the maximum number of tool executions set to 3.
3.2 EVALUATION SETUP
Datasets We evaluated models on GSM8k (Cobbe et al. 2021 and MATH (Hendrycks et al. 2021
along with 8 out-of-distribution datasets, namely GSM- (Gao et al. 2022). SVAMP (Patel et al. 2021), ASDIV (Miao et al.) 2020). TabMWP (Lu et al. 2023), SingleBQ. SingleOP, AddSub, and AritiArith Kkoncel-Kedziorske et al. 2016), as illustrated in Table 5 in Appendix. The 10 assorted datasets collectively encompass mathematical problems spanning basic arithmetic to competition level, covering middle and high school curricula and various mathematical domains. The problem formats comprise tabular- based, free-form, and multiple-choice questions, ensuring a thorough assessment of the model's mathematical reasoning aptitude.
5

---

Published as a conference paper at ICLR 2024
Table Results on 10 mathematical reasoning tasks. MAWPS results are averaged over four tasks: Singleeq, Singleop, Addsub, and MultArith. Vanilla models are tested with CoT. The best results in each section are in blue the second-best results are underlined, while the results of our best model are bolded.
ZS: shot inference without demonstrations.
Model
Size
Tools zs
GSM8K MATH
GSM-Hard
SVAMP TabMWP
ASDiv
MAWPS
AVG
Used training?
Proprictary Models
GPT-4
x 92.0
42.5
64.7
93.1
67.1
91.3
97.6
78.3
GPT-4 (PAL)
X
94.2
51.8
77.6
94.8
95.9
92.6
97.7
86.4
CharGPT
X
80.8
35.5
83.0
72.3
ChaGGPT (PAL)
X
78.6
38.7
67.6
77.8
79.9
81.0
89.4
73.3
Claude-2
X 85.2
32.5
PaLM-2
540B
80.7
34.3
Open-Source Models
LLaMA-2
7B
X
13.3
4.1
7.8
38.0
31.1
50.7
60.9
29.4
LLaMA-2
7.2
47.4
a. RFT
7B
V
51.2
Platypus-2
X
14.4
5.4
8.6
36.7
26.5
47.9
58.4
28.3
WizurdMath
54.9
10.7
20.6
$7.3
38.1
59.1
73.7
44.9
CodeLLaMA (PAL)
7B
X
34.0
16.6
33.6
59.0
47.3
61.4
79.6
47.4
Toolfomer
29.4
40.4
44.0
40.1
73.9
88.8
TORA-COPE COCO
7B
72.6
44.6
56.0
70.4
$1.6
78.7
91.3
66.5 (+19)
LLaMA-2
13B
X
24.3
6.3
13.6
43.1
39.5
56.3
70.4
36.2
LLaMA-2 SFT
13B
22.3
46.3
35.8
75.0
42.6
LLaMA-2 RFT
13B
55.3
Platypus-2
WizanlMath
13B
63,9
14.0
28.4
64.3
46.7
65.8
79.7
51.8
MMA (PAL))
13B
X
39.9
19.9
39.0
62.4
59.5
65.3
86.0
53.1
ToRA
13B
72.7
43.0
57.3
72.9
47.2
77.2
91.3
65.9
TORA-CODE
13B
75.8
48.1
60.5
75.7
65.4
81.4
92.5
71.3 (+18)
LLaMA-1 RFT
34B
V
57.9
CadeLLaMA (PAL)
34B
53.3
23.9
49.4
71.0
72.4
91.5
60.7
TORA-CODE
34B
80.7
63.7
80.5
70.5
84.2
93.3
(+14)
LLaMA-2
70B
X
57.8
14.4
36.0
73.6
57.5
76.0
92.4
58.2
LLaMA-2 SFT
70B
69.3
14.9
39,0
64.0
53.0
71.3
84.8
56.6
aMA-2 2 RFT
70B
64.8
Platypus-2
70B
X
45.9
15.0
24.6
74.3
47.3
72.7
91.1
$3.0
WizardMath
70B
81.6
22.7
50.3
80.0
49.8
76.2
86.2
63.8
LLaMA-2 (PAL)
70B
55.2
18.3
74.6
71.9
60.3
ToRA
70B
82.7
86.8
Metrics We report accuracies of predicted answers. Following Lightman et al. 2023). we round numerical values and sympy for parsing expressions. Since the SingleEQ. SingleOP, AddSub, and MultiArith datasets focus on different aspects of basic arithmetic, we report their average results under the collective term MAWPS KKoncel-Kedziossi et al. 2016) for all methods.
3.3 BASELINES
Proprietary Models We present results from an arra SoTA LLMs, such as OpenAI's GPT-4, ChatGPT (gpt-3. (.pttg..-ttuut.. 5-turbo), Google's PaLM-2, and Anthropic's 's Claude-2. By default, we report CoT prompting results, and include PAI. [Gao et al. 2022) prompting results for selected models.
Open-Source Models Base models comprise LLaMA-2 and CodeLLaMA with CoT and PAL prompting. Supervised Fine-Tuning (SFT) employs CoT rationales from the original GSM8k and MATH dataset (15k samples) for fine-tuning. Rejection sampling Fine-Tuning (RFT) leverages multiple models to generate diverse reasoning paths for fine-tuning (Yuan et al.) (023). WizardMath augments data using ChatGPT, and conducts SFT and RLHF. Platypus-2, the top model the LLM Leaderboard is fine-tuned with Open-Platype reasoning datasets (Lee et al. 2023). We also compare ToRA with Toolformer (Schick et al. 2023) which is model trained utilize calculators.
https::/www.swn http:///o...e///oggggaaaaa//a//a 1tt.1mmm_eedarrbrrrr
https://huggingffae c

---

Published as a conference paper at ICLR 2024
Table 3: Results on MATH subtopics.
Intermediate Algebra
Number Theory
Counting Probability
Model
Size
Tool
Precalculus
Geometry
Prealgebra
Algebra
Overall
Proprictary Models
ChaGGP (PAL)
18.5
19.2
23.2
48.5
43.0
62.7
45.4
38.7
GPT- (PAL)
32.8
29.3
38.0
58.7
61.0
73,9
59.1
51.8
Open-Source Modela
WizardMath
78
x
6.2
6.0
6.5
7.6
9.5
18.1
16.3
11.2
TORA-CODE
7B
35.1 (+28.9)
31.0 (+25,0)
24.0 (+1..5)
50.7 (+1.1)
30,6 (+21.1)
55.0 (+36.9)
61.7 (+4544
44.6 (+33,4)
wro Shaping
29.7
(-6.3)
(338)
(o1.7)
(40)
40.2 (-4.4)
w/o Rationale
7B
25.5
14.7 (-16.)
15.4 (8.6)
(-.9)
(.0.)
51.0 (-4.)
52.4
36.8 (-7.)
WizardMath
11.5
11.0
28.5
21.1
15.0
TORA-CODE
13B
35.7 (+29.3)
311 (+24.)
25.7 (+142)
55.6 (446.0)
39.5 (+28.5)
58.7 (+30.2)
66.7
(+45.0)
48.1 (+3.0)
w/o Shaping
13B
32.8 (-2.9)
(cS.))
52.6 (-.0)
(-1.1-
(-.6))
61.2 (-5.5)
(43.5
w/o Rationale
138
27.1 (-8.6)
15.8 (-1.3)
(-949
(-..1)
36.9 (-2.6)
(634))
56.5 (-1.2)
40.2 (-7.9)
TORA-Cos
34B
/
38,9
34.6
27.3
$7.8
41.4
63.7
67.7
50.8
w/o Shaping
(-.99
29.9 (-4.))
24.6 (-2.7)
(-2.2)
(+0.2)
63.8 (+0.1)
(-6.3
41.4 (:3.4
w/o Rationale
34B
28.3 (-10.6)
15.8 (-18.8)
18.0 (-9.3)
(2.4)
40.7 (-0.7)
58.6 (-5.1)
57.5 (-102)
41. (88.)
WizardMath
19.2
35.0
ToRA
70B
37.1 (+28)
(+17)
30.1 (+13.2)
54.6 (+38.1)
40.3 (+21.1)
64.9 (+22.2)
66.6 (+316)
49.7 (+25.6)
wro Shaping
70B
33.8(-3.3)
28.9(-1.5)
27.1(-3)
53.01-1.6)
38 38.01-2.3)
62.26-2.7)
64.21-2.4)
47.36-244)
w/o Rationale
70B
26.76-10.4)
14.76-15.7)
20.36.9.8
48.965.7)
39.2(-111
59.8(-5.1)
$7.6699
41.6.82)
3.4
MAIN RESULTS
Table presents the results of TORA on 10 mathematical datasets, highlighting the following salient observations: (1) Using interleaved formatting and output space shaping. ToRA consistently surpasses prior state-of-the-ar open-source models across scales, achieving 13% to 19% absolute improvements across 10 tasks. (2) T-RA-70B substantially outperforms ChatGPT with both CoT and PAL prompting on GSMSk (84.3% vs. 80.4%) and MATH (49.7% vs. 38.7%), while T-RA- CODE-34B is competitive with GPT-4 solving competition-level MATH dataset with code (50.8% vs. 51.8%).
(3) The accuracy of ToR A-CODE is about 5% higher than TORA A of the same size, demonstrating that continued training on code data significantly benefits program- basedd tool use. fine-tuning negatively affects out-of- displays superior generalization. For instance, WizardMath-700. underperforms the base model on TabMWP (49.8% vs. 57.5%), while TORA-708 A-70B effectively generalizes to this tabular reasoning task (74.0%). (5) ToRA attains fast zero-shot inference speed, averaging 1.02 tool interaction rounds per problem, while effectively addressing problems that require interactive tool utilization.
(4) While rationale-basee
generalization, TORA
3.5
ABLATION STUDY
Rationale-only
Program-only
Tool-integrated Reasoning
61.6
50
42.5
40
37.5
Accuracy
33.6
30
27.8
20
14.9
7.2
LLaMA-2-7B
LLaMA-2-138
LLaMA-2-708
GPT-4
Figure 4: Comparison of three formats: (1) Rationale-only: step-by-step natural language reasoning
like CoT: (2) Program-only; solving problems with programs like PAL; (3) Tool-integrated Reasoning used by ToRA: interweaving rationale and program execution to solve problems. We evaluated GPT-4 with few-shot prompting. We trained LLaMA-2 models to reason in the three types of formats, respectively. For fair comparison, do not apply output space shaping for all LLaMA-2 models.
7

---

Published as a conference paper at ICLR 2024
3.5.1 COMPARISONS OF FORMATTING
To evaluate the efficacy the reasoning format adopted by ToRA which interleaves rationales with programs, we compared it with Ratlonale-only and Program-only formats using GPT-4 and LLaMA-2 trained with the same size of data from MATH. As shown Fig the ToRA method consistently surpasses Rationale-only and Program-only approaches. Remarkably, using LLaMA-2, the TORA method achieves substantial improvements of 29.0% and 6.7% over Rationale-only and Program-only, respectively. With the closed-soure GPT-4, the improvements are 19.1 % and 9.8%, respectively. This emphasizes the effectiveness of integrating natural language rationales with programs.
3.5.2
EFFECTS OF OUTPUT SPACE SHAPING
ToRA -Sapping Correction
TaRA- Correction
ToRA
GSMBk
MATH
80
48.1
Accuracy (80)
74.9
75.8
44.6
44.6
44.6
46.7
72.6
73.5
Accuracy
71.1
40.2
70
68.1
65
35
7B
13B
7B
13B
Correction is trained on TORA-CORPUS without shaping. (2) ToRA-Comectin employs only the sampling strategy for shaping, trained with up to 4 additional valid trajectory samples per problem. (3) ToRA utilizes both the sampling and correction, also trained with up to 4 additional trajectories per problem.
Figure 5: Ablation output space shaping strategies using CodeLLaMA: (1) ToRA
We assess the effectiveness of the output space shaping strategies presented Section 2.3 specifically
sampling and correction. As shown in Fig [ and Table 3 (1) Output space shaping yields considerable average improvement of 3.4% and 4.0% absolute for GSM8k and MATH, respectively, with greater benefits for smaller models; (2) Applying the sampling strategy results in a 2.7% absolute improvement on average, while additionally incorporating correction offers a more substantial boost of up to 4.5%, without using more training data; (3) Output space shaping benefits even the largest model T-RA-70B, with a notable improvement from 47.3% to 49.7% on MATH. These findings highlight the effectiveness of our shaping strategies across different model sizes and datasets.
3.6 ANALYSIS
We investigate the benefits, detailed patterns, and remaining challenges of tool interaction for mathematical reasoning on the challenging MATH dataset. Performance breakdowns on all subtopics of MATH are reported Table 3
Benefits from Tool-Integration for MATH Sub-topics As shown in Table 3 ToRA outperforms WizardMath by around 45% in Algebra and Number Theory, which is attributed stimulating and shaping tool-use behavior. Problems from the two sub-topics typically need intricate computation and data manipulation. Algebra mainly focuses on solving equations and application problems, while many Number Theory problems can be tackled using brute-fore approaches through code.
Patterns of Library Usage for Problem Solving presents the most frequently used libraries for different sub-topics and the corresponding accuracies of their solutions. Tool-use behavior on different mathematical areas demonstrates distinct patterns. sympy and its internal solves vers are primarily employed for algebra-related topics. Precalculus exhibits extensive matrix operations via matrices, resulting in a high accuracy. Number Theory depends on algori thms like god and 1cm. Geometry mainly the application of other tools limited, signifying the potential for improvement.
library for Fraction-basee computations, while the
Detailed Impact of Rationale Different Topics Table 3 shows that using interleaved format, in contrast to merely writing the program, leads to significant improvements across all subtopics, especially in Precalculus, Algebra, and Geometry, where notable increases range from 8.6% to 18.8%
8

---

Published as a conference paper at ICLR 2024
Library Usage Frequency for Each Topic
Library Usage Accuracy for Each Topic
100
sympy
matraces
sympy
matrices
80
solvers
binomial
80
solvers
minomial
Frequency (%)
cational talculus
algerithm
Accuracy (%)
rational
algorithm
caleulus
40
40
20
20
Int Alg.
PreCail
Gonmetry
Num. Th.
C&P
Pre.sg
Algebma
Ovemal
Int. Ali
ProCall
Gounetry
Num. Th.
C&R
PreAig
Alpehr
Overall
Figure 6: Library usage frequency and accuracy on each sub-topic of MATH.
Appendix F.1 provides representative examples demonstrating how the rationale aids in planning, multi-round self-correction, and finalizing answers.
Table 4: The failure modes of the ToRA on MATH, and their corresponding percentages in random samples analyzed by humans. We include specific examples of each failure mode in Appendix F
Error Type
Definition
%
Examples
Reasoning Error
Mistakes due incorrect reasoning steps or missing conditions.
38%
Ex.
Hallucination
Fabrication numbers or answers.
5%
Ex.
Diagram Understanding
Misinterprelation of the input diagram. Incorrect use external tools, especially when
21%
Inappropriate Tool Use
10%
Ex.
the problem can't be solved directly with libraries.
Syntax Error
Persistent syntax errors despite multiple correction attempts.
9%
Ex.
Runtime Error
Errors during program execution, unresolved by retrying.
9%
Rationale-only Error
Cannot be formalized into program and the rationale is incorrect.
3%
Ex.
False Negative
Correct answers that don't 't fully match the ground truth.
5%
Ex.
Remaining Challenges in Mathematical Reasoning for ToRA To better understand the failure
modes and remaining challenges, we manually annotated 100 randomly selected trajectories from the MATH test set, identifying and categorizing their failure modes. The results are shown in Table 4 Primarily, incorrect reasoning steps constitute the primary source of errors for ToRA complex math reasoning tasks (38%), ), with some hallucination issues also evident during problem interpretation and answer finalization (5%). Secondly, the misinterpretation of input diagrams contributes significantly to the error rate (21%). This is particularly noticeable in Geometry, Precalculus, and Intermediate Algebra. The diagrams in the MATH dataset are usually detailed in text using the Asymptote language (Hendrycks et al., (222T). thus making it challenging for ToRA to comprehend diagrams purely from textual descriptions. Thirdly, issues with tool usage include Inappropriate Tool Usage (10%). Syntax Error (9%), and Runtime Error (9%). These problems frequently arise when ToRA fails to use tools correctly after several corrections or attempts. There are certain inputs that fail to formalize well as programs (3%), which require abstract reasoning rather than computation. Finally, we also found that there are false negatives when using automatic indicators, i.e., correct predictions that are misjudged as wrong, but the proportion is relatively small (5%).
CONCLUSION
This paper presents ToRA, a series of novel Tool-integrated Reasoning Agents that synergistically combines natural language rationale with program- based tool-use for mathematical problem solving. Our approach demonstrates the potential of integrating external tools in the reasoning process, enabling language models to effectively tackle complex quantitative tasks. ToRA achieves state- of-the-art performance on 10 diverse mathematical reasoning tasks, substantially outperforming existing rationale- based and program- based approaches. Furthermore, our systematic analysis of the benefits and remaining challenges of tool interaction provides valuable insights for future research, contributing to the development of more advanced and versatile reasoning agents.

---

Published as a conference paper at ICLR 2024
AUTHOR CONTRIBUTIONS
Zhibin Gou proposed the interleaved tool-use format of TORA and curated TORA-CORPUL dataset, implemented the training and evaluation pipeline, conducted experiments and analysis on all datasets, implemented baselines, and was a main contributor the paper writing. Zhihong Shao proposed the project, conducted preliminary experiments, proposed and implemented the training and evaluation pipelines, proposed and trained all TORA models with output space shaping as well as ToRA variants in the ablation study, designed and oversaw experimental analysis, and contributed to many parts of the paper writing. Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen provided research mentorship, oversaw project coordination, and advised and contributed to many parts of the writing.
ACKNOWLEDGMENTS
Zhibin Gou and Yujiu Yang were supported by the National Natural Science Foundation of China (Grant No. U1903213) and the Shenzhen Science and Technology Program Zhihong Shao and Minlie Huang were supported by the NSFC projects (Key project with No. 61936010) ), and were also supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604).
0GGG20220331100000000
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm technical report. arXiv preprint (rXiv:2305.10400 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning. pp. 2206 2206-2240 PMLR, 2022.
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303. 12712, 2023. doi: 10.485500ar2x,,2233..2222 12712. URL
48550/arXive 1271
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizii. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541, 2006.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
abs / 2110.14168
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. 2023.
Edward A Feigenbaum, Julian Feldman, al. Computers and thought, volume 7. New York McGraw-Hill, 1963.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller lan- guage models towards multi-step reasoning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Confer- ence on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10421-10430. PMLR, 2023. URL https://proceednng mlr /ttprssss///////// wt23/d
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program- language models. arXiv preprint arXiv:2211.104: 2022.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 3r5.1:2305.11738 2023.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 2015.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11852-14882 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 18653/633vvv 1/2023.aalllongg.... longloong.300 URL https: https////aaaatntt.....oooo
2023 a.lacl- long 830
11

---

Published as a conference paper at ICLR 2024
Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533, 2014.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. CoRR, abs/2210.116100 2022. doi: 10.48350/rXXiv22100 11610. URL htt.s: doi www.oorg/ooo.. 41w.://01.880///// arxiv 2222221111..1
Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James T Kwok. Backward reasoning in large language models for verification. arXiv preprint 2xXiv23308007755 2023.
Rik Kedize--Keziioskii, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.
1152-11555 1157, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-111 1136. / htr.o/anooooooolooo//o///---1110
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of Ilms. arXiv preprint w:Xiv:2308.07317, 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang. Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 5315-5333, 2023.
Tian Liang. Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang. Yujuu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19/18, 18, 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ily Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. The Eleventh International Conference on Learning Representations,
2023. URL htttp:://///eeeeeeee htwp:/iwwwwee..trreeeeeeee htt.:eeeeoorruaa..oooy...er..ee
Haipeng Luo. Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruet. arXiv preprint XXXx:2308.005833 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07788
2023.
Shen-yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, Online, July 2020. Association for Computational Linguistics. doi: 10. 12..1666/1/1222..allll 20200 acl-mann. 92
URL https: //t http:///aaaaaanoooo...o.o.o /rg/
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In Proceedingy of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 2021.
12

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report, 2023.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.000. 2023.
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205. /2255, 2022.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2090-2094,, Online, June 2021. Association for Computational Linguistics. doi: 10. (4.18633/v/200/ -naacl-main 168.
//acl
/acl onthollogy.org2202.. naac -aaac--ma in. 1 168
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ 16, 2023. URL /ettts://arxtie
/aagg//bbsss / 6200. 01116
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 2023.
Stanislas Polu and Iyy Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009. 033393, 2020.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-14, 2021.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint 2xXiv:3302.0476: 2023.
Zhihong Shao, Fei Huang, and Minlie Huang. Chaining simultaneous thoughts for numerical reasoning. In Yoav Goldberg. Zorntsaa Kozareva, and Yue Zhang (eds.), Findings of the As- sociation for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2533-2547,, Association for Computational Linguistics, 2022. doi: (in.:.3//1//20222.nnnnnn..g. 18..pig................. URL https: :/tp:::///o.... org/ 10. 10e1318611122222 Fircings-ennlp...
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Synthetic prompting: Generating chain-of- chain-of-thought demonstrations for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Нawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 30706-30775. ppwpss/e02//s//eeeeeeeeeeee
PMLR, 2023a. https::///pcoeeeee
mlr. www.prrsssss.... v202/ 22ao222//// html
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmenner large language models with iterative retrieeallgennraation synergy. CoRR, abs/2305. 15294, 2023b. doi: 10.485500/rXixx23306 15294. URL httpp:://oi...oo org//.0..ogg100 48550/arXiv 2305. 15294
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovinn Ilama model.
hattps://githin
cwm/tat laah-lab/aae sttsssanoorrr.ooo alpaca] 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Ilama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 09288, 2023b. doi: 10.485500arXiw.237000222 URL htttps://oii.or 10. 18550/arXivv 2 2307 09288
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https: http://pppneeeeeeeeeeeeeeee///ep http:/iwwwweeeeeeeeeeeem few.roooiiw.eeeeoouuu forum? id= <d=_violMeSB_L
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023, URL htttp:::/ppenreviiiieeee/ nwt.:/w..eeeeeeeeeeeeeeeeeeeeeee forrmm?
id=WE_ vluYUL-X
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308. 01225, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint NXii:2300.006633 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 155:16676-15888 2022.
Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. Evaluating and improving auggenede craputatiionintensiin math reasoning. arXiv preprint arXiv:2306.02444 02408. 2023.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi. Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308. 07921, 2023a.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 4471-4485, 4485, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 11653//1/2023..... lonn long
1: Long Papers), pp. 4471
URL https http://///aaaatoooooooooo.o
200 3023.acl-lona 2255 245.
14

---

Published as a conference paper at ICLR 2024
CONTENTS
Related Works
15
Evaluation Datasets
15
Additional Experiments and Analysis
17
Accuracies of Closed-Source Models on MATH
17
Effects of # Valid Trajectories for Output Space Shaping
17
C,3 Impact of Output Space Shaping in Relation to Question Difficulty
17
D Detailed Information of TORA-CORPE
18
Prompts
20
Examples
22
Success Cases
22
F2
Failure Cases
24
A
RELATED WORKS
Mathematical Reasoning Recent research has greatly improved reasoning in LLMs with step-by-step
natural language reasoning (Polu & Sutskever 2020 Wei et al. 2022 Zhou al. 2023b Zhu et al.)
2023 Huang et al.] 2022 Liang et al. (023). However, natural language reasoning struggles with
complex computations and symbolic manipulations. To overcome the limitations, recent research has exploited tools like calculators (Cobbe et al. 2021 Shao et al. 2022), code interpreters (Mishra et al.
2022). and symbolic solvers (Zhang al. (202). Program- based methods (Gao al., 2022 Chen et al. 2022 Shao et al.) (2033)) transform reasoning tasks into program synthesis tasks, thus offering complementary advantages over natural language reasoning. but they face challenges in nuanced reasoning, planning, and error handling (Gou et al. 2023). where natural language reasoning should be more suitable.
Tool-Augmented Language Models Augmenting LLMs with tools can largely alleviate LLMs limitations and improve reasoning and generation performance (Parisi et al.) 2022 Mialon al.
2023
Yao et al. 2023), Recent work demonstrates the benefits of integrating retrievers (Borgeaud
et al.
2022 Shao et al. 2023b), search engines (Nakano et al. (022),, and multi-tool approaches
(Schick et al.
2023 Paranjape al. 2023 Gou et al. 2023) to improve generation.
Knowledge Distillation Knowledge distillation (KD) transfers knowledge from teacher models to student models (Buciluâ et al., 2006
Hinton al. 2015). Using L.L--generrted trajectories for
fine-tuning is form of KD (Fu et al. 2023 Taori et al.) 2023 Peng et al.) 2023] Ho et al.) (023).
Our proposed ToRA shows that learning interactive tool-use trajectories promising direction adapt language models to reasoning tasks.
B EVALUATION DATASETS
We present statistics and examples of the ten evaluation datasets in Table
15

---

Published as a conference paper at ICLR 2024
Table 5: Statistics and examples of the 10 evaluation datasets. In the main result table, we present the average accuracy of SingleEq, SinglcOp, AddSub, and MultiArith under the collective name MAWPS.
Dataset
OOD? #Samples Example Problem
The ice cream parlor was offering deal, buy 2 scoops of ice cream, get 1 scoop free. Each scoop cost $1.50. If Erin had $6.00, how many scoops of ice cream should she buy?
GSM8K
Cobbe
IND
1319
et al. 2021
For constant e, cylindrical coordinates (r, 0),, find the shape described by the equation
MATH (Hendrycks
IND
5000
et al. 202
2 C.
(A) Line (B) Circle (C) Plane (D) Sphere (H) Cylinder (F) Cone. Enter the letter of the correct option.
Jean has 30 lollipops. Jean cats 8714250 of the lol- lipops. With the remaining lollipops, Jean wants pack- age 871 8714250 lollipops in one bag. How many bags can Jean fill?
GSM-Hard
Gao
OOD
1319
et al. 2022
During summer break 819058 kids from Lawrence county go to camp and the other 668278 kids stay home. How many more kids spent their summer break at the camp compared to those who stayed home?
SVAMP
Patel
OOD
1000
ct al. 2021
Mrs. Hilt saw iPod for sale. The price tag said the iPod cost $128, but sign announced that was on sale for "35% off." How much would the iPod cost after the discount?
ASDiv
Miao et al.
OOD
2215
20201
Stem
Leaf
Read the table regard- ing "eight lifting results (lbs)". Mr. Morrison, P.E. teacher, wrote down how much weight each of his students could lift. How many people lifted at least 28 pounds?
2
3, 6, 7, 8, 8 0,7,9
3
TabMWP (Lu et al.
OOD
1000
4
1,5
2023
6
3,
8,
3,4, 4. 7,
8
5,
SingleEq (Koncel- Kedziorski et al., OOD
Alyssa's dog had puppies. She gave 7 to her friends. She now has puppies left. How many puppies did she have to start with?
508
2016)
SingleOp (Koncel. Kedziorski et al. OOD 562
Rachel removes 47 bottle caps from jar. There were originally 87 bottle caps in the jar. How many bottle caps are left in the jar?
2016)
AddSub (Konell Kedziorski et al.
Sam went to 14 football games this year. He went to 29 games last year. How many football games did Sam go to in all?
0OD
395
2016)
MultArith [Koncel Kedziorski et al.
Paige had 43 math problems and 12 science problems for homework. If she finished 44 the problems at school, how many problems did she have do for homework?
0OD
600
2016)
16

---

Published as a conference paper at ICLR 2024
Table Accuracies of ChatGPT and GPT-4 on the MATH dataset, with breakdown w.r.t different mathematical subjects. We apply PAL prompting and the Tool-integrated Reasoning method used by ToRA the two dlosed-soure models.
Intermediate Algebra
Number Theory
Counting Probability
Model
Tool
Precalculus Geometry
Prealgehra
Algehra
Overall
CharPP (PAL)
18.5
19.2
23.2
48.5
43.0
62.7
45.4
38.7
GPT-4 (PAL)
29.3
73.9
GPT-4 Tool-integrated Reasoning)
40.0
37.2
44.1
68.9
67.3
82.2
75.8
61.5
Training Set
GPT4 (Tool-intograted Reasoning)
$1.0
51.5
42.5
77.4
72.2
89.8
85.1
64.3
w/ best@l Lo
72.9
70.0
58.9
91.6
81.7
95.5
96.3
83.1
C ADDITIONAL EXPERIMENTS AND ANALYSIS
C.1 ACCURACIES OF CLOSED-SSURCC. MODELS ON MATH
Table 6 presents the detailed accuracies of GPT-4 on the MATH dataset. The Tool-integrated Reasoming method used by ToRA significantly outperforms PAL prompting when directly applied to the closed-sourge GPT-4, further demonstrating the benefits of synergizing natural language reasoning and program-base based tool use.
C.2 EFFECTS OF # VALID TRAJECTORIES FOR OUTPUT SPACE SHAPING
GSM8k
MATH
50
13B
13B
76
75.4
48
Accuracy (96)
74
Accuracy (96)
48
45.1
45.8
72.6
44,6
44.6
72
46
43.4
41.8
69.4
68.1
40.2
2
2
#Trajectories for Shaping
#Trajectories for Shaping
Figure 7: Effects of using different numbers of additional valid trajectories per question for output space shaping.
As shown in Fig 7 it is beneficial to increase the number of additional valid trajectories for output space shaping.
C.3 IMPACT OF OUTPUT SPACE SHAPING IN RELATION TO QUESTION DIFFICULTY
We compare the effects of output space shaping on MATH problems of different difficulty levels (from level level 5) in Figure g and present the statisties of MATH problems different levels in Table 7
As can be seen:
Across these different difficulty levels and model sizes, output space shaping generally brings significant improvement of 4.0% on average across different model sizes.
Output space shaping brings significant improvements for difficult, long problems. E.g.. with TORA-CODE-IB 13B, shaping does not significantly improve level I to level problems, but it brings substantial improvement of 5.4% to 5.7% for level 3 to level problems.
17

---

Published as a conference paper at ICLR 2024
Performance Comparison Different MATH Levels
12.8
ToRA-7B
+3.2
0.2
ToRA-7B
Shaping
ToRA-138
ToRA-138
Shaping
70
ToRA-348
ToRA-34B
Shaping
GPT-4 PAL
60
+.6
Accuracy
13.
30
+ 2.9
20
Level
Level 2
Level 3 Levels
Level 4
Level 5
Figure 8: Impact of Output Space Shaping in Relation to Question Difficulty.
After using shaping. TORA-CODE-34B outperforms GPT-4 PAL on problems from Level Level but there still gap Level (27.3% vs. 30.0%). These problems usually longer (average about 248.4 448.4 characters), require more reasoning steps (>1,000 characters) to solve, and more often include diagram inputs (about 20%) These observations may guide future work to focus more on solving these more difficult problems.
Table 7: Statistics of MATH problems at different levels. Average Answer Length indicates the average length of ToRA outputs; Training query coverage indicates the proportion of queries with at least one valid trajectory in TORA-CORPUS relative to the total queries in the original dataset.
Level I
Level
Level 3
Level
Level 5
# Test Samples
437
894
1131
1214
1324
Avg Question Length
123.8
150.9
169.1
203.0
248.4
Avg Answer Length
503.1
655.8
751.2
881.6
1083.8
Training query coverage
97.7%
91.6%
86.5%
81.3%
68.0%
D DETAILED INFORMATION OF TORA-CORPHH
We provide more detailed introduction to the data construction process, quality control, and data statistical information, beyond Sec. 2.2
Data Format and Quality Control In our preliminary experiments, we found that the tool- integrated reasoning trajectory format generated by zero-shot prompting was somewhat chaotic. Therefore, we designed few-shot prompting control the reasoning format, which effectively improved data quality. On the other hand, we increased the annotation success rate by sampling, ensuring comprehensive coverage of the training query.
Data Filtering Process
For the data constructed, we filtered out paths that produced incorrect answers by matching them with standard answers. To prevent the model from learning incorrect
18

---

Published as a conference paper at ICLR 2024
Table Accuracy of TORA-CORPUL on GSM8K and MATH training set. TORA-CORPUS-Greatt uses only the greedy trajectories, while ToRA-Corpus- 16k combines sampled trajectories.
GSM8k
MATH
Intermediate Algebra
Number Theory
Counting & Probability
All
All
Precalculus
Geometry
Prealgebra
Algebra
TORA-CORPUS C
94.4
64.3
51.0
$1.5
70.0
77.4
72.2 81.7
89.8
85.1
TORA TORA-CORPUS-IT CORPUS-16k
98.2
83.1
72.9
70.0
58.9
91.6
95.5
96.3
Table 9: Statistics of TORA-CORPUS-166
GSM8k
MATH
Total
# Train Samples
7,657
7,881
15,538
Avg Question Length
236
189
211
Avg Trajectory Length
678
704
691
Min Trajectory Length
218
119
119
Max Trajectory Length
1,713
2,486
2,486
intermediate reasoning processes, further filtered out data samples with intermediate program exccution errors.
Dataset Statisties In Table compared the annotation accuracy (i.e., sample coverage) of the training set on GSM8k, MATH, and MATH subtopics of TORA-CORPP Greedyy using only the greedy trajectories, and ToRA- CERRA-CORPU--OC combined with sampled trajectories. Furthermore, in we reported the statistical data of TORA-CORPUS
Table 9 average question length, average, minimum, and maximum trajectory length, as shown in the following tables.
such as the number of samples,
As described in Section 2.2 we annotated interactive tool-use trajectories for the training questions from MATH with GPT GPT-4 achieves success rate below 65% using greedy decoding. As MATH was originally annotated with natural language rationales, to improve the annotation success rate, tried provide GPT-4 with the human rationales hints (Zelikman ct al. 2022). However, when using this method, GPT-4 tends to replicate the hints and ignore tool-use outputs especially when the outputs are inconsistent with the hints, thus failing to produce high-quality trajectories. Hence, we deferred the utilization of the already-annotater natural language rationales for future investigations. Instead, we employed nucleus sampling recall valid trajectories for questions that remained unsolved through greedy decoding. This approach significantly boosted annotation accuracy to 83.1 %.
Rationale as Hints
19

---

Published as a conference paper at ICLR 2024
E РROMPTS
We present instructions and example few-shot prompts of Tool-integrated Reasoning for querying GPT-4.
Listing 1: Prompts of Tool-integrated Reasoning for GPT-4.
Integrate step-by-step reasoning and Python code to sclve math problems using the following guidelinea:
Analyze the question and write functions to solve the problem; the function should not take any arguments.
Present the final result in LaTeX using "boxee! 1 without units. Utilize the 'pi' symbol and "Rationsl" from Sympy for $\piš and fractions, and simplify all fractions and square roots without converting them to decimal values.
Here are scme examples you may refer to:
Question:
Convert the point (0, -3 \3qrt(3), 3) in rectangular coordinates spherical coordinates. Enter your answer in the form $ (irho, \theta, Aphi), , where $\rho > 0,8 sc Me \theta < \pi, and $0 \le \phi \le Api.$
Solution:
To convert the point (x, Y, z)$ in rectanqular coordinates spherical coordinates ((rro, \theta, \phi) we use the following formulas:
1. $irho = \eqrt + y 2*2
2. $\theta faretan/\fraclyy {x}}
3. $1phi
|arccos(\fract (Arhe))
Let's write a function:
"lyyhon
from sympy import sqrt, atan2, accs, pi
def pectangular_to_sphrriaa ():
x, y, 2 - 0, -3+sqrt (3), 3 rho sqrt (x**2 yau2 2**2)
theta atan2 (y, phi acos (z/rho) return rho, theta, phi
spherical rectangular print (pheepecccccaaaooorddiatttt ...
ootput (6, -pi/2, pi/3)
The required form of the answer is $(rro,, \theta, \phi) where $\rho 08, $0 Ale \theta < 2lpis, and $0 Ale \phi \ie \pis. Since our initial output has sitheta -lpi/2s, which doean't mest the condition for $ithetas, we add $2\pis adjust its value.
Rence, the spherical coordinates for the point ( -31sqrt(3), 3)$ are $ \boxed (6, \ \frac()pi)
Question:
Compute $1011_2 2 1100_2 + 11.1_2$. Express your answer in base 2.
20

---

Published as a conference paper at ICLR 2024
Solution:
Let's write a single python functicn for binary computation:
'python
def jinary_sum_diff(>
Convert binary to decimal
numl int ("1011", 2)
num2 int ("101", 21
num3 int ("1100", 2)
num4 int ("1101", 2)
Perform the operations result - numl + num2 - num3 + num4
Convert the result back binary result_binary format (result, "b")
return result_binas
result binary_sum_difff
print (result)
ooutput
10001
...
The answer expressed in base is $\boxed (bboxed[1000) (10001_215.
Question:
Find the matrix that corresponds to projecting ento the vector (tbegin(pmatrix) 2 A -3 (pmatrii)
Solution:
'python
from sympy import Matrix, simplify
def projection _
Matrix -3]) (v v.T) / (v.I v)
Simplify the elements = F.applyfunc (simplify)
return
result projection_matrixx) print (result) ...
ooupuut
Matrix(114/13, -6/131, 1-6/13, 9/13111
The matrix is (bboxed/ \frac(4)(13} & ()) (13) (1a) \frac[9) (mnd(iaitr
Question: Solve the incquality AAffrc(x'22 - (x + 5) < 0.1]
Solution:
Let's use 'sympy' to solve the inequality:
"Ppyhhon
from sympy import symbols, simplity
21

---

Published as a conference paper at ICLR 2024
from sympy. yympy.solverss www.:sooveers..eeaalttieiee. inequalities import solve univariate inequality
from sympy, www.y..ore.rllation... import StrictlessThan
def solve_inequality()
= symbcls('x'))
expression (x**2 25) /
inequality StriatlessThan (expression, 0)
solution wtwe__univrriate_ineuueee (inequality, x, relational-False
simplitied_solution simplity (sclution)
return simplified_solution
result solve inbuuuiiyyyylyty ()
print (result) 1.
'oupput
Union (Interval open (-o0, -5), Interval. open (-5, 5))
The is $\boxed (-Vinfty,-D) (cup
F EXAMPLES
F.1 SUCCESS CASES
Listing Success case for ToRA: Rationale for planning.
Question: What real values of $x$ are not in the domain of =x 11 1 !??
Solution:
The domain of the function is all real numbers, we need to find the values of for which the denominator of the function is zero.
The denominator of the function is the sum of the absolute values of the two terms inside the absolute value.
Each of these terms is quadratic in x, and the reots of a quadratic equation are real unless the equation is identically zero.
So, we need to find the values of for which the two quadratic equations are identically zero.
" 'python
from sympy import symbols, Abs, sclveset, from sympy.corr.rolation import Ne
def non_domain_values(( symbcls('x')
These the values for which the function is not defined rootsl solveset (Ne (x**2 6*x + 8, 0), x, domain=S. Reals) roots2 = solveset (Ne + 6, 0), x, domain=S.
The roots of quadratic equation are always in the domain of the function,
need to find the values of that not in the union of the roots
non_domain_vaaues S. Reael (rcotsl, union (roots2))
return nen_domain_valuge
result nea_domain values print (result)
output
(2)
22

---

Published as a conference paper at ICLR 2024
The real value of $x$ that is not the domain the function boxed(2)$.
Listing 3: Success case for ToRA: Self-Correcting Errors with tool feedback.
Question:
The difference between two perfect squares is 133. What is the smallest possible sun of the two perfect squares?
Solution:
We can solve this problem by noting that any perfect square can be written as (a + b) "2 or (a - b) ~2 for some integers $aş and $b$. the following problem: Find the b)* given that (a
So we can zewrite the question
minimum possible value for $ b) =2 1335.
'python
from sympy import symbols, Eq, sclve
def
***
Find the smallest possible sum of two perfect squares given that the difference between them is 133.
Returns:
Integer: The minimum possible sum of the two perfect squares.
a, symbols('a b²)
Equation for the difference of the squares
diff_eq Eq((a + b)++2 (a - b) a 133)
Solve the equation for b b_solutions foleffdiffeed b)
Calculate the sum of the squares for each solutien of b sum_squares p_so ++2
b_sol) for b_sol in
b_solutions]
Find the minimum possible sum of the squares min_sum_squares min (sum_squares)
return min sum_suuusaaae squares
result min_sum_perrfot__quurr
print (result)
Moupput
(a 133/ (4xa)) + (a + 133/ (4xa) **2
The problem seens be more complex than anticipated and the solution was incorrect. Since the difference between two squares can be factored, we can write:
$x 2 (x y) (x y
$133$ has two pairs of factors: 1 and 133, and 7 and 19.
So, either Sxty 133$ and Sx-y 1$, $xty 199 and $x-y 79.
For (133, 1), $x \frac(133 + 1) (2) = 67$ and Sy = \frac|133 11 665.
For (19, 7), $x \Frac(19 (2) 13$ and Sy \frac(19 7) (2) 65
23

---

Published as a conference paper at ICLR 2024
(13, result the smallest sum of the two perfect squares. so, s medify the previous solution:
ppython
def pet_sssmprreettssurrree ():
min_squares 113++2, 6++2]
san__sum_qquare sum (mii_ssuurrsss
return min_sum_squaree
oouput
205
...
The smallest possible sum of the two perfect squares is $ //boxed/2055
Listing 4: Success case for ToRA: Finalizing answer.
Question:
Compute MKsum_Ik Minfty \frac 2) (k + 11! (K + 2) 1).A
Solution:
'python
from sympy import symbols, s, Sum, factorial, simplify
def infinite_sume sum)) expr (k + 2) (factorial (k) factorial (k 1) ffctoriil (k 2)) sum jum_expr = (expr,, (k, 1, oc))
result = sun_expr.doit
return result
result infinite_sum()
print (result)
ooupuut
Sum((k 2) (factorial (k) factorial + 1) factorial (k + 2)), (k, 1,
o0))
The sum does not converge simple expression, so the answer boxed [k M)nlin \fracik + 2)ik! (k 11! + (k 2) !)1
F.2 FAILURE CASES
Listing 5: Failure case for ToRA: Reasoning Error.
Idx: 4289
Type: Preal gebra
level: Level 2
Question: Whhat is the smallest positive multiple of $13$ that is greater than $500?$*
Solution:
'pyython
def smmaleest_ppoiiive__utttbbll
"**What is the smallest positive multiple $135 that is greater than $500?$""*
Start from 500 and keep adding 13 until we get number that is greater than 500
500
whi 500:
24

---

Published as a conference paper at ICLR 2024
13 return num
result
print (result)
oouputt
513
The smallest positive multiple of $13$ that greater than $500s is 9 (5135S
Execution: None
Prediction: '513'
Ground Truth: "507'
Score: [False]
Reference CcT: 'Dividing $500$ by $13$ gives quoticnt $38$ with a remainder of $6$. In other words, \MSCC Aldiv 13=13 Wedot 38+6.VV]
So, the smallest positive multiple $138 that is greater than $500$
is MM133 ledot.391) (box)
Listing 6: Failure case for ToRA: Hallucination.
Idx: 2775
Type: Intermediate Algebra
Level: Level
Question: "Find the eighth term of the sequence $1440, 1716, 1848, idots$, whose terms are formed by multiplying the corresponding terms of two arithmetic sequences."
Solution:
'python
from sympy import symbols, Raticnal, simplify
def eighth_termi
"*"Find the eighth term of the sequence $1440, 1716, 1848, \idotss, terms formed by multiplying the corresponding terms two arithmetic sequences. """
al, 61 1440, 36
a2, 62 16, 2
Calculate the eighth term of each sequence terml al 7+d1
term2 = a2 7xd2
Multiply the terms result terml term2
return result
result eighth print (result)
oouput
50760 ..
The eighth term of the sequence is &booxed (bboxed557760)5 50
Execution: None
Prediction: *50760'
Ground Truth: '348'
Score: [False]
Reference CoI: 'If $a_n =an + b$ and $b + ds are two arithmetic sequences, then their term-wise product takes the form Waa_nb_n an+b) (cn+d) An Bn N.whrr SA, B, CS are constants.
25

---

Published as a conference paper at ICLR 2024
Therefore, sequence $x An 1 and $x_2 1848.$ Then we have the equations WWVVbegi aligned) 6= 1440, VIV 2+3+C6-1716, \111 42+234C6-1848. Wonnd (Vlgged/\/JSubraeee
CS has $x 1440, 9x
SC-1440$ from the sccond and third equations, have SA+B-2769 and $4A+23-408, $2A-B-204.$ Then \VAA = (2A+B) (A+3) = 204 276 -72, aland
276-A-348.$ we conclude that
the given sequence has the general formula \V[x_ -72n^2 348n 1440. Then the eighth term is MM/Vbbegnnaallggee x_7 &= -72 Vicaot
7-2 348 Meddt 7 1400 = 348). (aiinedigg
Listing 7: Failure case for TORA: Diagram Understanding. Navigating the complexities of input diagrams is key to solving many problems in Geometry, Precalculus, and Intermediate Algebra. This ideal process encompasses not just visual modalities, but also involves interactions with diagrams, such as adding auxiliary lines. Nonetheless, the utility of computational tools like SymPy is, unfortunately, only marginally beneficial.
Question: A square has side length of 10 inches. Congruent isosceles cut off each corner so that the resulting octagon has equal side lengths. How many inches are in the length of one side of the octagon? Express your answer as a decimal to the nearest hundredth.
right triangles
[asy]
size (150);
draw ( -0 0)
(0.7))) draw ( (5*sqrt (2)/ sqrt (2)),,0)-10 + sqrt (2) ), linewidth
(0.7));
(110 (5+sqrt (1)/ sqrt (21)), (2) (1 sqrt (2))), 1inewidth(0.7))
dr, (10-(5 (2) (2) (1+sqrt (5ept) (2) sqrt (2) 10),
1inewidth
draw (5+sqrt (1) sqrt (2)) 10100- 5*sqrt (2)
/ (1 sqrt (2)
}) 1),
[/asy]
Solution:
The sice length of the octagon is the hypotenuse of the right triangie formed by the side of the square and the diagonal of the octagon
The hypotenuse is the sum of the side length of the square and the length of the diagonal of the octagon
The length of the diagonal of the octagon is 5+sqrt (2) sq)t
'pyyhhnn
from sympy import sqrt, Rational
def octagen_side_ 10
octagon_sto side length square_sise see__sidd_________ length + 5vsqrt (2)/( + sqrt (2)
return round (octagon side Length, 2)
result octagon side 1ength print (result)
'oupput
12.93
The length of one side of the octagen is (.boxed(12.99) inches.
**Ground Truth**:
Recall that the ratio of the hypotenuse to the leg of an isosceles right triangle is $1sqrt Looking at the removed triangles, we see that the three segments that make up the side of square are (2)), Sas, and (s/\sqrt(2)9, where $a$ is the side length of the octagon.
26

---

Published as a conference paper at ICLR 2024
Setting the sum of these three side lengths equal 10 inches, find \begin(alignx)
\implies A
8-10 Aimplics V1
(1ssrrt(2)+1)===1 \implies \\
Nend(align=)) where we have raticnalized the denominator twice: AI
trac(\sgrt(21112) A and \I
(1)gr(()))) (tta:///eaaaaaa1aaaaaarraaaaaaa1 Acdot' (ract (ssgrt
(21-1]=10((sqrr(()>)))
M To the nearest hundredth, $s-\boxed 14)$ inches.
[asy]
(150))
real 5 (1)s (1))
draw (2) sqrtt (ttp::/(1111ap(00000000000001110 sqrt (2))))
draw
(10 (10
(2) =0rr((2)))0---1105ssss (2) + sqrt
draw (00000asssrrr(0//1taaaaaaa)))1))
(2)),
draw ((10 (2) +ss((()))))))))1111113 - (2) sgrt (2)
}));
label ("Sss", (10-a/ (2), [133000000||/|ssrr(1000000/11100
label [Ssjdaapoassyyle//rrrtt//s/////1
(10,10-a/ 1.0,00=/(22sqrr) (2)) E);
label (atpai/ppleeeeeeeeeaaaaaaaattlaa (10, (2+sqrt (2))) BB;
label (*$""", (10,5),
draw (rightanglemark (10, 0), (10, 10), (0, [/asy]
Listing 8: Failure case for ToRA: Inappropriate Tool Use.
Idx: 2489
Type: Intermediate Algeøra
Level: Level
Question: 'Let 1,$ $2 2, $2_3$ be complex numbers such that $1z_1
2, $12_2| 3, and 2331 4.$ Find the largest possible value of
ni\[ia_l 2_2 2 2_31-2 + 1 - "2.A1
Solution: "python
from sympy import symbols, simplify, Max from synpy.functions import Abs
def max_oxpression_valle (
"*"Let $2_1, $2_2, $2_3$ be cemplex numbers such that $12_11 2, |=_2 3, and 2_31 4.$ Find the largest possible value of Mi2_1 ~_ + |2_1 2_31-2 + 1z_2 - 2_31"2.\]**1
Define the variables
21, 22, 23 symbols('z1 z2 z3")
Define the expression
expr Abs (z1 - x2)++2 +*2 Abs (zl - 23) x+2 + Abs (z2 23) +*2
Define the conditions conditions [Abs (z1) - 2, Abs (22) 3, Abs (23) 4]
Find the maximum value of the expression under the conditions value (expr, sconditions)
simplify the maximum value Limplifiedmaxx_allu value = simplify (max_valuse
return value
27

---

Published as a conference paper at ICLR 2024
result max_expresssoon_allle print (result)
ooupuut
Max (Abs (z1) 2, Abs (z2) 3, Abs (23) 4, Abs 22) + Abs (21 23)
**2 + Abs (z2 23) x*2)
The largest possible value of $ 2_21-2 12_1 2_31-2 12_2 2_ 122 is $\boxed/25)s.
Execution: None
Prediction: '25'
Ground Truth: '87'
Score: [False]
Reference CeT: 'We have that (bagglgg \nz_ wtw.innnnnnnneiiieeeeee \1oeeriine/2]_ ==
2_11-2, W\looz2 \lover1neez)_22 6= 12_2 2, 1iillnz_3 1 Voverlinc(2)
_3 6= 1.2.::2....onttaa..gg.LLl...ss. http://wwww.....oooogoggggg///// |_
2_2 [^2 |2_1 2_3 2_2 2_31 \1llin== [2_1 2_2) (1loveriinn
2_1 z_2)) [z_] 2_3) (loverinnnn 2_31) (z_2 (232-233))1
overline(z_2- 2_3)) \illlns- (1)
ov)rline(z]_2) + + -2_:-233) [Nloverlineisi]]-
2_2 z_3) (Nloveriimeiz 3
2l_1 z_1 \loveriime \loerriineiij_3) i1llln 2_1 Vloverlinel
N 2.2 +22 Moverline(z)
+ 2 1loverline(z 211 \loverlineizi_3 \overline(z]_1 2_3
2_1 \\overline(2]_3- 2_2 \loverline(2]_3- 2_2 [toverline(2]_ \
overline 2_3 2_2 [ttellinne2__3ll1lme 212_11-2 2
212_31'2 [2_] \lover1ine(z)_2= [loverline 2_2 #_1 |1
3 [loverlinez)__1 z_3 z_2 [loverline(z]_3 11 overline(z)_2 z_3) (mllend(alg
Adding these two equations, we + 2_31
2
2_2 2_31 [2_1 2_2 2_31 3|2_11 312_21 + 312_3 312_3
|^2. T \n htg.:///ooo.ooooaaaooooo///o///.
[2_1 1211- -2_3122
2_2 - z_31 4- 4- : /2 3 +312_3122 1z_1 + 2_2 +
Villins\lle 3 Mcdot 2-2 + 3 Medoot 3-2 3 Medot 4^2 W1ln 87. In
\ Jen equal ity to occur, we must have $z_1 2_2 + 2_3 0.
Without loss of generality, we can assume that $z_ 2.$
[asy] \nunitsize cm) Inimpair zone, ztwo, zthree; ithrreyyalnzcne (15) /nnztrre (itzaa/ (-114 (15) \nnndraw (1,))e(())0),)),, Andrnn (t green) indraw ( (Circle((),)))) , (uccc(((00000..roeeeonncccccccdd
\nnnz = (2,0); Anztwo
2one, E)
Andot ("$z_2s" ztwo, N) ; ("$2_38", 2three, \0) nAlternative: For equality to occur, we must have + + + 0.$ Without loss of generality, we can assume that $2_1 2.$ Then iys so that 2 iy, where $xş and $ys are real numbers. We peed\n\
$z_2 z_3 -2.9 Let $z_2
12_2
1 = x + y-2 &= 9 AMMA 12_31-2 = (x + 2) + y &= 16. \n lend
|Subrracingg the first equation from the second, we get $4x 7,8 or $x Uldfrac34.$ One solution is $z_2 - Vidfrac34 + it! and $z_3 -V\dfrac(11]4 111dFrac(331sgqt (15) This example shows that equality is possible, so the maximum value is $\ (3xxxed(37)..3
Listing 9: Failure case for ToRA: Syntax Error.
Idx: 106
Type: Algebra
Level: Level 5
Question: 'Below is a portion of the graph of a function, $y=h (x) $: AnAn asyl \nimport graph; size (8cm); real 1sf=0.5; pen dps=1inewidth (0.7) fontsize (10); defaultpen (dps); pen ds=black; real xmin=-0.75, xmax 28.25,ymin=-1 yma. ymax=10.22 AnAnpen (0. 0. /n/+ grids/ pen gs=linewide (0.7) 1egepra
(hhhrrrdtttt.)))eeooogg11111111 ("2 2"); real gx-1, ,
28

---

Published as a conference paper at ICLR 2024
nfor (real
http:///haaai.a../.o//oooo//a//r draw ymin)
*gx;
-- ymax)) ,5); for (real httpr//w.........ooo.o.////////r
gy) draw ( (a-min,))--xnaaii (xmax,1), AnNaxaxis xmin,
above=true); http:///waaaaaaaaaaa..oo...aaaar
Arrows (6), (
f1 (real x) (return (x
* (xx77_5)/16xx) http://wwwwa...a......o..oooaaaa ymin) -- (xmin) ymax) (xmax, ymax) -- (xmax,
(" (x)
$"
\ntaWnat is the sum of all integers $x$ in the
interval shown ($0) xille 8$) such that (x)
Solution:
" "python
from sympy import symbols, solve
def sum of x
"""Below is portion the graph of function, Sx=h
[asy]
import graph; size (8em) real 1sf=0.5; pen dps=1inewidth ton +fnttsize
(10); defaultpan pen da=black; real xmin=-0, 75, xmax=8 2..xx8..20 25, ymin --1.25, ymaxx00.22
pen ogcecq=rgb(0.. (0. 75, 75,0.75);
/*gria*/ pen gs=linewiath (0. +cgegegtLinetyype ("2 2"); real gx=1, 2
for (real i-ceil (xmin/gx) +gx; wtpp:i//lffoooooo (xmax/gx) -gx; i+=gx) d ymin) (i, ymax) , for (real i-ceil +gy1e--ooo (ymax/gy) +gy;
draw (xmin, 1) (xmax,
Label laxis; laxis.p=fontsizer
xaxis """,xmin, xmax, Ticks http::/hhhhall..a../oo000/001002 true); ("", (6), above=tru))
above=
Ticks http::/ww....caa.oo.o..ooooo...r
Arrows
real fl (real x) (return (x-0.5) (x-2.5) 6.5) (
draw (graph -5,,, 8.25 25), iine (inewidth(1))
clip somin,yming -- (xmin, ymax) (xmax, ymax) -- (xmax,ymin) --
label (x) (8. (8.5,8),)) 8); [/asy]
What is the sum of all integers $x$ in the interval shown ($0\1e x\ie 89) such that (h ?mmn symbols('x')
h_x (x 0.5) (x 2.5) (x - 6.5) (x - 7.5) / 16
Find the x-values where (x) x_values [x for range (9) if b_x. (x, x)
Calculate the sum of these x-values sum = sum sum(x_values)
return sum_x
result sum oum_of__x) print (result)
output
TypeError: cannot detormine truth value of Rolational
The code above has error. The 'if' statement only be used with boolean values, but the sh_x. (x, boolean value. We need to find different way to solve this problem.
comparison is not
Let's analyze the problem. We are asked to find the sun of all integers 9x5 in the interval shown (s0\1e x\le 85) such that (x)
29

---

Published as a conference paper at ICLR 2024
We can see that (x) is parabola that opens upwards. For parabola that opens upwards, the maximum value of the parabola will occur at the midpoint of the interval, which is sx = \frac(0 + () = 48.
For 8x 49, $h (x) is below the line $y xs, so $h (x) Ale x$. For 48, (x) is above the line $y xê, so 9h (x) x$.
Therefore, we only need to find the sum of the integers $x$ in the interval $0 \ee Ale 48 such that (x)x) x$
Here's s the corrected code:
'python
from sympy import symbols, solve
def sfm_of_x():
Find the sum of all integers x in the interval shown ($0\1e x\le 85) such that (x)
Returns:
int: The sum of all integers x in the interval shown ($01le xALe 8$ such that >x)
..-
symbols ( ('x')
h_x (x - 0.5) (x - 2.5) (x - 6.5) * (x - 7.5) / 16
Find the x-values where (x) and x_values [x for x in range (5) if b_x. subs (x, x) x]
Calculate the sum of these x-values sum sum suu(x_values)
return sum_x
result sum_of_x()
print (result) ...
oouuuut
SyntaxError: invalid decimal literal
The code abeve still an error. The "h_x. subs (x, x) x comparison not boolean value. Wo need to find a different way to solve this problem.
Let's analyze the problem. We are asked find the the interval shown (so1le x\le 88) such that (b(x)>x$.
all integers
We can see that (x) is parabola that opens upwards. For parabola that opens upwards, the maximum value of the parabola will occur the midpaint of the interval, which
\frac(0 (2) 48.
For fx 48, Sh(x) $ is below the line $y x&, so $h \le x$ For Sx
49, (x) $ is above the line $y x8, so Sh (x) x$.
Therefore, we only need to find the sum of the integers $x$ interval $0 Ale x Ale 45 such that $h (x) > xs.
Here's 8 the corrected code:
"pythoo
def sum of x
30

---

Published as a conference paper at ICLR 2024
Find the sum of all integers x the interval shown ($011e x\le 85)
such that (x)
Returns:
int: The sum of all integers in the interval shown (501le x/le ) such that (x)>x.
Define the interval interval range
Find the x-values where (x) > x
x_values = [x for x in interval if * (x 4) (x 6) (x 7) / 16 x
Calculate the sum of these x-values sum = sum(x_valuce)
return sum
result sum sum_of_x() ()
print (result)
oouuuut
SyntaxError: invalid decimal literal
Reach max function call limit.
Execution: Nene
Prediction: innt Ground Truth: '26'
Score: [False]
Reference CoT: 'We could check (x) separately for each integer Sxs from $0$ to $8$: for example, (0) Napprox 3.8$, so (0))00, but $h and so on.An'nHowever, it is
(1) -0.78, so Sh (1) easier to see at a giance which $x$ satisfy (x) by superimposing graph of $y-xs on the graph of (x= \ndraa Andraw ",,
graph; siza():m); real
1sf=0.5; pen ougp::aeeccciidl.oooooaasaaaaaad ottp://eeeeceeeeeeeeeeaaeaeeeaa pen ds #black; real http:///uuaa.aaa.........a..aa..
linetype ("2 2"); real htgp::/h..o.....o.......aaaaaaa.
floor htg.a//.......//////./..a...aaaa http://w..g...oooo/////////////r
i=cell
laxis; : haa.aaa.ea..............aaaaaaaa
y
\nreal
htt.ahaeeecaaaaaa......../aaaaa. (graph (max)--(xxase)
(zeal I
[tt.://.hhh..aaaaa/....aaaaaaaar -xxx)xxxx (xmax, blue); dot (st) http::/.hhhphee..........aaaaaaa
(10,0) dot ((6, six blue dots above mark the integer points (xxx)s which lie below the graph of (x)
blue); dot ( (1t6g//)).baeee//aaaaaaa/////ooo
indicating that (x >xs. Their 5xf-coordinates are $0,3,4,5, which add up to $1lboxed
Listing 10: Failure case for ToRA: Runtime Error. ToRA failed to fall back from program timeout rationale-baus solution. For Intermediate Algebra and Precalculus problems like this example, direct brute-force solutions are often infeasible, resulting in exceptions like timeout. Addressing these challenges requires complex reasoning over math expressions, and may involve sophisticated problem-solving and proof techniques like forward and backward reasoning (Jiang et al. 2023). as well as result verification (Gou et al.) 2023 Zhou et al. 2023a).
1dx: 2261
31

---

Published as a conference paper at ICLR 2024
Yype: Intermediate Algebra
Level: Level
Question: "For some positive integers $m and sn, S1art/// Magrt {281} + Visqtt - (2grr((28) Msgrrim im (n)). \1jEnter the
ordered pair (m, .
Solution:
"python
from sympy import sqrt, solve, symbols
def (inn___it))
"**For some positive integers Smş and $n, (23 (8qrt(28)) + \eqrt \sqrt(28)) (sqrt(m + \sqrt/n:1. Enter the ordered pair s (m,n)
m symbols n
Define the given equation equation sqrt (23 sert (28)) sqr - sqrt (28)) sqrt (m sqrt (n
)
Solve the equation for and solutions solve (equation, (m, n)
Filter out the solutions where m and are not positive integers solutions (sol__, m, sol_n) for sol sol in solutions if sol
and soi > and m.is integer and sol n.is integer]
return sclutions [0]
result find_m_nly
print (result)
oouuuut
TypeError: cannot determine truth value of Relational
"..
The error the code above is due to the fact that the 'solve' funct on in SymPy does not handle square roots very well. Instead, we can manually selve the equation for $mş and SnS using the properties of square roets.
Let's dencte $1sqrt \sqrt as $aş and Siarrt (23 (28)) $b$. Then the given equation can be rewritten as $a b = Asgrt (m (n))5
Squaring both sides, we get $a*2 2ab + b = m \eqrt(n)s.
Since $a*2 23 + \sqrt and $b^2 23 \sqrt (28)$, can substitute these values into the equation get $28 2\sqrt \sqrt
\sgrt
Solving this equation for Smş and $n$ gives Sm 49$ and $n = 725.
Therefore, the ordered pair (m, n) $bboxed! (49, 72) }$.
Execution: None
Prediction: (49,72) 72)
Ground Truth: (46,2004)
Score: [False]
Reference CoT: 'Lot $x Vagrt (23 + Msgrt
(28)) (28))
Magrt (23 Migrr
(28:).$ Then ShenAn\\een b
&= 23 + (1sgrt(28)
(1ggr((23 +
[1sqr((28) \lsgrt (lagrt((r))))
+ 23 (Nsgrt((8)
WIl \ng= 46
\1sgrt 28) iiilins- 46 \1sqrt(501) in- 46 Aserr
(2004) Andlen//aalennnnnnnnmmm / Hence, $ (m, n) Wbboxed (46,2004) )
Listing 11: Failure case for ToRA: Rationale-only Error.
32

---

Published as a conference paper at ICLR 2024
Idx: 2638
Type: Intermediate Algeora
Level: Level
Question: 'The graph of Sy (x) $ is shown below. [ (nnnitsize cm) Anknreal func(red) x) (An real yı if
x 0) { -2 - An if (x >= & x 2) ly sqrt -x- \n if
(x >= 2 ss x <= 3) ly = 2* 22)))n retuumiyy) i, n:\n)
nfer = =5 i <= 5; ++i) (1a)
draw Andraa (5,0),550))) (5,0), (M)
(0,5), (1)-0)0.)) [gh.,,aae_eaaa.......0..a22,a011
(")) fx)
(3,-2), [nfi1)))nnl/a/////wwhhhee is the graph of $y =f==((x))))m
asy]\nunitaizee0.5 cm) [n/npiccurrr graf; i, n;inineeal func real x) [\n real yihn if 2=
( -x)]n if (x
2+ - 2 sgrt (x 2) 2n (x >= 46 x 3)
& x
ly =
funch (real x) I'n return
fune (x));\n/\llnree) (funn)((un((-))) \n))))))))))nnnn/oo (n 1; <= 5; ++n) (
(real x) I'n (-fun(
funce (real x) (In return
graf[n] new picture;/n for (1 -5-
5; ++11 (An (graf
1, o|ap::3.h_..aal......aaaaaaaaaaa n Hmn -, hrgp::/hhhh....aaaaa.aaaaaaaaaa.
n],"Sys", (0,5), ||t.a.a...............aaaaaaaaaa oat.ppaaccc.a22220.......aa..aa.
ndraw ndraw red); htgp:p/ehhhhaaaaaaao..a/aaaaaaar
graph orgpphheee_Sppppoo//aaaaaooaaaad [21, "3", [1,_="))||_aaaa000000.0000100111 ( org
"b"",
"E" htgpg//...aaaaaaaa....../oo/aaaa
( (12 (agp:/2000000..aaaaaaaaaaaaaaaaa (16,-1))
[g] (st
(graf (51)) (5]))//n/asy/ http://yyyyy//nnettreeeeee the
letter of the graph of Sy (x)
Solution:
The correct graph is the one that reflects the original graph y f()
about the axis.. This is graph D.
So, the answer $lboxed/ (t)
Execution: None
Prediction: "p"
Ground Truth: "B"
Score: [False]
Reference CcT: 'The graph of Sy = (xx) $ is the reflection of the graph of $y = f in the sxs-axis. The correct graph is $ Wboooxe(())) \ Vtext (B
11.5'
Listing 12: Failure case for ToRA: False Negative.
Idx: 2228
Type: Intermediate Algeora Level: Level 3
Question: 'Find all the real roots ef\n\M((12x - 1) (6x - 1) (4x - 1) (3x
1) 5.Aeerr all the real roots, separated commas."
Solution:
'python
from sympy import symbols, solve, simplify
def find_real_roots()
"**Find all the real roots of
(12xx 1) (6x 1) (4x 1) (3x - 1) = 5.A]Enter all the real roots, separated by commas. - symbola('x')
equation (12xx 1) (6*x 1) (4*x 1) (3*x 1)
33

---

Published as a conference paper at ICLR 2024
roots solve (equation, reat_roots [root for root in rects if root.is_reat
return real_roots
result find_real_rooe ()
print (result)
"ooutpu
1-1/12, 1/21
1.1
The real roots of the equation are S \frac[1]2215..
Execution: None
Prediction: 12.2 17 Ground Truth: 2. 12 Score: [False
Reference CoT: 'ee can expand using the pairs (3x - 1) (12x 1) 36x
15x 1$ and (6x - [) (4x - 1) 24x 10x + 1, s3\n\\[(36xx2
15x 1) (24x*2 - 10x - 1) = 5.\ $y 12x"2 5x. Mhy +
96y 5y 4 44 factors as ((y 1) (3y + 0.$ Rence, $y (1)r or Sy
1) (2y 1) 5.A1]hhss simplifies
0,4 which
$12x"2 5x \)frac(1] then $24x"2 10x
= 0, which factors /einll/(2x (12x + +1 = O.AW/Hence, Sx = =\
frac(1H21$ $x =
$12x^2 5x -llfrac
(4)(3), thenln\1136x"
15x 4 0,001 has no real solutions.
n'nthereefore the real roots are \frraxe (11(12))5*
-lifrac
34


Reviews:
[{"review_id": "0BJfx4ruLc", "reviewer": "Reviewer_mvjp", "summary": "The authors present a framework for improving mathematical reasoning by combining natural language descriptions with program synthesis and execution. A hybrid tool-integrated reasoning context is used to sample candidate reasoning trajectories for the GSM8K and MATH datasets. Candidate trajectories are then verified and those that are successful are added to the ToRA corpus. This corpus is used to fine-tune intermediate models. Such models are then employed as in the initial setting to sample candidate trajectories with feedback from teacher correction to aid the completion of partial trajectories. This final set of valid trajectories is used for further fine-tuning and finally producing the ToRA fleet of models, extended from the LLaMA-2 and CodeLLaMA base families. The resulting models show considerable performance boosts on a range of diverse mathematical reasoning datasets.", "questions": "As noted above, additional discussion of failure modes would be beneficial.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The idea is clear, well-constructed, and well-explained. The figures are excellent and the algorithm is clearly laid out. The resulting models show considerable performance increases under a range of evaluation settings confirming the efficacy of the strategy.", "weaknesses": "While the authors have presented what worked well, there is a considerable amount to be gleaned from the failure modes. The authors loosely allude to failure cases including geometric problems and program timeouts, and provide single examples in the appendix, but there are surely more interesting patterns. It would be wonderful if the authors could provide more specific examples and comment on more systematic classes of errors beyond these simple categorizations. For example, are there certain patterns in the natural language specification of the original problem or rationale construction that fail to formalize well as programs? Were the patterns informative in terms of which problems were amenable to imitation learning vs which required output space shaping in order to produce initial valid reasoning trajectories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NPEeseEgJx", "reviewer": "Reviewer_V1TN", "summary": "This paper introduces TORA (Tool-integrated Reasoning Agents), which seamlessly integrates natural language reasoning with external tools to solve complex mathematical problems. By combining language models' analytical capabilities with computational efficiency tools, TORA significantly outperforms open-source models on 10 mathematical reasoning datasets.", "questions": "1.It appears that the method proposed in this paper shows more significant improvements on smaller models, as the performance of the 7B, 13B, and 70B models shown in Figure 1 does not appear to differ significantly. How to explain this phenomenon?\n\n2.The alternating reasoning approach between natural language and tool usage proposed in this paper is fundamentally similar to the plan paradigm of Thought, Action, and Observation alternation in REACT [1]. Do you think this paradigm will become the dominant paradigm for agents to solve complex reasoning problems?\n[1] ReAct: Synergizing Reasoning and Acting in Language Models, ICLR 2023", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper proposes a two-stage training framework that utilizes training data alternating between natural language and code language to enhance the reasoning ability of language models in mathematical reasoning tasks. The experimental results demonstrate the significant improvement of this approach across 10 datasets.\n\n2.The paper is generally well-written and the figures and tables presented are clear and easy to understand.", "weaknesses": "1.From Figure 5, it can be observed that the performance of the model does not significantly decrease when output space shaping is removed. More experiments are needed to demonstrate whether the performance improvement in this stage is due to this training strategy rather than additional data and more training epochs.\n\n2.Regarding the TORA-corpus proposed in this paper, more detailed information is needed regarding the data construction process, quality evaluation, and dataset statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNfGQirWBM", "reviewer": "Reviewer_U6Sz", "summary": "This paper introduces TORA,\n a series of Tool-integrated Reasoning Agents designed to enhance mathematical problem-solving by combining natural language reasoning with external tools. TORA models are trained using interactive tool-use trajectories, employing imitation learning and output space shaping techniques. Experimental results demonstrate that TORA outperforms open-source models on various mathematical reasoning datasets.", "questions": "- In the Conclusion section, the authors mention that \"our systematic analysis ... paving the way for the development of more advanced and versatile reasoning agents\". **How and why does this analysis pave the way?** \n- What is the difference between Tool-Integrated Reasoning (Algorithm 1) and existing methods (e.g., PAL (Gao et al., 2022), PoT (Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks, https://arxiv.org/abs/2211.12588))?\n- \"For numerical values, we perform **rounding**, while for expressions, we employ **sympy** for parsing.\" Are these two techniques used in the baseline methods? \n\n- Are there any empirical results can support the ``Output Space Shaping improves **diversity**''. Furthermore, the diversity measure is not defined in the paper.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is easy to follow\n- TORA achieves good performance on math datasets", "weaknesses": "- **Limited of technical novelty**: \n  - Using imitation learning to improve the mathematical reasoning ability of open-source models has been proposed in many recent works, e.g.,\n    - Scaling relationship on learning mathematical reasoning with large language models, https://arxiv.org/abs/2308.01825\n    - WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct, https://arxiv.org/abs/2308.09583\n    - MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models, https://arxiv.org/abs/2309.12284\n    - MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning, https://arxiv.org/abs/2309.05653\n  - At the Output Space Shaping step, the authors use nucleus sampling to generate more reasoning paths and pick the corrected paths. This is a technique widely used in the works listed above. The only difference is that this paper fixes some of the preceding portions of wrong trajectories, while existing works resample the whole trajectory. However, validating which portions of trajectories are correct is very challenging, and the authors need to **enumerate** possible preceding portions of wrong trajectories, which is time-consuming. In my opinion, the existing method (i.e., re-sampling and picking the correct paths) is simpler and may be more effective.\n\n- **Concerns on reproducibility**: As training data are not provided, reviewers and readers can't check the reproducibility of this paper. Note that existing works like WizardMath, MetaMath, and MAmmoTH have released their training data for the community to reproduce their results. Moreover, checkpoints of TORA are not provided in the appendix for checking reproducibility.\n\n\n\n- \"TORA outperforms WizardMath by around 45% in Algebra and Number Theory, **which is attributed to stimulating and shaping tool-use behavior**.\" From Table 3, we cannot conclude that the better performance of TORA is due to stimulating and shaping tool-use behavior, as \nWizardMath uses augmented data **from LLaMA**, while TORA uses data generated **from GPT-4**. Note that GPT-4 is much more powerful than LLaMA. \n\n- In section 2.2, greedy decoding is used for generating trajectories from GPT-4. Thus, only one path per question can be obtained in TORA-CORPUS dataset. In my opinion, the accuracy of LLaMA trained on TORA-CORPUS has yet to saturate (e.g., plot the accuracy w.r.t. #samples of TORA-CORPUS).  To generate more trajectories, a simple approach (which is widely used in the above works) is to use temperature sampling (rather than greedy decoding)  and pick the correct ones for training.\n\n- As temperature sampling can generate more samples from GPT-4, TORA-CORPUS can be more diverse (compared with greedy decoding). To verify the effectiveness of Output Space Shaping in Section 3.5.2, it is better to **augment more data from GPT-4 and let the accuracy of LLaMA trained on the TORA-CORPUS saturates first**. Otherwise, it is difficult to say whether the improvement of LLaMA is from more training data or the proposed Output Space Shaping.\n\n- Ablation study of hyperparameter $n$ (maximum rounds). in experiments, $n=3$ is used. Question: is the performance of TORA sensitive to $n$?\n\n- writing:\n  - \"forward and backward reasoning, as well as result verification\": references for forward/backward reasoning, verification\n  - \"Addressing these challenges requires complex symbolic reasoning over algebraic expressions\": references for symbolic reasoning \n  - where is the definition of $\\theta$ in (4)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VHE2WFeeP", "reviewer": "Reviewer_Gq26", "summary": "The paper presents ToRA, a method of utilising interleaved reasoning and tool-use traces to improve language models' capability in mathematics. The authors used GPT-4 to get a collection of interactive tool-use trajectories for mathematical problem solving, and fine-tuned open-sourced language models on this collection to improve their performance.", "questions": "How should we improve upon ToRA? Since the pipeline relies on a proprietary model to generate the dataset, and the final performance of the strongest ToRA is still inferior to GPT-4, how can we further improve?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The performance gains on the downstream mathematical reasoning benchmarks is impressive.\n- The ablation studies convincingly proved the necessity of both the rationale and the program parts of the ToRA pipeline.", "weaknesses": "- Adopting complicated mathematical fonts when unnecessary only takes from the readability of the paper. There is no need to use them when the default fonts suffice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)