SBSC: Step-by-Step Coding for Improving Mathematical Olympiad Performance
Authors: Kunal Singh
Ankan Biswas
Sayandeep Bhowmick
Pradeep Moturi
Siva Kishore Gollapalli
Venue: ICLR 2025 Poster (2025)
Keywords: math AI
LLM math reasoning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We propose Step-by-Step Coding (SBSC): a multi-turn math reasoning framework that enables Large Language Models (LLMs) to generate sequence of programs for solving Olympiad level math problems. After each turn/step, by leveraging the code execution outputs and programs of previous steps, the model generates the next sub-task and the corresponding program to complete it. This way, SBSC, sequentially navigates to reach the final answer. SBSC allows more granular, flexible and precise approach to problem-solving compared to existing methods. Extensive experiments highlight the effectiveness of SBSC in tackling competition and Olympiad-level math problems. For Claude-3.5-Sonnet, we observe SBSC (greedy decoding) surpasses existing state-of-the-art (SOTA) program generation based reasoning strategies by absolute 10.7% on AMC12, 8% on AIME and 12.6% on MathOdyssey. Given SBSC is multi-turn in nature, we also benchmark SBSC’s greedy decoding against self- consistency decoding results of existing SOTA math reasoning strategies and observe performance gain by absolute 6.2% on AMC, 6.7% on AIME and 7.4% on MathOdyssey.

Published as a conference paper at ICLR 2025
SBSC: STEP-BY-STEP CODING FOR IMPROVING MATH- EMATICAL OLYMPIAD PERFORMANCE
Kunal Singh, Ankan Biswas, Sayandeep Bhowmick, Pradeep Moturi, Siva Kishore Gollapalli Fractal AI Research Mumbai, India (kunal. grunal.singh) e afractal..a
ABSTRACT
We propose Step-by-Step Coding (SBSC): multi-turn math reasoning framework that enables Large Language Models (LLMs) to generate sequence of programs for solving Olympiad level math problems. At each step/turn, by leveraging the code execution outputs and programs of previous steps, the model generates the next sub-task and the corresponding program to solve it. This way, SBSC, sequentially navigates to reach the final answer. SBSC allows more granular, flexible and precise approach to problem-solving compared to existing methods. Extensive experiments highlight the effectiveness of SBSC in tackling competition and Olympiad-level math problems. For Claude-3.5 -onneet, we observe SBSC (greedy decoding) surpasses existing state-of-the-ar (SOTA) program generation based reasoning strategies by absolute 10.7% on AMC12, 8% on AIME and 12.6% MathOdyssey. Given SBSC is multi-turn in nature, also benchmark SBSC's greedy decoding against self-consistency decoding results of existing SOTA math reasoning strategies and observe performance gain by absolute 6.2% on AMC, 6.7% on AIME and 7.4% on MathOdyssey. Scripts & Data is uploaded at this link
INTRODUCTION
Mathematical reasoning has emerged as critical benchmark to measure the advanced reasoning and problem-sol solving abilities the Large Language Models (LLMs) (Brown et al., 2020 Chowdhery
al. 2022 Achiam et al. 2023 Reid et al. 2024 Anthropic, 2023 OpenAl June, 2024). This is due
to the complex and creative nature of the numerous reasoning steps required to solve the problems.
Chain-of-Thougee (Wei et al. 2022 and Scratchpad (Nye al. 2021) prompting strategies helped LLMs to solve problem using a step-by-step thought process. Program-Aree Aidud Language (PAL) (POT) (Chen al.) 2022) introduced problem-solving via program generation where the answer is obtained by executing the generated program. Tool-Integrated Reasoning Agent (ToRA) (Gou al. 2023) & Mathcoder (Wang et al. 2023a) introduced tool- integrated math problem sol solving format where model outputs natural language reasoning followed by program generation to solve the entire problem using single code block and incorporates code- interpreter output for either summarizing the program output to get the final answer and terminate; or re-attempt the problem in the subsequent turn using the same format. For brevity, let's call ToRA's defined way of tool-integrated reasoning (TIR) strategy as TIR-ToRA.
(Gao al. 2022 & Program-Of-Thouuee
The current generation advanced LLMSs such as GPT-4o (Achiam et al. 2023). Hlaud-3.5-Soomme
(Anthropic) 2023) and Gemini-ultra (Reid et al. 2024) have achieved high scores on elementary GSM8k (Cobbe et al. 2021> high-school level MATH (Hendrycks al. 2021) ) by leveraging these
reasoning strategies via in-context learning (Brown et al. 2020 Chowdhery et al.) 2022). Multiple
studies (Yu et al. 2023b Yue et al. 20233 Toshniwal et al. 2024 Gou et a Wang et al. 2023a
Mitra et a 2024 Beeching et al. 2024 Shao et al. 2024) have tried supervised fine-tuning (SFT)
approach to distill these reasoning formats using propricty models like GPT4 (Achiam et al. 2023).
These studies show significant performance improvement over GSM8K and MATH benchmarks.

---

Published as a conference paper at ICLR 2025
1.1 MOTIVATION
However, recent math specific competition and Olympiad-level benchmarking on Math Odyssey (Fang et al. 2024). OlymiadBench (He al. 2024), and the American Invitational Mathematics Examination (AIME) & the American Mathematics Competitions (AMC) (Beeching et al. 2024
DeepSeek-Al al. 2024 Reid al. 2024) questions show that the state-of-the-ar (SOTA). both
generalist and specialist, LLMs continue to struggle with advanced math reasoning. These results highlights the limitation of the existing math prompting techniques. (Tong et al. 2024) highlights the severe bias towards easy problems that exists in the SOTA SFT datasets which originates primarily due to the ineffectiveness of the current prompting strategies in complex math problem-solvine Often, multiple chains are generated via self-consistency decoding (Wang et al. 2022) and majority voting is done to boost the accuracy which is unlike how humans solve problems.
Fundamentally, both PAL & TIR-ToRA generate single program block to solve the entire problem. Additionally, ToRA framework allows the model to re-attempt the program generation in case of execution error. These approaches show improved performance over COT on elementary & high school level math problems. However, solving olympiad-leve math problem requires coming up with complex and creative solution that constitutes of numerous elaborate intermediate steps which eventually leads to the answer. Often, it is not feasible to solve a complex problem entirely using a single program block and result, these prompting strategies fail systematically address each detailed step of the groblem-solving process. It tends to overlook specified constraints, edge cases or necessary simplifications, which are often encountered in Olympiad-level problems.
1.2 OUR CONTRIBUTION
Olympiad level math problem-solving can be viewed as solving/exploring an intermediate sub- coscket-connept in depth; and discovering + solving the next critical sub-task dynamically basis the accumulated knowledge of previous bub-tasks/key--coeee Step-by-Step Coding framework (SBSC) which is a multi-turn math reasoning framework that leverages existing programming (Naman Jain 2024) and in-context learning skills (Brown et al., 2020) of the current generation of L.MMs, particularly Claude-3.5-Somm (OpenAl, June, 2024p. In each turn, it leverages code-interpreter results and knowledge of previous sub-tasks solutions or concept-explooations to define and programmatically solve the next sub-task. Thus it uses code generation the reasoning strategy solve an intermediate sub-task explore an intermediate concept/step. Thus, providing detailed focus to each step of problem solving unlike PAL & TIR-TORA. SBSC allows intermediate key-step be discovered, and be explored and refined (if needed) before being appended to the chain of steps whereas in PAL & TIR-ToRA all the intermediate steps always stitched together.
explorations. To this end, we propose
(Anthropic (2023) & GPT-4o
We investigate the performance of SBSC on last 11 years of AIME & AMC-12 questions. We also benchmark on Olympiad-subset of MathOdyssey dataset along with math questions from OlympiadBench. We compare our method (greedy decoding) against greedy-decoding generation of existing reasoning strategies: COT, PAL & TIR-ToRA. We also show SBSC (greedy decoding) effectiveness by benchmarking against self-consistency decoding results of COT, PAL & TIR-ToRA. We conduct extensive ablations to understand the benefits of our approach such as sensitivity to exemplars, topic-wise analysis and measuring improvement in program refinemen//debugging ability over ToR- due to the granular nature of SBSC process.
SBSC: STEP-BY-STER CODING FRAMEWORK
Solving complex math problems, such as competition Olympiad-level ones, involves creative thinking, applying diverse mathematical knowledge, and dynamically creating subsequent strategies insights emerge. One must discover sub-tasks dynamically, rigorously explore intermediate concepts, and carefully handle constraints and edge-cases. Since PAL & TIR- TORA generates single code block (even during self-correction step; incase of TIR-ToRA) to solve problem, they lack the flexibility or granularity to emulate this. To address this, we introduce SBSC.
SBSC is multi-turn, code- onde-generation based math reasoning prompting strategy where at each turn: the model generates an intermediate sub-task task and corresponding program to solve that sub-task by leveraging the outputs of the previous turns. At the end of each turn, code interpreter is used to
2

---

Published as a conference paper at ICLR 2025
execute the program block to generate the solution for the intermediate sub-task. The intermediate sub-task depends on the results of the previous turns and the question. The code snippet for the it sub-task directly incorporates the exccution results of the previous code snippets by directly defining them as variables and symbols. This way SBSC makes LLMS generate sequence of programs over multiple turns solve complex math problems.
Problem:
Fnnd the s.m of all positive inlegers Sins such that when $1"3"2"3-334
+n'38 is dhidded Sn-58, the remaindr is $175.
Turn (S,, Ct. or)
Turn 2 (5g, Ey. oz)
Turn (8,, Cy,
Let' aparoads prbiicn stppl skop
code Skep 1. tepul Rea ins and delivelle lo
Stop Sphrp the countin beaed on gaon contacce
Stoc 3. Sohro for
Irpvett swiloss, Ea,
sueracioo
From symyy fee aed spaladt, Tup sulus, somma Tte (atee(a):
From spayy inport symbolo, Es, sodro, cormitionyood woilivesTes, atiiggo-ree)
det sue of cuber(e)::
spriools (nb
return somettooneees, [K, nl)
relume (intg 1,
syeholss e*) of oubeste))
sun cobes n112/4 #31//2 ->42/4 equtiinn
sonbolat t' poslliee integreirsw)
equeligr tulMoolnnnejj4 wttg/2 st2g/4 35
Man cutes 17 (wed nis)
pelat (rr of colos from to ((som_rshos)"
Salve the equation
ecuation ta(Ceun cubes prantle Ecuutiion: (equation]
so lultiors solvetenuntion pe Inf(("Solattoon [solations]) {")
ofrace enttys
273 eiperiite Inplapetinl whe hotgeraaa smaa) 1 arraja 15,
1/2// 17, 53, #)
Turn (6 Ca. 0.)
(ty, Cy,
Turn (8,,
Colculate
Stop Vorily the colutons
Step 3 (lacised);
Ohack conditiar for cange of values
sympy Lecori symbolk, bu, sctin, somation cocationo [rS, J63]
free ojesy taport tyskols, i4. orlve, somettio
froe saapy insert samolls, Tu, solve, sumnt im def use of cubes (e):
(f of astottons: Jun woteeiaii
Nallelltte solutiing
caluriise
(cslinten))
solusiooss (28, i53]
reture pencit 2r/5/2 111//4
(nbbe(t) 5 In
the n in ranpect, 1mm)):
colutiine []
af1//4 arra/2 1192/4
Sun of col ittans: 200
prins/eTTor a [n]:?)
fhork up 1866, offict seedul
preat(""'sa of uibe: (ave (obss))
[fun_of cubizze)) 223
dddt END OF CODE
print "souution: [sclatioon)
Sobutiore: 178,
The fina answer 16 (boxeciZ34)
can ansiieenuss,
For a 1812
Sen cabes 17100000... fecaisder: $7.0
(a) Example multi-turn SBSC response for AIME problem. Pink boxes denote the sub-task 8; at the i-th step. blue boxes denote the program e; to solve s; and >>> denote the corresponding execution output og. The red curly brackets indicate reusing outputs from earlier steps.
Turn 2
find aneliow degerr
Let's splve the problem stop and en P'thon function find the surn of
- 201
clwded by (n-
tive intraom in such that 1 1 + 243 313 locors WVS I dividid by
We start by using the fomuua the sum of oubes of first ((n)) positis intagers:
in 5/1. fiee restetrdee is
{ e 243 - 5AS laakth w'd baddonn 5
Slep Slep I'lmn
thus, we need. tomp toffegeer' rquar gmod
cuses tommuam d'd laakde
bestive intagors is
sot
uqqullion. int. (T fee TH20iigh k/in 5) some
beeddee 18-7wi
"*Congntenoon condition We need l'wefillecdnnn "802/runt" wore
5f 58 y
Aua Skacky Skep Flan.
"Sginelll The congraeeea" ot 'I Weltileetrnn 11PP ighn - 17'
1. "Expand the couation" Rowilo the equation tams of Kn and simpliy
some néoger
"TS wik duuellity Aredyom when repressonn wet ln masitden (m155
condiion": rewnite conditon: | bretttre 11/2Aeghh0"' lequit 17' (emosth-5)
i tr: Dofommin values of vn jhat safidy condition
"Sur
"*len and solve" lees porsible values of V V and check the condil on ind all voli (n
Sopare syery
def find valomafts
Min
i op-syaboted (w", "ategen-TTun)
sp.apedella nn, Intwae-True)) cupe (n 9 (n (0-13 33++3
exrreasion (r 1)// 23-12
for nange(), 1880):
Autertt over ressmmalle range to find wll pasible n afjict rangee, needed all possible If (1 5) 12:
equatten sp iafeooeeessso (n.+ 5), 17) Adjiaa
masded Fand posuible
colutiore qu pe.ciiateeuatinn, n)
for solutions
spaand(ec])
sie vivlles Tind_pum_of_n
not en cann
returs rocults
Waooodd values 123")
valuus färd values()
sun valuss sumin walaes)
>55 (hound(B)
petnt (ooness wilueg332))
239 No elgeeities are inplerented to solve equation modeoetsa(e e 17445/8, n 3) 17
The final Vaoxed(()
(b) Example TOR-ToRA response for the same problem, which is not solved correctly. In first turn, it tries to solves the problem at once using rational and program. It encounters error and in second turn, tries to fix the entire approach and solve again but the solution incorrect.
Figure I: Comparison of SBSC and TIR-TORA frameworks for same AIME problem
Our inference procedure is inspired by ToRA (Gou et al. 2023). Solution chain is initialized with the Prompt containing method instructions followed by exemplars and the current question q. At each step, LLM G first outputs a subtask s,. If s; generation ends with stop-word "4 a HEDD OF CODE", extract the final answer. Else, it continues to generate program code o; ending with
3

---

Published as a conference paper at ICLR 2025
stop-word "output". We then pass c; code interpreter and obtain the execution message output o; + (c.). The solution chain is updated by concatenating it with s,,4,,; and loop continues till we get "# # HEND OF CODE". For the 2th turn and @ denoting concatenation, the sequential process can be generalised as (except for the last turn where just the final answer is generated)
S cj ~ G(- p p p (s1 @c + (o)) @ (s2 02 ① +02) @ ......(: θ 2 ()-) (I)
Step-wise sequential approach of SBSC ensures that every part of the problem addressed with exact precision, reducing the risk of errors that might arise from false assumptions or skipped steps. In case the code execution for any step results an erroneous output, SBSC better able to rectify that particular step. In depth understanding of SBSC (multiple examples & comparisons) at A.6
We present example responses from both SBSC and TIR-ToRA for problem from AIME figures aaland [ respectively. As seen in case of TIR-TORA, A, the initial program generated by the model runs into an execution error. At the next turn, attempts to rectify the error and comes up with a new approach and the corresponding program. This time, the code executes correctly but due to reasoning error the final answer wrong. On the other hand, we see that SBSC is progressing step-by-step. tackling individual sub-tasks with separate programs and utilising outputs of previous steps. In the third step, it runs into a code execution error but succeeds in rectifying it using a different approach in the very next turn. Further, observe SBSC checking the validity of the generated solutions in the fourth step before proceeding with the final step and ultimately reaches the correct answer.
2.1 SBSC EXEMPLAR DESIGN
To enable SBSC framework in LLMs, we rely on in-context learning abilities (Brown et al. 2020) of LLMs explored by multiple previous works such (Chen et al. 2022 Gao et al. 2022 Gou et al. 2023) etc. We also use a system prompt similar to previous works. With respect to exemplar laa design. to enable program generation, we borrow learning from PAL (Gao et al. 2022) & POT (Chen et al. 2022) to have meaningful variable names in the code and using natural language comments within programs(Chen et al. 2022). To enable intermediate tool (code interpreter) usage, we leverage the use of stop words similar to in (Gou et al.) 2023). Sample SBSC exemplars can be found at A.11
A.12
3
EXPERIMENT
3.1 BENCHMARK DATASETS
We mainly use problems from 4 popular math competition datasets for benchmarking our performance: AIME, AMC, MathOdyssey (Fang et al. 2024) and OlympiadBench (He et al., 2024). covering multiple domains, mainly: Algebra, Combinatorics, Number Theory and Geometry. We use problems of last 11 years from AMC and AIME, obtaining questions and answers (Q&A) in LTEX format from the AoPS Wiki website, MathOdyssey (Fang et al. 2024), a popular benchmark for LLM math reasoning, consists of problems of varying difficulties. We include the 148 problems belonging to olympiad-level competitions. OlympiadBench another challenging benchmark for LLMMs containing olympiad-level multilingual scientific problems. We select only math related questions, in english language.
3.1.1 DATASET PROCESSING DETAILS:
First, we filter out all questions having reference images associated. Second, we process the questions to have integer type answers if they are already not in that format. All AIME problems have unique integer answer ranging from to 999, while AMC-12 problems are of Multiple Choice Question(MCQ) format. Similar to NuminaMath (Beeching et al. 2024), remove all the answer choices from each AMC-12 question and modify the representation of the final answer for the question, wherever necessary, to ensure integer answer. In case of OlympiaBBench and MathOdyssey, simply modify the question as needed. For this, we prompt GPT-4o to append an additional line at the end of each problem suitable. Following is an example for demonstration:
4

---

Published as a conference paper at ICLR 2025
Original Question: An contains one red ball and one blue ball. A box of extra red and blue balls lies nearby. George performs the following operation four times: he draws a ball from the urn at random and then takes ball of the same color from the box and returns those two matching balls to the urn. After the four iterations the urn contains six balls. What is the probability that the urn contains three balls of each color?
Answer:
Modified Question: An urn contains one red ball and one blue ball. A box of extra red and blue balls lies nearby. George performs the following operation four times: he draws ball from the urn at random and then takes ball of the same color from the box and returns those two matching balls to the urn. After the four iterations the um contains six balls. What is the probability that the urn contains three balls of each color? the answer is represented as a fraction m in its simplest terms, what is the value of m+n? Integer Answer:
Final test set, contains 330 AIME, 475 AMC-12, 158 MathOdyssey & 504 OlympiadBench problems.
3.2 BASELINE CONFIGURATIONS
We benchmark against three prompting/reasoningg strategies: COT (Wei al. 2022). PAL (Gao et al. 2022) TIR-ToRA (Gou et al. 2023). We use gpt-4o 40- 20-200 2024 000---0--0 13 and C Claude-- 3. Sonnet as base LLMs for our experiments. For all datasets and all reasoning frameworks, we use 4-shot setting. Maximum number of turns (n) SBSC is set to 15. For greedy decoding inference, we use cemperature-0 and max_ wee__ttkenss1000 1024 and also, we run 3 times and report average. For greedy decoding of TIR-ToRA ToRA, we keep ToRA strategy the model attempts to solve the entire problem in the single turn, in case of execution error or readjustment tries to re-attempt in subsequent turns). We also benchmark SBSC's greedy decoding results against self-consistency (SC) (Wang et al. 2022) decoding results (majority @ 7) of COT, PAL& & TIRTTORA. We do this primarily for two reasons: First, SBSC takes multiple turns before arriving at the final answer (on average 6-7 turns per problem. Table in Appendix (A.1) and Secondly, to benchmark against the reliance of the current existing prompting strategies majority voting for boosting accuracy. For SC decoding, we use empe rat rature=0. and top_p=0._ 9. Note: experimentally observe that for 4, there is insignificant increase in accuracy for TIR-ToRA so we set n-4 4 for TIR-TORA during SC decoding.
15 well (Note: this is because although in TIR-
Note: PAL (Gao al., (022) work also reports a combined approach with Least-to-Most (L2M) prompting strategy (Wang et al., 2022). L2M-PAL that is essentially two stage. We implemented it as per the reported examples in the PAL work. We benchmark on AMC AIME dataset. We observe that L2M-PAL at best matches PAL or TIR-ToR, scores. Detailed results available in appendix 14.14 Hence for our main results, we stick to PAI. TIR-ToRA along with self-consistency decoding due to resource optimisation and wider adaption of those prompting strategies for math-problem solving. For more discussion on L2M-PAL please check A.144
3.3 PROMPTING/FEW-SS
EXEMPLARS
For both AIME and AMC, we select 90 questions each, drawn from problems of years other than those included in the evaluation datasets. These questions were prompted with COT, PAL, TORA and SBSC to generate corresponding solutions in accurate format. For each dataset, we create a subset of 10 problems correctly solved by every method and finally select a combination of 4 exemplars among them. For MathOdyssey as well as Olympiad Bench, we use AIME exemplars as these datasets are of similar difficulty level. We provide the chosen exemplars and system-prompts. used in the main experiments, for different methods in Appendix (A.9 A.10 A.1I A.12) & repository here
4 RESULTS
We report the percentage accuracy of all the methods with different base LLMs and across all the benchmarking datasets in Table On AMC dataset. SBSC shows an absolute improvement over TIR-ToRA (greedy decoding) by roughly 11% using Claude-3.5-Somne and 7% using GPT-4o. SBSC
5

---

Published as a conference paper at ICLR 2025
Table Benchmarking SBSC against different math reasoning methods across 3 datasets: We report the average accuracy( in percentage unit) over 3 runs. Best result in each setting is highlighted in bold second best is underlined. Absolute improvement performance by SBSC over the previous best method in each setting is indicated in subscript.
Method
AMC
AIME
MathOdyssey maj @?
Olympiad Bench
greedy
maj
greedy
maj@ @
greedy
greedy
maj
Claude-3.5 Mlude-3.5-Sonne.
COT
31.16
35.79
9.09
10.91
11.89
16.89
39.35
42,46
PAI.
35.79
36.42
27.48
28.79
27.23
31.01
41.07
44.44
TIR-ToRA
38.59
43.16
24.64
26.67
27.23
32.43
47.69
50.60
SBSC (Ours)
49.33
16.2
35.45
10..7
39.86112.6
117.4
53.31
12277
GPT-4o
COT
35.94
37.47
10.39
12.12
13.51
17.57
41.80
47.22
PAI.
36.48
38.11
24.63
26.97
15.74
20.27
41.67
46.43
TIR-TORA
37.33
40.42
22.42
25,45
19.59
23.64
43.32
49.61
SBSC (Ours)
44.55
14411
30.7
13.77
26.55
1229
48.74
-0.87
greedy decoding results outperforms SC decoding results of TIR-TORA by absolute 6% and 4%, for Claude-3.5-Sonie and GPT-4o respectively. We similar absolute improvements in accuracy on our AIME dataset too. SBSC outperforms its nearest competitor (PAL) by 8% and 6% with greedy settings and SC settings by 6.7% and 3.7%, for Claude-3.5 3.5--onnttt and GPT-4o respectively. For MathOdyssey, SBSC improves by as much as 12.6% and 7% over TIR-ToRA while showing improvement of 7.4% and 3% over its SC variant, for Claude-3.5-Someg & GPT-4o respectively. On OlympiadBench, for GPT-4o, SBSC matches SC results of TIR-ToRA and is better than the second best greedy variant by more than 5 5%. While for Claude-3.5-Sonnet, SBSC shows absolute improvement of nearly 6% and 3% over TIR-ToRA in greedy and SC setting respectively. Standard deviation values at A.13 We conduct additional studies, presented in appendix A.2 that show SBSC surpasses previous methods on two more additional challenging datasets JEE-Bench (Arora et al. 2023) and OmniMATH (Gao al. 2024). We also show in A.3 that SBSC is effective for open source model as well.
ABLATIONS & ANALYSIS
5.1 SENSITIVITY TO EXEMPLARS
70
50.10%
Yh
40
20
76.00% 27.705
20-
30
35
10
Number of Shots
AMC 3 Yrs
AIME 3 YrS
Figure 2: Effect of Number of Exemplars
Figure 3: Sensitivity to choice of Exemplars
We study the effect of number/choice examples in prompting on SBSC's performance using Claude-3.5-Somnee
subset of AIME and AMC data. As shown in Figure 2 increase in performance when increasing the examples from to 4, which then starts to saturate as decision of using 4-shot
we observe notable
further increase the number of examples to 6 and 8. This justifies setting. To understand if the choice of exemplars affect the accuracy or not, we conduct a sensitivity analysis. We randomly sample 4 exemplars out of the already created pool of 10 exemplars three

---

Published as a conference paper at ICLR 2025
times to create 3 variations of 4-shot prompts: v1, v2, and v3. In Figure 3 that the performance remains stable irrespective of the exemplars used.
5.2 SBSC EXEMPLAR TUNING
Natural language comments present within a program have proven to be useful (Gao et al. 2022). So, in each of the SBSC exemplars, we provide suitable comments in natural language within the Python program for each turn to help guide the model.
Table 2: SBSC performance comparison across prompt variations using Claude-33 3.5 3.5-Sonne.
Full Prompt
Without Comments
Without Line
AMC Yrs
67
62
60
AIME Yrs
27
19
16
For few-shot learning, apart from relevant exemplars, the LLM also benefits from general instruction
at the beginning (Zheng al. 2024] Gou et al.] 2023 Wang et al. (2023a) that provides guideline or context about how model should approach the task, particularly those requiring logical reasoning,
multi-step operations, etc. This can be specially useful when the task requires more nuanced
understanding and when the instructions need be followed rigorously, as the case with SBSC. Kindly refer to A. and A.12 for detailed prompts.
In particular, we highlight one line from the instructions part of the prompt wherein, the model is specifically being instructed invoke a code rectification step ensure that the error is not propagated further, leading to a wrong answer. It also ensures the model focuses only on the intermediate step. If the executed code snippet returns error, use it to correct the current step's 's code snippet. DO NOT restart solving from Step 1.
In Table 2,
study the importance of these components in particular: the comments within the code snippets and line mentioned above. Our findings suggest that removal of either these components lead significant decrease the performance, indicating how each of them are crucial aspects of our exemplar prompts.
5.3 CODE DEBUGGING ABILITY
60
TIR-TRRA
S85C
20
40
Accuracy
Error Problom
Figure 4: Comparison of Debugging Abilities
We present the superior ability our method to resolve an error related code execution. If at any step of the trajectory chain, the program returns an execution eтror, we consider that to be an error step. We visually represent this, using Claude-3.5-3 3.5-Sonet responses across AMC, AIME and that SBSC is able to recover from even multiple wrong steps and reach the correct final answer quite easily when compared to TIR-ToRA whose performance drops steeply increasing error steps. This be attributed to the fact that SBSC, being precise and granular, tackles only a focused part of the problem and finds it easier to correct its mistakes compared to T which tries correct the program at the problem level.
MathOdyssey datasets in Figure 4 where
7

---

Published as a conference paper at ICLR 2025
5.4 TOPIC-WISE ANALYSIS
TIR-TOR mIJA
00.00
00.00
69.90
10.00
Aggh
Donnay
ANC
MITH COVSSRV
DATASET
Figure 5: Topic breakdown analysis
We GPT-4o-mini (OpenAl June, 2024)
classify problems from AIME and AMC, while MathOdyssey and OlympiadBench already contained topic labels. Our test set primarily comprised of: Algebra, Arithmetic, Combinatories, Number Theory and Geometry. In this study, we benchmark the solutions obtained using Claude-3.5-Sonnet As can be seen in Figure 5 our method outperforms COT & TIR-ToRA (against both greedy and self-consistency decoding) in all the individual topics and across all the 4 datasets, thereby proving beneficial for all topics. This highlights the generalisation ability of our approach extending different types and complexities of problems.
5.5 SBSC ACCURACY CORRELATION WITH CODING CAPABILITIES OF LLMS
We study the correlation of code related capabilities of the LLMs with respect to their success with SBSC. Since coding capabilities of a model pivotal towards successfully following and executing our SBSC approach, we make comparison involving LLMs with varying coding abilities. Figure
6 shows that the SBSC scores are correlated to the code generation abilities the corresponding models for all cases that were evaluated subset of AIME and AMC data. The code- gedereenerttoon scores were taken from LiveCodeBench (Naman Jain 2024 benchmark.
AMC Yrs
Claude 3 3.5-Sonnet
GPT-4o
Accuracy
Couuee3
Claude-3 Claude-3.5-5omel
Claude-J-Soonnet
GPT
Claude So-onnet
Coude:3
Claude 3 Halku
20
25
30
40
45
50
LiveCadeBench (Cutoff: 9/1/2023 9/1/2024)
Figure 6: SBSC accuracy correlation with coding ability of L.MSs
8

---

Published as a conference paper at ICLR 2025
5.6 SBSC SELF- SELF-CONSISTENCY
Self-consistency (SC) decoding Wang et al. 2022) has proven to be effective in boosting accuracy via sampling multiple chains and taking majority voting. We employ SC decoding assess the upper bound of our approach. For this study, we use temperature=C... and top_p=0.7.
SISC SBSC m
57.14
Accuracy(%)
AMC
AIME
Figure 7: SBSC scores with Self-Consistency (maj @ (7)
We generate chains using Claude-3.5- 3.5-onnntt for each problem of last years of AMC and AIME; and consider the majority voted answer the prediction to be compared against the ground truth. We notice from Figure that the maj@7 7 accuracy higher than that of greedy decoding, following the usual trend with other prompting approaches like COT, PAL, etc.
More Ablations in Appendix:
We present average cost per question analysis in A.4 and show that our main results Table already accounts of token normalized comparison.
We additionally show, in A.5 that exemplar preparation for SBSC, like previous methods, did not require any manual effort apart from format inspections. We also create a larger pool of AIME exemplars show the stability and generalisability of SBSC by sampling 9 sets of exemplars and reporting accuracy on last 3 years AIME and MathOdyssey questions in A.5
We also present study on sympy usage by SBSC and TIR-ToRA in A.7 We show that sympy usage consistent in both the methods, however SBSC achieves higher accuracy if we compare the accuracy among those questions.
6 RELATED WORK
Recently, significant advancements across various research directions have been made enhance the mathematical capabilities of large language models (LLMs). One of the major ones has been along the prompting and thinking strategies such as Chain-of-Thought (COT) method (Wei et al.) 2022, Kojima et al., 2022) that has shown to evoke multi-step thinking in LLMs before arriving at the answer. These methods struggle with complex and symbolic computations. For this, PAL (Gao et al. 2022 & POT (Chen et al. 2022) suggest making LLMs perform reasoning by writing program and offloading the computations code interpreter. TIR-TORA (Gou al., (023) does rationale generation in one go beforehand and then it codes the entire solution to get the final answer using
single code block. Additionally, in case of error, tries to re-attempt similar format. Another line of research has been around pre-training and supervised fine-tuning (SFT). Multiple studies
(Shao et al. 2024| Ying et al. 2024 DeepSeek-A et al. 2024 Azerbayev et al. 2023 Lewkowycz
al. 2022 Paster al. 2023 Taylor et al. 2022) have shown pre-training LLMs on high-quality
maths tokens results in increased mathematical knowledge and reasoning abilities. Recent approaches
(Yu et al. 2202bb] Gou al. 2023 Yue et al. 2023 Wang et al.] 2023a Shao al. 2024 Toshniwal
(ct al.] 2024: Mitra et al. 2024 Beeching et al.] 2024 Yin et al. 2024 Tong et al. 2024 have tried
query/problem augmentation along with creating synthetic reasoning paths/trajectorie using teacher

---

Published as a conference paper at ICLR 2025
model like GPT4 (Achiam al. 2023 for SFT. These methods showed significant improvement in the math reasoning abilities of the model. Also, some studies (Wang et al. [223b]] Yu et al. [2023a] Xi
et al. 2024 Chen et al., 2024 Lightman al. 2023b) provide alternative to manual annotations
for process supervision Lightman et al. 2023a).
Previous program generation-based methods such PAL. & TIR-ToRA have followed static approach and hence struggle with complex problems. These methods typically attempt to solve the problem in single, large code block with minimal iterations(nn case of TIR-TORA). But they lack producing/exploring any intermediate steps. SBSC dynamically decomposes complex problems into sequence of manageable sub-tasks. SBSC embodies exploration by discovering new sub-tasks basis the previous sub-tasks. SBSC is highly focused as these sub-tasks are explored individually via program generation to generate intermediate outputs. SBSC has superior step- level self-correction abilities. Hence due to these major advantages, SBSC achieves significant improvement over TIR-ToRA & PAL across 4 different benchmarks as can be seen Table
CONCLUSION
We introduce SBSC, a multi-turn math reasoning framework that tries to enable LLMs to solve complex math problems. SBSC pursues the solution, step-by-step with each turn dedicated step, and arrives at final answer via multiple turn.. At each turn, an intermediate sub-task and its corresoonding program solution is generated leveraging the exccution outputs and solutions of all the previous sub-tasks. We show performance improvements of SBSC over TIR-ToRA, PAL & COT on challenging math problems. We also show that greedy-decoding results of SBSC outperforms self-consistency results of other prompting strategies.
8 FUTURE WORK
Given the detailed, dynamic and flexible step-wise nature of problem-solving along with the fact that its leverage program generation conclude a key-intermediate step. we believe SBSC reasoning format could be highly useful for guided decoding strategies such in Dutcome-Supervin Model (Yu et al. 2023a), AlphaMATH (Chen et al. 2024), Q* framework (Wang et al. 2024). would be well suited for step-wise preference optimisation for reasoning such as in (Lai et al. 2024).
Value
SBSC trajectories could be used also for imitation learning via SFT.
REFERENCES
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, al. Gpt-4 technical report. 2023. htttpp::/pp...semattttss wtw.:ccccccccccc.o........o
I
Anthropic. Introducing claude 3.5, 2023.
Fed9cc1
Addendum. pdff
Daman Arora, Himanshu Gaurav Singh, and Mausam. Have Ilms advanced enough? challenging problem solving benchmark for large language models. ArXiv, 150744 2023. URL
https://api www.nticsccoo...ooo.m.m ott//oorpusD::258666668.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv, 10.231110.0631,, 2023. URL httpt::///// api www.stiisscoolla.tt..
org/Corpus
Edward Beeching, Shengyi Costa Huang. Albert Jiang, Jia Li. Benjamin Lipkin, Zihan Qina. Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 7b tir. https naduminnaaatt 7B-TTR 2024.
//huggingfacce
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel
10

---

Published as a conference paper at ICLR 2025
Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot leameer. ArXiv, abs/2005. 14165, 2020. URL httts://app: www.seeaaticcccccc ar www.oo/////rooo// 1118177778
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. ArXiv, abs/2405.0355 2024. URL https://api semant
org/CorpusID:: 266605444
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res., 2023, 2022. URL BRLbttts://app www.seeaaanniteeeeeeeeeeeeiieiie cholar http:///cccccoooooooooooooospppo 2233811775
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha l'syyascchekko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana. Pillai, Marie Pellat, Aitor Lewkowyzz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstemn Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, 2aw/22040023111 2022. URL bttps: semant wwtttcicchhhaa.. wwwp/ooppsID: (49.0ppp
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021. URL
https://api. www.ettetttii.. www.httnnccchooooooo ar. www.aar.rrrrrroreeeeee ID:
DeepSeek-AI, Deepseck-v2.5: Combines deepscek-v2-chhe and
2024.
DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang. Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao. Bing-Li Li Wang, Jun-Mei Song, Deli Chen, Xin Xie, Kang Guan, Yu mei You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Breaking the barrier of closed-source models in code intelligence. ArXiv, abs/2406. 1931, 2024. URL /appi.p..... semanticschi a
Fuli Luo, and Wenfeng Liang. Deepseek-codert
::: 270562123
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmark- ing mathematical problem-solving skills in large language models using odyssey math data. ArXiv, abs/2406. 18321, 2024. URL hattps://appi 27C737739
Bofci Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- math: A universal olympiad level mathematic benchmark for large language models, 2024. URL
https://arxiz: or..orgg.a. abs/ 2410 0. 07985
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program- language models. ArXiv, Abs/2211.114433 2022. URL
https://api. semant icscholarcrrr. http://aapp..enntnicccc........
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. ArXiv, abs/2309. 17452, 2023. URL https: :ttttp::::/p./ /app. www.nt wttp/orppssDDDD 263310335
11

---

Published as a conference paper at ICLR 2025
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. ArXiv, abs/2402. 14008, 2024. htttps/t: .emannt ccsho a DTc/Coorpus1D:2677700
Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03874, 2021. URL https: semant cschol ar. rouus
2321 34811
Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey -part 2: Surpassing ol-preview through simple distillation, big progress or bitter lesson?, 2024. URL httphttss//axi...oo//
abs/2411 6489
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://apl
semanticscholar: htt.://aa...a.rooooppo00:0/11110
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of Ilms, 2024. URL httph::t//aarttt
org/abs/ 2406. 18629
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative rea- soning problems with language models. ArXiv, abs/2206. 14858, 2022. URL https://api semanticscholl .ww...ogg//ee wwt.:oggggoooo.ooo Corpus ID: 2501 44408
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. 2023a. URL
https://arxi:: http://..w.rg.aa/////.22222
2305.20051
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv, aws//a/5./2aaae 2005000.2200,,...0 2023b.
258987659
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. ArXiv, Abs/2400.148800 2024. URL https://api
semanticscholl
Cttp/orppsIII 2 267897618
Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar- Lezama Koushik Sen Ion Stoica Naman Jain, King Han. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint, 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Seratchpads for intermediate computation with language models, 2021. /t/.atppp.///xxi..oo//a/////////
00114
OpenAl. o1, 2024. URL htttp:://ppeai.cooo ol/
OpenAl. "hello gpt-4o.", June, 2024.
URL
hello gpt-4o7
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. ArXiv, abs/2310.06786, 2023. URL https:
7/api. www.sseeaanncccc.ccooooo
www.or///oooo..o Arrg/oorpusD.... 22I.D:::3333665
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. 01 replication journey: A strategic progress report part 2024. URL (tRl.tttts://// wwt.:/o..o///./....e./
18982
12

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap. Jean- Baptiste Alayrac, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, abs/2403.05550, 2024. URL httphhht///// htaps:///ppp ott.pttttee////oooeooooooooooooo
semanticschoaar.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024. httthttt:://ett http:://app..eemttticcicce.eeeee www.ntiitssciii.i
org/CozpusID::
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshom, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv, abs//211.099/9 1.09085, 2022. URL [RLLhhtp: http///a....maaiiccccccccoaaaaaa
CorpusID:25355322
Yuxuan Tong, Xiwen Zhang, Rui Wang, Rui Min Wu, and Junxian He. Dart-math: Difficulty- aware rejection tuning for mathematical problem-solving ArXiv, htts/2407.13690 2024. URL
https://apl. semant iescchcslaas wtt.:ar..ror//ooooi:0::::::0
Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-II 1: 1.8 million math instruction tuning dataset. ArXiv, abs/2402. 10176, 2024.
htRt::ts:://app semant icscholar.
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q+: Improving multi-step reasoning for llms with deliberative planning, 2024. URL https
/arxiv.org/abs: /206. 14283
Ke Wang. Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in Iims for enhanced mathematical reasoning. ArXiv, abs/2310.03731, 2023a.
URL https
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Y.Wu, and Zhi- fang Sui. Math- shepherd: Verify and reinforee Ilms step-by-step without human annotations. ArXiv, abs/2312.08935, 2023b. Utt/https:/// /aatttps://ap..seee semantiicchhnccc
266209760
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. ArXiv, abs/2203. 11171, 2022. URL URL hnttt::/t:
semant cscho lar www.olar..rrrr/ooooo org/Corpus TD: 7595263
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11900 2022. URL inttps://app wemanticschcee arr. org/ Corg//orpsssD 246411621
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, Honglin Guo, Wei Shen, Xiaoran Fan, Yuhao Zhou, Shihan Dou, Xiao Wang, Xinbo Zhang, Peng Sun, Tao Gui, Qi Zhang, and Xuanjing Huang. Training large language models for reasoning through reverse curriculum reinforcement learning. ArXiv. abs/2402.0588 058008 2024. htt.:htts:///e
semant cschola wwwpoollllrrrgg/ooooooeooo :
Shuo Yin, Weihao You, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath-code Combining tool- use large language models with multi-perspective data augmentation for mathematical reasoning. ArXiv, abs/2405.07551, 2024. URL httpt:tt////pp/
demanticschor warcchoola.... www.chooo......r//rp............ org/Corpps..r..
269756851
Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong. Kuikun Liu, Ziyi Wang. Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang. Kai Chen, Open math large language models toward verifiable reasoning. ArXiv, abs/2402.06332, 2024. URL /api. commantisscooli 26761 667617098
and Dahua Lin.
org/CorpusID)
13

---

Published as a conference paper at ICLR 2025
Fei Yu, Anningzhe Gao, and Benyou Wang. Ovm, outcome--uupervised value models for planning in mathematical reasoning. In NAACL-HLT, 2023a. URL wtt.:/ppp...mnntccccc...........
ert/CorppusD:22522000
Long Long Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zheng Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv, abs/2309. 12284, 2023b. URL https://app
semant i ccscholast lar. wttp/orroooossseo TD: 262084054
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv, abs/2309.05653, 2023. URL URL https::///ant https///ap..smanisssiiiieee sewanticscholaatt.... www.aaorrgoooooooooooaaaa 261696697
Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng- Heng-Tze Cheng, Ed H. Chi, Quoe V Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models, 2024. URL htttt:t//taxiiiitt www.rriwi.oor.rro.e. abs 231 06117
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Olivier Bousquet, Quoe Le, and Ed Huai hsin Chi. Least-to-most prompting en- ables complex reasoning in large language models. ArXiv, abs/2205. 10625, 2022. URL
https://api www.nticsccooolaaa...oooooom http:s//.ap...matttccccooo...... :ttp:/oopps1:::2:30000
A APPENDIX
A.1
NUMBER OF STEPS IN SBSC
In Table 3
we present the number of turns taken per question by SBSC responses obtained using Hlaude3.5-Soonee across the different datasets.
Table Table showing number of turns/steps used by SBSC
Number of turns steps
AMC
AIME
MathOdyssey
21
12
8
57
19
101
47
19
79
51
63
43
28
41
43
14
42
31
10
12
18
others
59
66
23
Average turns steps/Problem
6.0
6.9
6.4
A.2 RESULTS ON JEE-BENCH AND OMNIMATH DATASETS
We present results on additional benchmarks:
(Arora al. 2023) and Omni-MATH
(Gao et al. 2024). For JRF-Bench,, we evaluate on 98 numerical answer type questions. For Omni- MATH, we filter out 576 questions out of that with all 26 Qs from calculus topics and 110 Qs randomly sampled from each of the remaining topics. We AIME exemplars for evaluation these two datasets to also further show how generalizable SBSC is.
For JEE-Bench in Table we observe significant improvements compared to other methods. For Claude 3.5 Sonnet, SBSC's greedy decoding takes an absolute lead (against greedy decoding) of 16% over TIR-TORA. 20% over PAL and over 30% COT. Similarly benchmarking against self-consisted offfissnist decoding (majority @ 7) of other methods, SBSC takes an absolute lead 11 % over TIR ToRA, 16% over PAL. and over 25% COT.
14

---

Published as a conference paper at ICLR 2025
Table 4: Benchmarking accuracy on JEE-Bench dataset.
Model
COT
PAL
TIR-ToRA
SBSC
greedy
maj @
greedy
maj @
greedy
maj @7
greedy
maj @7
Claude 3.5
Sonnet
32.65
37.76
42.86
46.94
45.92
51.02
62.24
"1+11.22
GPT-4o
29.59
33.67
35.71
38.78
32.65
37.76
50.00 +14.99
71+11.22
On Omni-MATH dataset in T SBSC(greedy decoding) shows absolute improvement over TIR-ToRA (greedy decoding) by roughly 10% using (laude-3.5-Sonne and 6% using GPT-4o. SBSC greedy decoding results outperforms self-consistency decoding results of TIR-TORA by absolute 6% and 4%, for Claude-3.5-Soman and GPT-4o respectively.
Table 5: Benchmarking accuracy on OmniMATH dataset.
Model
COT
PAL
TIR-ToRA
SBSC
greedy
maj @7
greedy
maj @
greedy
maj @7
greedy
maj @7
Claude 3.5
Sonnet
23.09
26.22
32.81
35.94
33.68
37.15
43.06-49.38
"145.91
GPT-4o
27.78
29.86
34.38
37.15
33,85
36.28
40.45-+6.0r
1+3.40
A.3
BENCIMARKING OPEN SOURCE MODEL: DEEPSEBKCODER V2.5
We performed benchmarking with DeepSeekCoder 2.5 DDepSeek- Al 2024) which an open source model. Note: given the resource constraint and that most of the frontier open source LLMs such as DeepSeekCoder 2.5 require multiple gpus and days to host and run the experiments respectively, we leveraged this open source model via api provided by deepseek which was highly cost-effective. We present the results on 2 test-sets: AIME and AMC as used in the main results.
We can clearly observe in Table that SBSC outperforms both greedy decoding and self-consistency decoding results of TIR-TORA. PAL & COT on both last 11 years AIME & AMC.
Table 6: DeepSeckCoder 2.5 accuracy for SBSC, TIR-ToRA, PAL & COT on last 11 years AIME & AMC test-sets.
Model
COT
PAL
TIR-ToRA
SBSC
greedy
maj@7
greedy
maj@) @7
greedy
maj@7
greedy
maj@1 @7
AIME
8.48
10.30
18.79
20.00
19.70
20.61
23.64
"f43.03
AMC
24.84
26.53
29.47
30.53
31.16
32.42
34.53 +33.37
"1+2.11
A.4
COST AND TOKEN COMPARISON ACROSS SBSC, TIR-TORA, PAL & COT
We also have done study with Claude 3.5 Sonnet to compare the average cost per question. We have taken average over 50 AIME+AMC questions. We leveraged the prompt-caching of repeated input tokens while doing this experiment. Specifically, we cache the system prompt and exemplars for all the methods. The results are presented in the table below. As shown the comparison in Table SBSC is 3X of TIR-ToRA, 4X of COT and 7X of PAL. When comparing average cost per question even in non-caching mode, from Table & we observe that SBSC is 6,2X of TIR-ToRA. 7X of COT and 14.3x of PAI. Hence, Table already
fair comparison part well even if take
15

---

Published as a conference paper at ICLR 2025
average cost per question under consideration. In Table 1, already compared I chain of SBSC with chains of other methods. Just as experimental detail, we would like to mention we leveraged prompt caching for our reported experiments with Claude 3.5 Sonnet.
Table 7: Average Cost per Question for different methods using Claude with Prompt Caching.
Claude with prompt caching
COT
PAL
TIR-ToRA*
SBSC*
Input tokens
110.75
107.75
479.25
4788.90
Output tokens
543.95
353.20
685.15
1070.00
Cache tokens
2754.00
1255.00
2692.20
23610.60
Cost/Question ($)
0.0093
0.0060
0.0125
0.0375
Cost ratio (SBSC/Methods)
4.00
6.25
3.00
1.00
SBSC & TIR-ToRA values represent the sum of all turns per question
Table Average Cost per Question for different methods Using Claude without Prompt Caching.
Claude without prompt caching
COT
PAL
TIR-ToRA *
SBSC*
Input tokens
2858.75
1356.75
3565.45
32846.70
Output tokens
536.05
267.35
543.70
1161.50
Cost/Question (S)
0.0166
0.0081
0.0189
0.1160
Cost ratio (SBSC/Methods)
7.00
14.30
6.20
1.00
SBSC TIR-ToRA values represent the sum of all turns per question
In Table 9 present the average token count per question for exemplars and system prompt across all the methods. We would like to highlight that due to prompt caching mode available in LLMs, the entire prompt is charged only once.
Table 9: Token Usage Comparison Across Different Methods (Note: All values are in tokens)
Dataset
Method
COT
PAL
TIR-ToRA
SBSC
AIME
gystem_prompp
11
42
123
208
exemplars
2347
1049
1588
4012
AMC
system_prompt
11
42
123
208
exemplars
1557
1013
1554
3486
In case of SBSC, the last turn requires the largest context length as last turn(n) input consists of system prompt exemplars question output trajectory from 1 to n-1. We present the study in Table I0
Table 10: Token Analysis of System Prompts and Trajectories (Note: All values in tokens)
Dataset
System Prompt + Exemplars
Trajectory n-1
Total
AIME
4,220.00
899.20
5,119.20
AMC
3,694.00
835.85
4,529.85
16

---

Published as a conference paper at ICLR 2025
A.5
EXEMPLAR SENSITIVITY ANALYSIS WITH LARGER POOL
A.5.1 BRIFF ON EXEMPLAR CURATION & CRAFTING AND WHY IT DIDN'T REQUIRE MANUAL EFFORT
To enable the multi-turn reasoning for SBSC, we use combination of system prompt and few-shot prompting (exemplars) similar to what POT, PAL, TIR-ToRA used. Similar to past works, our exemplar generation process didn't require any manual effort other than writing the system prompt and manually inspecting the final pool for format check.
To prepare the exemplars,
We first used O-shot prompting (system prompt) to generate responses for questions in the SBSC format. As mentioned in 3 "Prompting/eewSSi Exemplars", for both AIME and AMC, We select 90 questions each, drawn from problems of years other than those included in the evaluation datasets.
We observe that out of 90, we only had 15 questions that were solved across the methods (COT, PAL, TIR-ToRA, SBSC). This churn was particularly due to the low accuracy of COT method.
A pool of 10 common problems and corresponding exemplars were finally selected. In this step, we ensured the diversity in the problem set (we used GPT-4o to get the question categories) and ensured that SBSC format was accurately followed in them. The distribution was: 3 Algebra, 3 Number Theory, 2 Geometry, 2 Combinatorics.
However, the selection of exemplars from the is completely random.
Additionally, as mentioned in section 2. we also ensure certain practices (mainly 2) for program generations in the exemplars which are inspired from PAL paper. First to have meaningful and coherent variable names and second is to having natural language intermediate comments within the program. We observed that the base LLM followed the first point (meaningful variable names) by default. For the second point, which is the natural language comments within the code, we used GPT-4o to add comments within the generated programs for the final 10 curated exemplars. PAL paper already established (and used) these two things improve the performance. We even measured the impact of the comment based exemplar tuning in section 5.2 and Table
The only thing manual in this entire exemplar curation process was writing the system prompt and final inspection if SBSC format was correctly followed not.
A.5.2 ADDITIONAL EXEMPLAR SENSITIVITY & GENERALIZATION ANALYSIS WITH BIGGER POOL
We conduct an additional study expand the pool of exemplars for sensitivity analysis as suggested by the reviewer. We expand the pool to 30 AIME exemplars (prepared in the same manner as made initial 10 exemplars as mentioned above and in section 3.3 "Prompting/Few
Exemplars") that were solved where we have Algebra, 8 Number Theory, 8 Combinatories and 6 Geometry questions. We randomly sample 4 exemplars from these 30 exemplars times to create 9 sets of 4-shot prompts: v1, v2, v3, v4, v5, v6, v7, v8, v9. We test them on the same set of last years AIME questions.
For further testament to our generalisation, we use AIME exemplars on MathOdyssey test-set.
We clearly observe in Table [] that the performance AIME remains stable, experienced in previous study in figure 3 with exemplar-set randomly sampled from 30 AIME Qs. Also, from Table 12 we can see the performance on MathOdyssey stays stable with even 9 AIME exemplar sets.
We have utilized AIME exemplars for benchmarking SBSC on MathOdyssey I OlympiadBench I JEE-Bo Bench Omni-MATH[[ We can observe on all these datasets. SBSC performances much better than other SOTA methods. This proves the generalization of SBSC and also proves that there has been over fitting of exemplars done.
17

---

Published as a conference paper at ICLR 2025
Table 11: Last YrS AIME Accuracy for Different Exemplar Combinations
Exemplar Variations
Last 3 Yrs AIME Accuracy (%)
vl
31.11
v2
32.22
v3
32.22
v4
33.33
v5
32.22
v6
30.00
V7
28.88
v8
33.33
v9
28.88
Table 12: MathOdyssey Accuracy for Different Exemplar Combinations
Exemplar Variations
MathOdyssey Accuracy (%)
39.86
38.51
v3
37.17
v4
38.51
41.89
V6
37.84
V7
39.86
v8
39.86
49
37.84
A.6 UNDERSTANDING SBSC IN DETAIL.
In this section, we demonstrate some scenarios where SBSC has been successful while TIR-ToRA has failed, with the help of some example questions and investigating the responses obtained from the two models.
Let's consider the question in Example involving a geometric progression of numbers written in logarithmic form, which TIR-ToRA gets wrong.The method uses binary search technique, which not very precise when dealing with exact values required for mathematical problems, especially when fractions are involved. The solution uses function to check whether the logarithms form geometric progression which introduces additional complexity and potential inaccuracies because writhmetic.Also, this
involves comparing ratios that may not be exactly equal due to floating-pon single-turn method tends overlook specified constraints or necessary simplifications, which are often encountered in Olympiad level problems and instead makes false assumptions.
The question in Example 2 is an example scenario where TIR-ToRA fails because it makes an incorrect assumption. It misinterprets the Lipschitz condition and incorrectly makes simpler assumption that the difference f800) - f(400) is equal to the maximum possible difference, which is 200. While the magnitude of the difference is bounded by 200, it does not mean that the actual difference will always be 200. Iterative solutions, as are often the only way out single program based solutions, not clearly defined or understood by the LLM.
sometimes lead to infinite loops, especially in cases where the stopping condition
As can be in Example 3 the single code is unable to take advantage of the factorization of 20
18

---

For the problem in Example the first sub-task correctly identified by SBSC factorize the large number 2020 into smaller powers, paving the path towards an efficient solution. In the subsequent steps, it takes care of constraints like ensuring that m and n are integers and avoids over-counting by properly iterating Likewise, in Example 4 we see that our method generates each term of the sequence one at a time, allowing for intermediate checks to confirm whether each term is of the desired form f Thus, it accurately identifies the terms that meet the required condition and ensures that only the correct Published as a conference paper at ICLR 2025
which is key to solving the problem efficiently and instead iterates over very large range of potential values for m, leading to inefficiency. The upper bound 2020 is extremely large and the sheer number of iterations causes timeout.
Example presents a scenario where TIR-TORA makes up an assumption about the problem and writes the code for terminating loop accordingly, which leads to a timeout error, the incorrect assumption leads to an infinite loop. It lacks intermediate checks that would provide insights into whether the sequence terms of the form T which is crucial for solving the problem and would have enabled it to chalk out the termination conditions suitably.
On the other hand, our Step-By-Step Coding method enforces a decomposition of the problem into smaller sub-task. Each sub-task is tackled independently by the L.M,, which generates code solve it and then uses the resulting output to suitably proceed to the next sub-task and this process continues till the final answer is reached. Such an approach ensures that every part of the problem is addressed with exact precision, reducing the risk of errors that might arise from skipped steps. Dividing the problem into multiple sub-tasks also allows make necessary simplifications that would make the future tasks, and hence the entire problem, easier to solve.
Going back the problem in Example [ SBSC starts by defining the logarithms and setting up the equations based on the geometric progression condition. It then simplifies the equations reduce them to a more manageable form, climinating unnecessary complexity and allowing straightforward solving. Throughout the problem, it uses precise mathematical formulations of the problem, ensuring the solution is accurate. Since this method isn't trying to solve the entire problem at one go, doesn't need make any assumptions to simplify the problem statement.
For the question in Example[2 it correctly interprets the problem, keeps applying the given Lipschitz condition as it solves each sub-task and finds the correct maximum possible value of (((0))))) (((())))) By systematically checking for constraints and edge cases at each stage, our method guarantees that solutions are not only accurate but also valid under all possible conditions. This is particularly crucial for problems with intricate conditions or multiple solution paths. Additionally, our process explicitly defines loop conditions ensure that termination criteria are clear, allowing for correction if issues like infinite loops arise.
valid powers, giving the correct answer.
indices are considered. This avoids the infinite loop risk and ensures the correctness of each step before moving We also notice that the model runs into error at the first sub-task itself but based on the corresponding code output, it able to take an alternate approach and successfully solves the sub-task, moving onto later ones. This example also highlights superior code debugging ability, as discussed earlier.
Example Question 1: There is a unique positive real number such that the three numbers logg 2x, log, andlog, x in that order, form geometric progression with positive common ratio. The number x can be written as where and are relatively prime positive integers. Find m +
Incorrect solution by TIR-TORA
Let's approach this problem step-by-step:
1. Define function to check if three numbers form geometric progression.
2. Use binary search to find æ that satisfies the condition.
3. Convert & to a fraction and return the sum of its numerator and denominator.
import math
from fractions import Fraction def solution (): I Function to check if three numbers form
fa goonetric progression
19

---

Published as a conference paper at ICLR 2025
det is ss__gemmetrcc. ptp.ettricc.pooreesioool.aa (a, b, c)
ratiol - b / ratio2 = c b return math.iscloss (ratiol, ratio2) and ratiol >
Binary search to find
left, right 1, 1000
while right left > le-10:
x (left + right) 2
log8_20 moth.log (2*x, 8)
1og4_x log (x,
log2_x = math. (x, 2)
if ies__eemtrrcc_oorressttte
(log8_2x, 1og4_x, log?_x)-
break
elif (logl_x / log8_ 2x) (log2_x 1og4_x):
Left
eise:
right =
fraction frac Fraction (x) iimit_denominator ()
Convert
return frac.numeran + prac.denominacen
result = solution()
print (f") [boxed (resultt))")
>>> output:
\boxed|2;
The answer is
2
Correct solution by SBSC:
Let's solve this problem step by step using SymPy-based Python code. Step 1: Define the symbols and set up the equations
from sympy import symbols, log, Eq, solve
Define symbols
x, r symbols ('x r!)
Define the three logarithms
logl log (2+x, 8)
log2 log (x,
1og3 log ix,
Sct up equations ions for geometric progression
eql (1og2 / logl, r)
eq2 Eq (1og3 / log2,
print ("Equation 1:", eql)
print ("Equation 2:", eq2)
>>> output
Equation 1: Eq (1og (8) +log (x) / (Log (2xg Equation 2: Eq (1og (4) /1og (2), z)
Let's continue solving this problem step by step using SymPy-based Python code. Step 2: Simplify the equations
from sympy import symbols, log, Eq, solve, simplify
Define symbols
x, r synbols ('x r!)
# Define the three 1ogarithns
log1 log (2+x, 8)
log2 log (x,
log3 log (x, # Set up equation ions for geometric progression eql (1og2 / logl, r)
eq2 (log3 / log2,
20

---

Published as a conference paper at ICLR 2025
Simplity equations
eql_simpi fied simplify (eq1)
eq2_simplifiin fied a simplify (eq2) print ("Sinplliiii fied Equation 1:", egl_simllified
print ("sinplified Equation 2:", eq?_simplified)
>>> output: Simplified Equation 1: Eq(r, 3x1og (x) (2+1og (x) + 2+1og (2))
Simplified Equation 2: Eg(r, 2)
Let's continue solving this problem step by step using SymPy-based Python code. Step 3: Solve the equations
from sympy import symbols, log, Eq, solve, simplify
Define symbols
x, r symbols ('x r')
I Define the simplified equations eql Eq(r, (x) (2+1og (x) (21og(2)))
eq2 Eq(r, 2)
Solve the system equations
solution = solve ( (eql, eq2), (x, )))
print ("Solution:", solution)
output
Solution: ((1/16, 2)]
Let's continue solving this problem step by step using SymPy-based Python code.
Step 4: Extract the value of and convert to a fraction
from sympy import symbols, log, Eq, solve, simplify, Rational
# Define symbols
r synbols ('x (''x r!)
Define the solution from the previous step solution (Rational (1, 16), 2)
# Extract the value of x
x_value solution[0][0] print ("x =", x_value)
Convert x
fraction
x = x_value. denominator
print ("x as fraction: ()/1)". format (m, n
output
x = 1/16
as a fraction: 1/16
Let's continue solving this problem step by step using SymPy-based Python code.
Step 5: Calculate m + n
from sympy import symbols, log, Eq, solve, simplify, Rational Define symbols
x, synbols ('x r²)
Define the solution from the previous step
solution (Rational (1, 16), 2)
# Extract the value of
v_value |olution[0]10]
Convert x to fraction
- wevaluuedenomiat...
Calculate
result
print ("m =", m)
print ("n =", n)
print ("m =", result)
21

---

Published as a conference paper at ICLR 2025
output:
16
17
### END OF CODE The final answer is 17
Example Question Consider functions that satisfy [f(x) - f(y) 0.5 yl for all real numbers and y. Of all such functions that also satisfy the equation f(300) f(900), what is the greatest possible value of f(f(800) f(((40))))
Incorrect solution by TIR-TORA:
To solve this problem, we need to consider the properties of the given function and find the maximum possible difference between (800) and f(0())))) Let's solve using python:
def
solution ():
#Given: I (x) - [ <= 0.51x
for all real and
#Also, f (300) (900)
#Maximum difference between (800) and (400)
max_diff_B00_400 = 0.5 (800 - 400) 200
#Maximum value of (800) - (400)
max_f800_f440 max
200
#Maximum difference between fff (800)) and f(f (400))
max_finai_diff 0.5 max_f800_F400 100
return max_ max_final_diie
resuit = solution()
print (f") (boxed [[rsultt]]]]]]]]
>>> output:
\boxed 100;
The answer is 100
Correct solution by SBSC:
Let's approach this problem step by step using SymPy-based Python code where appropriate. Step 1: Understand the given condition The condition If(x) f(y) 1/2lx yl implies that f is Lipschitz continuous function with Lipschitz constant 1/2. This means f doesn't 't change too rapidly.
Step 2: Analyze the equation f(300)=f(900) This equation tells us that f takes the same value at x=300 and x=900. Let's call this value k.
from sympy import symbols, Abs
= symbols (*k') print (I"E = (900)) (ki")
output: f(300) f (900) k
Let's continue solving this problem step by step. Step Apply the Lipschitz condition f(800) and f(400) We the Lipschitz condition to find bounds for f(800) and f(400) in terms of k.
from sympy import symbols, Abs, Mi n, Max
= symbols ("k')
For (800)
8000 min
Abs (800 9CC) /2
8_000 max = + Abs (800 9CC) /2
For f (400)
f_400_min
Abs (400 32
22

---

Published as a conference paper at ICLR 2025
f_400_mas max + Abs (400 3C0)
print (f is between [f_800_mis and (f_800_max) print (f"F(430) is between { [f_400_min] and { [f_400_max]")
output:
f(800) between [(400) is between k 50 and k 50
and k 50
Let's continue solving this problem step by step.
Step 4: Analyze ff((800) and f(f(400) Now we need to apply the Lipschitz condition again to ((((8))) and f(((40)))
from sympy import symbols, Abs, Min, Max
= symbols (*k')
For (f
ff_800_min k Abs (k 5C
k) /2 Abs 5C
ff_800_max
/2
For (4000)
fE_400_me = k Abs (k 5C
k)
_E_400 max k Abs (k 50 k) print
is between {ff_800_min) and [ff_800_maxt ")
print (f00(
betwoen (ff_400_min) and {ff_400_max)
>>> output f(f(800) is between k 25 and k 25
((E(0) is between k 25 and 25
Let's continue solving this problem step by step.
Step 5: Calculate the maximum possible difference Now need to find the maximum possible value of ((((0))) - ((((40))
from sympy import symbols, Abs, Min, Max symbois ('k')
Maximun possible difference
max_diff (k + 25) (k 25)
print (f"The maximum possible difference E(E((00))-E(((40)))
is (max_diff)")
>>> output:
The maximun possible difference (f (f is 50
### END OF CODE
The final answer is 50
Example Question 3: Find the number of ordered pairs of positive integers (m, n) such that m 20
Incorrect solution by TIR-TORA:
Let's approach this problem step-by-step:
1. Iterate through possible values of up the square root of 2020
2. For each m, check if mn divides 2020 evenly
Return the total count valid pairs
import math
def solution on (): count - 0
target 20++20
for in range (1, int (math. (math.sqrt (target)) + 1):
if target $ (m**2) = 0: = target // (m**2)
count += 1
23

---

Published as a conference paper at ICLR 2025
return count
result sol solinion()
print (f the number of ordered pairs (m is: ["\ ( (result]))"")
>>> output:
TimeoutError: [Errno 110] Connection timed out
Looks like got timeout enror, let's try different approach that should not result in any error:
import math
def solut (n count
target 20++20 for m in range (1, int (math. (target)) + 1): if target & (m**2) == 0:
target (m**2)
count +=
return count result = solution() print (F"The number of ordered pairs (m, is: f" "Wllboxed; (result]])"") (resul(](r)sll]]))
>>> output:
TimeoutError: [Errno 110) Connection timed out
This loop continues without ultimately getting answer.
Correct solution by SBSC:
Let's approach this problem step by step using SymPy-based Python code. Step 1: Define the equation and initialize variables
from sympy import symbols, Eq, solve, divisors, prime_factonn m, n synbols I'm n', positive-True, integer=True) equation Eq (m**2 * n, 20+=20) print ("Equation:", equation)
output:
ImportRrror: cannot import name 'p=ime_factoor! from 'sympy'
apologize for the error. It seems the 'prime factors' function is not available in the version of SymPy you't 're using. Let's modify our approach and use the 'factorint" function instead, which is more commonly available. Here's 's the corrected code: Step Define the equation and initialize variables
from sympу import symbols, Eq, solve, divisors, factorint m, synbols ('m n', positive=True, integer=True) equation Eq (m**2 n, 20xx20) print ("Equation:", equation)
>>> output
Equation: Eq 2
Let's continue solving this problem step by step.
Step 2: Factor 2020 and analyze its structure
from sympy import symbols, Eg, solve, divisors, factorint m, n synbols ('m n', positive=True, integer=True)
equation Eq (m**2 n, 20xx20) Factor 20-20
factorization factorint (20..20)
24

---

Published as a conference paper at ICLR 2025
print ("Factorization of 20"20:", factorization) Aza the structure
power_of_2 - factorization [2]
power_of_5 factorization [5]
print ("Power of 2:", power_of_2) print ("Power of 5:", power_of_5t
>>> output Factorization of 20^20: (2: 40, 5: 20
Power of 2:
Power of 5: 20
Let's continue solving this problem step by step. Step Determine possible values for m and
from sympy import symbols, Eq, solve, divisors, factorint synbols ('m n', positive=True, integer=True)
equation Eq (m**2 n, 20*+20)
Factor 20^20
factorization = factorint (20++20)
power_of_2 = factorization[2]
power_of_5 factorization
Initialize counter for valid pairs valid_pairs
# Iterate through possible values of m for m__ower_of__ in range powez_ož_2 + 1, 2): for m p__powrr_of__ of_5 in range power o 1): Calculate
n_value 2<m__power_f__2 5 S
Calculate no
p_weer__o___ power_off
2 power_of of_2
p_ooer_oo__ power_of of m power_of_S
Check if is positive integer if n_ppoer_of_2 >= and n_power_of_5 >= 0: palid_pairs +=
print ("Nunber of valid (m, n) pairs:", pairs)
>>> output
Number of val id nm pairs: 231
#### END OF CODE
The final answer is 231
Example Question 4: Consider the sequence (ax)_k 2 lof positive rational numbers defined
by 01
2020
and for k 2 1, if ag = m for relatively prime positive integers m and n then Determine the sum of all positive integers j such that the rational number can be
2021
m+ 18
0x+1
n+19
written in the form T+T for some positive integer
Incorrect solution by TIR- ToRA:
Let's proceed step-by-step:
Generate the sequence fractions starting from a₁
2021
2. For each term, check if it is in the form #+1 for some integer t. If so, add its index to running sum.
Generate the next term using the given formula.
4. Continue this process until we cycle back to the initial fraction. The sum of indices where the fraction is in the form E4T is our final answer.
from fractions Import Fraction
25

---

Published as a conference paper at ICLR 2025
def next_term,
return Fraction nammumatto 18, a. a.denominator + 19)
def flLo__form_t__vrr_____.l.s (fraci
(frac)
return frac.denominatoor frac.numerator
def solution
Fraction (2020, 2021)
i_sum
==
while Truc:
j_sum += )
next_term(a)
Check we've reached the initial fraction again (20cii 2021):
break
return sum result solution ()
print (f") (boxed (result)))))
Similar to the earlier example, this code results in a TimeOut error and then the loop keeps repeating.
Correct solution by SBSC:
Let's solve this problem step by step using SymPy-based Python code. Step Define the sequence and initial value
from sympy import Rational, ged
def next_termaa)
m, n = a aunmmrrtorr, a.derominator
return Rational (m + 18, - 19)
[Rational (2020, 2021)]
print [f"a[1] [a[0]]
>>> output
[[1] 2020/2021
Let's continue solving this problem step by step using SymPy-based Python code. Step Generate the sequence until we find term of the form t/(t+1)
from sympy import Rational, ged
def next_term
m, n a. aummerator, a.denomi nator
return Rational (m + 18, 19)
[Rational (2020, 2021)
i ==1
result 1]
while True:
if a a[-1l denominator a[-1] .uueraaoor + 1: result.append
print (f"Found a[li]] (a[-1]} of the form t/ (t+1))
+= a. append (aext_term
if j 100: # Limit to prevent infinite loop break
print ndi found: (result)")
>>> output
Found all 2020/2021 of the form (++1)
Found a 1019/1020 the form t/ (=+1)
Found a[8 = 161/162 of the form (/+1)
Found a[18: = 33133 of the form t/ (t+1)
Found a a30] -99/20 of the form t/ (t+1)
26

---

Published as a conference paper at ICLR 2025
Indices found: [1, 2, 8, 18,
Let's continue solving this problem step by step using SymPy-based Python code. Step Calculate the sum of the indicesfrom sympy import Rational, ged
def next_term (a):
m, n = a. a.numerator, a.denoninator
return Rational (m + 18, 19)
[Rational (2020, 2021)]
==1
result = []
while True:
if a [[=11_denominattn =- a[-1] -umeeraotrtoorr + 1:
result. aepuut..
+-
a. append (next (att_ttermaaa_
if i 100: # Limit to prevent infinite loop break
sum_of_indico sum (result) print Indlndicos found: (result;")
print (f"Sum of indices: (sum indicoo))
output Indices found: [1, 2, 8, 18, 30]
Sum of indices: 59
### END OF CODE
The final answer is 59
A.7 COMPARISON ON SYMPY USAGE BETWEEN SBSC AND TIR-TORA
In this section, conduct study understand the usage of SymPy library by SBSC and TIR-ToRA for last 10 years of AIME and AMC questions, shown in Table 13 Specifically, we present the number of questions which these methods utilized that library, and we present the accuracy for those set of questions. The number of questions is which both the methods utilized SymPy are similar. For both AIME and AMC problems from the last 10 years, SBSC based solutions have used SymPy only on 7 more questions in both test-sets. But when we look at the accuracy (among these filtered questions). we can observe that SBSC had over 10% absolute accuracy lead in both the datasets.
Table 13: Comparison between TIR-ToRA and SBSC questions solved utilising SymPy
Context
AIME (#Qs)
AIME (Accuracy %)
AMC (#Qs)
AMC (Accuracy %
TIR-ToRA
200
15.50
299
36.79
SBSC
207
30.43
306
46.73
Table 14: Performance common questions involving Sympy Usage by both the methods
Dataset
# Common Qs
TIR-ToRA (Accuracy %)
SBSC
with Sympy Usage
(Accuracy %)
AIME
163
15.95
28.22
AMC
252
38.49
47.62
Also, we further filtered the common questions, for both the test-sets, where both TIR-ToRA and SBSC made use of Sympy. We present this study in the table 14 We observed that more than 80% questions are common for both the methods across the test sets. In table [4] compare the accuracy of both the methods over these common questionsand find that SBSC has 10% absolute improvement across both the test-sets.
27

---

Published as a conference paper at ICLR 2025
Table 15: Pass @a scores for SBSC and ol scores last 3 years AIME questions.
Topic
SBSC
pass @
pass @4
pass @9
pass 16
pass
preview
mini
Algebra (21 Qs)
11
14
16
16
19
12
14
Combinatorics (23 Qs)
8
10
12
13
13
10
Geometry (32 Qs)
2
3
4
9
9
10
13
Number Theory (14 Qs)
7
9
9
11
11
Total (90 Qs)
28
36
41
49
52
39
44
A.8 STUDYING THE POTENTIAL OF SBSC TRAJECTORIES AND COMPARISON WITH O1
SBSC achieves SOTA but still room for improvement in accuracy: For frontier LLMs such as Claude 3.5 Sonnet, we showed that SBSC surpasses not only greedy-decoding but also self- consistency decoding results of ToRA, PAL & COT. We also show topic wise analysis well to show the improvement is across the topics. While we do achieve SOTA, accuracy but there still room for improvement as evident from the scores. On last 11 years of AIME questions, despite being SOTA SBSC achieves 36% accuracy in pass So there question on how much can SBSC trajectory really solve more.
ol like complex reasoning methods achieves new SOTA for COT based math reasoning: So- phisticated reasoning systems like o1 (OpenAl 2024) perform "long thinking" to do potentially tree-search/self vew--caarrcckl---feffftcototttbn
over natural language reasoning chains to arrive at the final trajec- tory (Huang al. 2024 Qin et al. [2224] OpenAl 2024). They achieve significantly higher scores, on AIME and AMC, compared to previous COT scores by frontier L.LMs such as Sonnet and GPT40
and use human like COT reasoning.
Table 16: Analysis of Error Classes in Different Mathematical Domains for last YrS AIME
Error Type
Algebra
Combinatorics
Geometry
Number Theory
(21Q)
(23Q)
(32Q)
(14Q)
Reasoning Error
0
10
10
1
Hallucination
0
1
Spatial Misunderstanding
7
0
Incorrect Answer Reported
5
0
Total Errors
2
10
23
2
include mistakes due incorrect reasoning steps missing conditions. Hallucination refers fabrication of numbers answers. Spatial misunderstanding involves misinterpretatiin given diagrams or geometric scenarios. Incorrect Answer Reported refers to cases with correct reasoning solution but the final answer is wrong.
Error definition:
Reasoning
Understanding SBSC's potential with pass @ k metric and surpassing ol: We argue that SBSC given unique step-by-ster coding nature holds of potential. It unlocks dynamic problem solving via discovering and solving intermediate sub-tasks.
We evaluate this first via quantitative analysis by calculating pass@k scores on last 3 years AIME questions
We observe that SBSC's pass @ 16 and pass @5 are able to solve 48 and 52 questions out of 90 AIME questions respectively. Pass @25 SBSC score surpasses ol mini by 9% and ol preview by13%. We also do topic wise analysis to understand the strengths and weaknesses of SBSC.
28

---

Published as a conference paper at ICLR 2025
We further do enror analysis the 38 questions that couldn't it be solved with pass @25 SBSC. We do this across the math topics. We define different error types.
By doing error analysis, we are able understand that SBSC made majorly reasoning errors during program generation leading to incorrect code. Hence majority of those 38 questions can be solved by SBSC reasoning format if the model avoids certain reasoning error.
However, we also understand that specifically in Geometry questions, SBSC struggles to achieve ol level accuracy even at pass @5.. We mainly attribute this failure to diagram/spa- tial understanding.
Note/Scope of this study: This study is to understand the potential of SBSC and not about claiming COT like reasoning is not sufficient. We believe SBSC based program generation based method could also unlock advanced mathematical reasoning. We have already demonstrated this across four datasets in the paper and 2 more ( JJE--Bench and Omni-Math) in appendix. We try to show more evidences with this study. Given, ol like new complex reasoning systems unlocks the SOTA/superior COT trajectory, so we also perform comparison against them.
Analysis SBSC's pass @askkk k accuracy last 3 years AIME
Why use pass @k k for SBSC: Here we leverage self ceffisonsss strategy to make model generate multiple SBSC trajectories examine the pass k curve by increasing k from to 25. We are using pass @k k metric as we want to know if the model can generate correct SBSC based reasoning path for a particular problem. ol explores multiple paths/trajectories to arrive at the correct trajectory. Once it can help us better understand that when we build critic systems or thinking systems that leverage SBSC, how much can these systems achieve. Hence in this study we focus pass@k for SBSC to understand its potential. We also benchmark against ol preview and ol mini. We do pass@1 for the ol models they already do through slow/long reasoning internally before outputting the final correct reasoning path. Also since they are new SOTA COT based method.
analyse SBSC's pass @k for few values of k,
Experiment details: We do this study on last 3 years AIME questions. We cl5ude-3.5- sonnet as the base LLM. For self-consistency we use temperature 1 and top p = 0.95. ol mini and ol preview represent the current SOTA COT based methods.
Results: We present pass@1 1, @,, pass @ 16, pass@25 for SBSC in Table 15
- Overall analysis: We observe that SBSC with just pass@4 approaches 01 level accuracy and with pass @ 9 it surpasses ol -p-preveew and almost touches ol-mini. With pass@) 16 and p5 SBSC takes significant lead wI. ol models. At pass@ 16 SBSC shows 6% absolute improvement with best ol mini and 11% improvement over ol-preview. At pass @25 SBSC shows 9% absolute improvement compared to ol mini and 13% improvement over ol preview. We can clearly understand that SBSC can potentially unlock more correct chains and solve 52 questions out of 90
Topic wise analysis:
Algebra: We can observe that with just p SBSC achieves ol mini level performance and with pass @5 25 setting, SBSC solves 90% of the algebra questions * Number Theory: With just pass @ 1, SBSC surpasses ol preview and matches ol mini. At pass @ 16, SBSC is able solve with 78% accuracy.
* Combinatorics: At pass @ 16 and pass@22 25, SBSC matches ol preview and surpasses ol-mini by absolute 13%.
* Geometry: This the only topic where SBSC lacks behind ol models.
Error Analysis: Within pass @ 25 the model was not able generate correct SBSC trajectory for 38 AIME questions out of 90 questions. We show topic wise error distribution below:
out of 21 from Algebra.
- 10 out of 23 from combinatorics
- 23 out of 32 from geometry
- 2 out of 14 from number theory
We manually go through the trajectories for these 38 questions. So we analyse 38 SBSC paths. We classify the emrors in multiple categories shown in Table T6
29

---

Published as a conference paper at ICLR 2025
A.9 PAL EXEMPLARS
In this section, we provide the prompts for Program- Sodee Language models (PAL) method. We initially used the default prompt mentioned the original PAL. paper, but the results were poor. We noticed that the response often contained textual reasoning before or after the program, which isn't the desired format for PAL. Hence, we modify the instructions to confine the responses only to include Python program and subsequently, also notice improved accuracy.
For AIME
Let's use python program solve math problems. DO NOT USE ANY TEXTUAL REASONING. Your response must start with: "pythhnn
Your response must end with: print(result)
Here examples you may refer
Example Problem: frog begins at Po (0, 0) and makes a sequence of jumps according to the following rule: from P. (I Dn), ; the frog jumps to Pat1, which may be any of the points
(In + 7n + 2), (I,, + 2, Un. + 7), (Inn 5, 10), or (rn - 10, yo - 5). There are M points (ar, y) with [a|| + lyl < 100 that can be reached by sequence of such jumps. Find the remainder when M is divided by 1000.
Example Solution:
def soluticn():
jumps (7, 2), (2, 7), (-5, -10), (-10, -5]
Set keep track of all reachable points, starting from origin (0, 0)
reachable set (1(O, 0)1)
Queue to process points, starting with the origin (0, 0).
queue (0,, 0}]
Breadth-first. search (BFS) to explore reachable points.
while queue:
$ Pop the first point from the queue.
x, y queue.pop (0)
Iterate over all possible jumps.
for dx, dy in jumps:
A Calculate new coordinates after the jump. nx, ny
dx,
dy
Check if the Manhattan distance is within 100 and the point hasn't been visited.
if abs (nx) abs (ny) <= 100 and (nx, ny) not in reachable: # Add the new point to the reachable set. reachable add (nx, ny))
Add the new peint the queue explore further. queue.appor append (nx, ny))
return 1en (en(achhabbe) 1000
result solution() print (result)
The AIME Triathlon consists of half-mile swim, 30-mile bicycle ride, and an eight-mile run. Tom swims, bicycles, and runs at constant rates. He runs fives times as fast as he swims, and he bicycles twice fast as he runs. Tom completes the AIME Triathlon in four and quarter hours. How many minutes does spend bicycling?
Example Problem:
Example Solution:
From sympy import symbols, Eq, sclve, Rational def soluticn():
symbels('x')
Set up the equation
eq Eq (Rational (1,2) /x + 30/ (10+x) + 8/ (5*x), Rational (17,4))
Solve the equation
solution - salve [0]
Calculate bicyeling time in hours
30

---

Published as a conference paper at ICLR 2025
bike time 30 (10 soluticn) Convert minutes
sike_time_ninuter int (bike_time 60)
result sclution minitmm
return bike_time_minee
print result
Example Problem: Let S be the increasing sequence of positive integers whose binary representation has exactly ones. Let N be the 1000th number S Find the remainder when N divided by 1000
Example Solution:
def soluticn():
count Initialize a counter to track how many numbers have been found
== 1 start checking numbers from upwards
while count 1000: & Continue the loop until find the 1000th number
# Check if the binary representation of the number 'n' has exactly 8 '1's if bin cow.t (11') 8: count
$ Increment the counter when number with 8 '1's
found
# IE this is the 1000th such number, return the remainder of n divided by 1000
if count 1000:
return n 1000
Move to the next number result sclution() print (result)
n
Two geometric sequences a,, 03, and b,, b2, by have the same com- mon ratio, with di 27 99, and a15 bi Find ag
Example Problem:
Example Solution:
def solution():
Initialize known values
al 27
b1 99
Calculate the common ratic We know that a15 bll, so:
# al * r-14 = bl * r<10
# 27 * r^14 = 99 r010
27 r04 99
= 99/27 11/3
r (11/3) ** (1/4)
Calculate a9
a9 al (r 8) return (a9)
result solution() print (result)
For AMC:
Let's use python program solve math problems.
DO NOT USE ANY TEXTUAL REASONING.
Your response must start with: "'yythnn Your response must end with: print(result)
Here some examples you may refer to.
Example Problem: Small lights are hung on a string inches apart in the order red, red, green, green, green, red, red, green, green, green, and continuing this pattern of red lights followed by green lights. How many feet separate the 3rd red light and the 2 red light? Note: foot is equal 12 inches.
31

---

Published as a conference paper at ICLR 2025
Example Solution:
def solution ():
Find position of 3rd red 1ight
3_a =
sumplete_cycless 3rd (n 3rd 11 //
remaining lighaninnnnggghsss.ooms 3rd = (n 3rd
1) 8
pos_3rd complete_cycles__1l 3rd 5 6
remaining_lights_b. 3rd 6
Find pesition of 21st red light
2_2ss
cmpplete_cycles. 21st (n_ 21st 1) // 2
Lemaining_lights_202 21st (n_21st - 1) 2
pos pos_2lst = cmpplete_cycles 21st *
remaining_ laciinngggghht lights 21st
Calculate the distance in inches
distance_inches
pos_21st pes_3rd 3rd
Convert to feet
distance_feet Distance_inches / 12 return distance_ feet result solution print (result)
Example Problem: A fruit salad consists of blueberries, raspberries, grapes, and cherries. The fruit salad has total of 280 pieces of fruit. There are twice as many raspberries as blueberries, three times as many grapes cherries, and four times many cherries as raspberries. How many cherries are there in the fruit salad?
Example Solution:
from sympy Import symbols, Eq, sclve def solution():
Define the symbols for the variables
b, r, c symbols ("b (*b r c')
Define the equations based the problem statement.
eql (q 2xb)
Equation 1: 2b
eq2 Eq(g, 3+c)
Equation 2: g
eq3 Eq(c, 4+r)
# Equation 3: c = 4r
eq4 Eq r
c, 280)
Equation 4: b +
280
Solve the system of equations
sol solve (eql, eq2, eq3, eq4))
return sollc]
result sclution() print (result)
Example Problem: Last summer 30% of the birds living on Town Lake were geese, 25% were swans, 10% were herons, and 35% were ducks. What percent of the birds that were not swans were geese?
Example Solution:
def solution() ():
Total percentage of all birds total 100
Percentages of each pird type geese 30
swans 25
herons 10
ducks = 35
Calculate percentage of birds that are not swans not_swans total swans
Calculate percentage of geese among birds that not geese aare__mmng_not_oeee
(geese not_swans) 100
Round to nearest whole number return round (gaeseaagaaa ) result sclution()
print (result)
32

---

Published as a conference paper at ICLR 2025
Example Problem: At twins and triplets convention, there 9 sets of twins and sets of triplets, all from different families. Each twin shook hands with all the twins except his/her siblings and with half the triplets. Each triplet shook hands with all the triplets except his/her siblings and with half the twins. How many handshakes took place?
Example Solution:
def solution():
# Number of twins and triplets twins = 9
triplets 6
Handshakes between twins
twin_handshakes (twins + (twins 2)) // 2
Handshakes between triplets
(triplets (triplets - 3)) // Handshakes between twins and triplets
twin wtw.trrpeee_handsaaeeeee handshakes (twins triplets)
# Total handshakes
@atal_handshaker = twin handshakes + triplet Sttpleethannshaaeer
wwin_trlplet_handebte hhanssaakess
return total_handshake
result solution()
print (result))
A.10 TIR-TORA EXEMPLARS
For AIME
Integrate step-by-step reasoning and Python code to solve math problems using the following guidelines:
Analyze the question and write functions solve the problem; the function should not take any arguments.
Present the final result in LaTeX using - without any units.
Utilize the 'pi' symbol and "Rational" from Sympy for π and fractions, and simplify all fractions and square roots without converting them to decimal values.
Once you get the code output, just display the output answer within 's' and terminate.
Here
examples you may refer
Example Problem: frog begins at Po (0, 0) and makes a sequence of jumps according to the following rule: from P. (I yn), ) the frog jumps Pa+1, which may be any of the points (In, + 7, + 2), (I,, + 2, + 7), (xn 5, 10), or (xn 10, yn - 5). There are M points (x., y) with |z| + |y| I 100 that can be reached by a sequence of such jumps. Find the remainder when M is divided by 1000.
Example Solution: Let's 's approach this problem step-by-step:
1. We need to find all points (x, y) that the frog can reach within the given constraint + lyl 100.
2. We'll Il use breadth-first search (BFS) algorithm to explore all possible points.
3. We'll 'll keep track of visited points to avoid revisiting them.
4. We'll count the number of unique points visited.
Here's 's the Python code to solve this:
From collections import deque def count_ count_reachable_poinn (): Possible jumps
jumps = [7,, 2), (2, 7), (-5, -10), (-10, -5) Set to store visited points visited set (I(,, ))
33

---

Published as a conference paper at ICLR 2025
Queue for BFS
queue deque (I( 0)1)
while queue:
X, = queue.popleft.
for dx, dy in jumps:
new_x, x, new_y
dx, y dy
# Check if the new pcint within the constraint and not
visited
is abs (new_x) abs (new_y) <= 100 and (new_x, new_y) not in
visited:
visited. add (new_x, new_y)
queue append append (new_x, new_y))
Count of reachable points is the size of visited set M len (visited)
Return the remainder when M is divided by 1000 return 3 1000
result
print (f") \boxxd (ressult))))
>>>
output Aboxed (373)
The answer is 373
The AIME Triathlon consists of a half-mile swim, a 30-mile bicycle ride, and an eight-mile run. Tom swims, bicycles, and runs at constant rates. He runs fives times fast as he swims, and he bicycles twice as fast as he runs. Tom completes the AIME Triathlon in four and quarter hours. How many minutes does he spend bicycling?
Example Problem:
Example Solution:
Let's approach this problem step-by-step:
1. Let's define variables:
Let x be Tom's 's swimming speed in miles per hour
Then 5x is his running speed in miles per hour
And 10x is his bicycling speed in miles per hour
2. We can set up equations based on the given information:
Swimming time: 0.5 x hours
Bicycling time: 30/ / (10x) hours
Running time: / (5x) hours
3. The total time is 4.25 hours, so we can set an equation: 0.5/x 30/ (10x) 8/(5x) 4.25
4. Let's solve this equation and find bicycling time.
Here's the Python code solve this:
from sympy import symbols, Eq, sclve, Rational
def solve_triathlon
x symbols('x')
Set up the equation
eq Eq (Rationa /x + 30/ (10xx) + 8/ (5xx), Rational (17, (17,4)) 4)
Solve the equation solution
solve(eg) [0] Calculate bicycling time in hours
bike bike_time time = 30 (10 solution)
# Convert to minutes
int (bike_time 60)
return f"\\boxed| [bike_ time minutemin
print (solve_triathlon, )))
>>> output: Aboxed (150)
34

---

Published as a conference paper at ICLR 2025
The answer is 150
Example Problem: Let be the increasing sequence of positive integers whose binary representation has exactly 8 ones. Let N be the 1000th number in S Find the remainder when N is divided by 1000
Example Solution:
Let's s approach this step-by hep-by-step:
1. We need to generate the sequence S of numbers with exactly 8 ones in their binary represen- tation.
2. We'll find the 1000th number in this sequence.
3. Finally, we'll calculate the remainder when this number is divided by 1000.
Here's the Python code solve this:
def
count_ oeeeeneeeeeee (n)
return bin (n) count (*11)
def soluticn() ()
count
=1
while count < 1000: if count_ones (n) == 8:
count +- 1
if count 1000:
return n 1000
n1 +=
result solution() print (result)
>>> output: Aboxed(32) (3)
The answer is 32
Example Problem: Two geometric sequences a1 022 03, and b,, b2, by.... have the same com- mon ratio, with 27 bj 99, and d15 b11 Find 0g
Example Solution:
Let's approach this step-by-step:
1. Let's 's denote the common ratio as
2. We can express a and b in terms of their first terms and r.
3. We can then equate these expressions to find
4. Once have r,
calculate
Here's the Python code to solve this:
From sympy import symbols, Eg, solve, Rational def fine_ ()
r symbols('r") a
à
b
b
eq B (27 r*+14, 99 r++10)
# Solve for r solution = seq)
expect only
positive real solution
r_value [sol for sol solution if sol.is_red real and sol 0] [0]
Calculate a a9 27 +_value++8
return ag result find_a9()
35

---

Published as a conference paper at ICLR 2025
print (f" \boxed| [result)]
>>> output Aboxed 363)
(363)
The answer is 363
For AMC
Integrate step-by-step reasoning and Python code to solve math problems using the following guidelines:
Analyye the question and write functions to solve the problem; the function should not take any arguments. Present the final result LaTeX using 'n' without any units. Utilize the 'pi' symbol and "Rational" from Sympy for and fractions, and simplify all fractions and square roots without converting them to decimal values.
Once you get the code output, just display the output answer within 'n" and terminate. Here are some examples you may refer to:
Small lights are hung string 6 inches apart in the order red, red, green, green, green, red, red, green, green, green, and so on continuing this pattern of red lights followed by green lights. How many feet separate the 3rd red light and the 21st red light? Note: foot is equal to 12 inches.
Example Problem:
Example Solution:
Let's approach this step-by-step:
First, we need understand the pattern of lights:
Every 5 lights (2 red 3 green) complete one cycle of the pattern.
Each cycle is 30 inches long (5 lights * inches).
2. We need to find how many complete cycles are between the 3rd and 2 red lights:
3rd red light is at the end of the 2nd cycle (as each cycle has red lights).
2tt red light is the end of the 11th cycle (as 21 10.5, rounded up to 11).
So there are 9 complete cycles between them (11 9).
3. Calculate the distance:
9 cycles 30 inches per cycle 270 inches
Convert 270 inches to feet: 270 + 12 = 22.5 feet
Here's the Python code to solve this:
from sympy import Rational
def calculaae__distaneel]] ()
lights_per_cycee
inches_per_light
inches_per_cycla lights_per_cyen
inches_per___ light
start_cycle = # 3rd red light is at the end of 2nd cycle
end_cycle = 11 # 21st red light is at the end of 11th cyele
cycles_betweed end_eycle start_cycles
total_inches = eycles_petweed
inches peceeessper__yllee__
total_feet = Rational (totall inches, 12) return ("ooed [[total_feet]]]
print
stance
>>> output: (boxed(45/2)
The answer is 45/2
Example Problem: A fruit salad consists of blueberries, raspberries, grapes, and cherries. The fruit salad has total of 280 pieces of fruit. There twice as many raspberries as blueberries, three times as many grapes as cherries, and four times as many cherries as raspberries. How many cherries are there in the fruit salad?
36

---

Published as a conference paper at ICLR 2025
Example Solution:
Let's s approach this problem step-by-step:
1. Let's define variables for each fruit:
number of blueberries
number of raspberries
g number of grapes
number of cherries
2. We can set up equations based on the given information:
2b (twice as many raspberries as blueberries)
3c (three times as many grapes as cherries)
с 4r (four times as many cherries as raspberries)
3. We also know that the total number of fruits is 280:
b + +rr+ga = 280
4. Now, let's write Python function to solve this system of equations:
from sympy import symbols, s, Eq, sclve
def solve_fruit_saldd))
b, F, g, с symbols ("D = g g
eql Eq (I, 2xb)
ec2 Eq(g, 3+c)
eq3 (q 4+r)
ec4 Eq
g c, 280)
solution = solve (eql, eq2, eq3, eq4))
return
print Solve_fruit__sales
>>> output:
(boxed(64)
The answer is
64
Example Problem: Last summer 30% of the birds living on Town Lake were geese, 25% were swans, 10% were herons, and 35% were ducks. What percent of the birds that were not swans were geese?
Example Solution:
Let's approach this step-by-step:
First, we need to calculate the percentage birds that were not swans.
2. Then, we'll calculate what percentage of this non-swan group geese.
Here's the Python code to solve this:
from sympy Import Rational
def calculate calculate_geese,
():
# Total percentage of birds
total 100
Percentage of swans 25
Percentage of non-swans non_swans total swans
Percentage of geese among all birds
geese 30
Calculate the percentage of geese among non-swans geese pesse__moog_non san_swans_ss Rational (geese, non son_swans) * 100 Simplify the fraction
result geese_amonq_non,
return 1" lbooodd [result))))"
print
>>> output
Aboxed (40)
37

---

Published as a conference paper at ICLR 2025
The answer is
40
Example Problem: At a twins and triplets convention, there were 9 sets of twins and 6 sets of triplets, all from different families. Each twin shook hands with all the twins except his/her siblings and with half the triplets. Each triplet shook hands with all the triplets except his/her siblings and with half the twins. How many handshakes took place?
Example Solution:
Let's approach this problem step-by-step:
1. Calculate the number of handshakes between twins
2. Calculate the number of handshakes between triplets
3. Calculate the number of bandshakes between twins and triplets
4. Sum up all handshakes
Here's the Python code solve this:
from sympy import Rational def calculate handshakes (): Number of twins and triplets twins
triplets - 6
Handshakes between twins
twin_handshaken (twins (twins 21) // 2 # Each twin shakes hands with all but other twin # Handshakes between triplets
(triplets (triplets - 3)) // 2 # Each triplet
shakes hands with all but 2 other triplets Handshakes between twins and triplets siin_triplet_handhhateee
twins + triplets + Rational (1, 2) Rach twin shakes hands with half the triplets
Total handshakes
total handshakes twin handshakes triplet Saapllethannshakee handshakes
twin hanesaaaeeeee
return E") \boxed! | lint (total handshakes) }))"
print (calculate_handshakes ())
>>> output
(441)
The answer is
441
A.11 SBSC PROMPT FOR AMC
You are given a math problem and you need to think of a stepwise approach/process to be followed to solve the problem. Use sympy-based python code codify each of these steps sequential manner. You must end each of your code snippet with print statement followed by output in the next line. Use the results from the output of the code snippets of each step and continue to the next step until you finish the final step and solve the problem completely.
each new code block, you must define the variables and symbols again based on the previous code snippets or their outputs suitable.
Use sympy in appropriate way, as and when required. Once you complete the last step, write this line: ### END OF CODE Then just print the final numeric answer (without any units) like this: The final answer is boxed ( J. If the executed code snippet returns error, NOT restart solving from Step 1.
correct the current step's code snippet. DO
Here few examples for your reference.
Example Problem: Small lights are hung on a string inches apart in the order red, red, green, green, green, red, red, green, green, green, and continuing this pattern of red lights followed
38

---

Published as a conference paper at ICLR 2025
by green lights. How many feet separate the 3rd red light and the 21st red light? Note: foot is equal to 12 inches.
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step 1: Define the pattern and calculate the number of lights in one complete pattern
From sympy import symbols, Rational
Define the number of red and green lights in one pattern red_
Leeen__Lightt__e__pttee Calculate total lights in one pattern
sights_per__ppeee pertpattern _ee_lights_eer_ptteeee + green green_Lights pattern
print (f"Number of lights in one complete pattern:
>>> output:
Number of lights in one complete pattern: 5
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps.
Step 2: Calculate the number of complete patterns between the 3rd and 21st red lights
From sympy import symbols, Raticnal Define the number of red lights we're considering Seear__re__iigeee end__ee__lihht 21
Calculate the number of red lights between the 3rd and 21st (inclusive) Seart_red__aet
red wighht__etweeeeeeeeeeeeeeeeeeeee
end red red_gihtt calculate the number of complete patterns
complete_patterno (red_lights__etteen)
1}
print (ffbure of complete patterns between the 3rd and 21st red lights:
>>>
output:
Number of complete patterns between the 3rd and 21st red lights:
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Calculate the total number of lights between the 3rd and 21st red lights
Step 3:
From sympy import symbols, Rational
Define the number of red and green lights in one pattern
-
green__ighhsseee_ptttrr pel_lights_per_patten green_lights_per_pateee
Lights_per_pattee
Number of complete patterns
complete_pattern. =
Calculate total lights
total_lights complete_patteree
lights_per. per_pattern # +1 for the
21st red light
print (f"Total number of lights between the 3rd and 21st red lights ( inclusive): lightss)
output:
Total number of lights between the 3rd and 21st red lights (inclusive) 46
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 4: Calculate the total distance in inches
From sympy import symbols, Rational
Total number of lights
totallliggtt 46
Distance between lights inches settaacc__eeteen__iie
=
Calculate total distance in inches @atral_dsttane_____ print (f"aat distance in inches: (total stance_ inch_inhhhh)))
Scotal_lights
di ditance_between_11
39

---

Published as a conference paper at ICLR 2025
>>> output:
Total distance in inches: 270
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 5: Convert the distance from inches to feet
From sympy import symbols, Raticnal Total cistance in inches total_aistance siscthece__nchess inches 270 Convert inches to feet inches_per__oot - 12
cittalleistanceeeeee print (f"Total distance feet:
Rational (total_distance_ inches_per_fone
feet)
>>>> output:
Total distance in feet: 45/2
### END OF CODE
45
The final answer is
Example Problem: A fruit salad consists of blueberries, raspberries, grapes, and cherries. The fruit salad has total of 280 pieces of fruit. There twice as many raspberries blueberries, three times as many grapes cherries, and four times many cherries as raspberries. How many cherries are there in the fruit salad?
Let's solve this problem step by step using SymPy-based Python code. Step Define the variables and set up the equations
Example Solution:
From sympy import symbols, s, Eq, sclve
Define symbols for each type of fruit
b, т, g, с symbols ('b r g c²)
Set up equations based on the given information
eql (b r
+ g + c, 280)
Total pieces of fruit
eq2 Eg (r, 2+b)
# Twice
many raspberries as blueberries
eq3 bq 3xc)
I Three times many grapes as cherries
eq4 (q 4xr)
Four times as many cherries raspberries
print ("Equations:")
print (egl)
print (eg2)
print (eq3)
print (cg4)
>>> output:
Equations:
(g + g + r, 280)
eq 2xb)
Eq(g, 3xc)
(g 4+r)
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 2: Solve the system of equations
From sympy import symbols, Eq, sclve
Define symbols for each type of fruit
b, r, с symbols
('b ('b r g c')
Set equations based on the given information
eql Eq
g c, 280)
Total pieces of fruit
eq2 Eq (r, 2xb)
Twice as many raspberries as blueberries
eq3 Eq (g, 3+c) Three times as many grapes as cherries
eq4 Eg 4+r) # Four times as many cherries as raspberries
Solve the system of equations
solution sclve((eql, eq?, eq3, )
40

---

Published as a conference paper at ICLR 2025
print ("Solution:")
print (solution)
>>> output:
Solution:
(b: 8, c: 64, g: 192, 16)
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 3: Extract the number of cherries from the solution
From sуmру Import symbols, Eq, sclve
Define symbols for each type of fruit
b, r, g, с symbols ('b r g c¹)
Set up equations based on the given information
eql Eq (b r + g + c, 280)
Total pieces of fruit
eq2 Eg 2*b) # Twice
many raspberries as blueberries
eq3 Eq (g, 3xc)
Three times many grapes cherries
eq4 (g 4+r)
Four times as many cherries raspberries
Solve the system of equations
solution - (olve((eq)) eq2, eq3, eq4))
Extract the number of cherries
cherries sclution[c]
print (f"The number of cherries in the fruit salad is: (cherries)
>>> output:
The number of cherries in the fruit salad is: 64
END OF CODE The final answer is 64
Example Problem: Last summer 30% of the birds living on Town Lake were geese, 25% were swans, 10% were herons, and 35% were ducks. What percent of the birds that were not swans were geese?
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step 1: Define the percentages of each type bird
from sympy Import symbols, Rational
geese Rational (30, 100)
swans Raticnal (25, 100)
herons Rational (10, 100)
ducks Rational (35, 100)
print (E"Fercentage of geese: (geese] ")
print (f"Fercentage of swans: (swans | ")
print (f"Fercentage of herons: [hercns] ")
print Ferreettaage of ducks: (ducks)")
>>> output
Percentage of geese: 3/10
Percentage of swans: 1/4
Percentage of herons: 1/10
Percentage of ducks: 7/20
Let's continue with the next steps to solve this problem. Step 2: Calculate the percentage of birds that are not swans
from sympy import symbols, Rational
geese Rational (30, 100)
swans = Rational (25, 100)
herons Rational (10, 100)
ducks Rational (35, 100)
not_swans
swans
print age of birds that are not swans: (not_swans] ")
>>> output:
Percentage of birds that swans: 3/4
41

---

Published as a conference paper at ICLR 2025
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps.
Step 3: Calculate the percentage of geese among the birds that are not swans
From sympy import symbols, Rat onal
geese Rational (30, 100)
swans Rational (25, 100)
herons Rational (10, 100)
ducks - Rational (35, 100)
not_swans swans
geese_among_net_swee
geese net_swans
print (f" of geese among birds that are not swans: [eese_among_not_swamr _
>>> output:
Percentage of geese among pirds that are not swans: 2/5
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps.
Step 4: Convert the fraction to a percentage
From sympy import symbols, 8, Rat cnal
geese Rational (30, 100)
Rational (25, 100)
herons Rational (10, 100)
ducks = Rational (35, 100)
_ot__waas__ = swans
geese_among_nnt_swans
geese nct_swans
percentage gpose_among_not.c
100
print (f af of geose among birds that are not swans: (percentage] |
>>> output:
Percentage of geese among pirds that are not swans: 40%
### END OF CODE
The final answer 40
Example Problem: At twins and triplets convention, there were 9 sets of twins and 6 sets of triplets, all from different families. Each twin shook hands with all the twins except his/her siblings and with half the triplets. Each triplet shook hands with all the triplets except his/her siblings and with half the twins. How many handshakes took place?
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step Define the given information
From sympy import symbols, Rational
Number of sets of twins and triplets
twin_sets
triplet_sett sets
Number of individuals
twins = twin
triplets - triplet_sets
print Nfmber of twins: (twins)")
print (f"Number of triplets: (triplets)")
>>> output:
Number of twins: 18
Number of triplets: 18
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Step 2: Calculate handshakes between twins
From sympy import symbols, Rational
Number of sets of twins and trip triplets
twi sets
42

---

Published as a conference paper at ICLR 2025
triplet sets
Number of individuals
twins = twin_sets 2
triplets = triplet_sets +
Handshakes between twins
wrin_handshakes (twins (twins 2 // 2 Each twin shakes hands with all twins except self and sibling
print (H"Hanssaakes between twins: haadshakndhh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> output
Handshakes between twins: 144
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Step 3: Calculate handshakes between triplets
From sympy import symbols, Rational
Numbor of scts of twins and triplets
twin_sets
triplet_scts
Number of individuals
twins - twin_sets
triplets - triplet_sets
Handshakes between twins
twin hanashakes (twins (twins 21) // 2
Handshakes between triplets
happlet_aanshakess (triplets (triplets - 3)) // 2 # Each triplet shakes hands with all triplets except self and two siblings
print Hlandsaaees between triplets: (triplet_handshakes,
>>> output
Handshakes between triplets: 135
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 4: Calculate handshakes between twins and triplets
From sympy import symbols, Raticnal of twins and triplets
Number of twin_sets
triplet_sets -
Number of individuals twins twin_
triplets = triplet_sed sets *
Handshakes between twins
twin hain_hannshakes = (twins (twins 2)) //
Handshakes between triplets
hriplet_handshake
(triplets (triplets 3)) //
Handshakes between twins and triplets
= twins * (triplets // 2) # Rach twin shakes hands with half the triplets
print (f Hannassaaeess between twins and triplets: (twin_tripled htw.sspplllt..assaakkee
>>> output:
Handshakes between twins and triplets: 162
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 5: Calculate the total number of handshakes
From sympy import symbols, Raticnal Number of scts twins and triplets
twin_sets = triplet_sets =
Number of individuals twins twin_sets
43

---

Published as a conference paper at ICLR 2025
triplets triplet
sets sets
Handshakes between twins
Ewin_hanéshakes = (twins + (twins 2)) //
Handshakes between triplets
triplet_haape haplet__anddhaaes (triplets (triplets - 31) /1
Handshakes between twins and triplets
twins * (triplets // 2)
Total handshakes
Sital_handshaaer = handshaass +
print (f"Total number of handshakes:
>>> output
Total number of handshakes: 441
### END OF CODE
The final answer is 441
A.12 SBSC PROMPT FOR AIME
You are given math problem and you need to think of stepwise approach/process to be followed to solve the problem. Use sympy-based python code to codify each of these steps sequential manner. You must end each of your code snippet with print statement followed by output in the next line. Use the results from the output of the code snippets each step and continue to the next step until you finish the final step and solve the problem completely. In each new code block, you must define the variables and symbols again based on the previous code snippets or their outputs suitable.
Use sympy in appropriate way, as and when required. Once you complete the last step, write this line: ### END OF CODE
Then just print the final numeric answer (without any units) like this: The final answer is boxed }. If the executed code snippet returns an error, use correct the current step's code snippet. DO NOT restart solving from Step 1.
Here are a few examples for your reference.
Example Problem: A frog begins at Po (0, 0) and makes a sequence of jumps according to the following rule: from P. (In, yn), the frog jumps to Pa+1, which may be any of the points (In, + 7, + 2), (I,, + 2, Un + 7), (In 5, vn 10), or (In, 10, yo. - 5). There are M points (x, 3) with |x] + |y] < 100 that can be reached by a sequence of such jumps. Find the remainder when M is divided by 1000.
Example Solution: Let's approach this problem step by step using SymPy-based Python code.
Step 1: Import necessary libraries and define the jump rules
from sympy import symbols, Abs
import numpy as np
Define the possible jumps from any position (x, def jumps (x, ;): return
(x 7, y 2) Jump by (7, 2)
(x 2, y 7), Jump by (2, 7)
(x 5, y 10),
Jump by (-5, -10) Jump (-10, -5)
10, y
print ("Jump rules defined.")
>>> output Jump rules defined.
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps.
Step 2: Implement a breadth-first search find all reachable points
44

---

Published as a conference paper at ICLR 2025
From sympy import symbols, Abs import numpy as np
from collections import deque
Reuse the junps function to define possible moves def jumps (x, y) return
(x 7, y 2),
(x 2, y 1),
[x 5, y 10),
10, y 51
Define the first search (BFS) function def ()
queue = deque 011) # Start BFS from the point (0, 0)
visited set (1 0)1) Keep track of visited points avoid
reprocessing
Process each point in the queuc
while queue:
x, y = queue. popleffo
for nx, ny jumps (x, y): # Try all possible jurps from the current point
if (nx, ny) not in visited and abs (nx) + abs (ny) <- 100: visited. add (nx, ny)) # Mark the new point as visited queue append (nx, ny) # Add the new point to the queue
explore further
return len (visited)
M
()
print Number of reachable points (M); (M))
>>>
output:
Number of reachable points (M) 1373
Let's continue with the final step solve this problem. Step 3: Calculate the remainder when M is divided by 1000
From sympy import symbols, Abs import numpy np
1373 M_value from previ step Find the remainder when is divided by 1000
remainder = M & 1000
print (f" Remainder when M is divided by 1000: (remainder )))
>>> output Remainder when M is divided by 1000: 373
### END OF CODE
The final answer 373
Example Problem: The AIME Triathlon consists of half-mile swim, 30-mile bicycle ride, and an eight-mile run. Tom swims, bicycles, and runs at constant rates. He runs fives times as fast as he swims, and he bicycles twice as fast as he runs. Tom completes the AIME Triathlon in four and quarter hours. How many minutes does spend bicycling?
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step 1:
Define the variables and set up the basic equations
From sympy import symbols, Eq, sclve
Define variables
8, b, symbols r') # s: swim speed, b: bike speed, speed
Set up equations based on the given information
eqll (g 5+5) Tom runs 5 times as fast as he swims
eg2 = lb, 2+r) # Tom picycles twice as fast as he runs
print ("Equation 1:", eql)
print ("kquation 2:", eq2)
45

---

Published as a conference paper at ICLR 2025
>>> output
Equation 1: Eq(r, 548) Equation 2: Eq(b, 2+r)
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 2: Set up the equation for the total time
from sympy import symbols, Eg, sclve
Reuse earlier code
s, b, r symbola b r!)
eql Eq 5*s)
eq2 (q 2+r)
Total time equation (in hours)
totatlltiii time 0.5/s 30/b 8/r #The AIME Triathlon consists of a half- mile swim, a 30-mile bicycle ride, and an eight-mile run. eq3 = (total__tim, 4.25) & Total time is 4.25 hours print (*Total time equation:", eq3)
>>> output:
Total time equation: (0.555/s 8/r + 30/b, 4.25)
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 3: Substitute the relationships between speeds into the total time equation
From sympy import symbols, Eq, solve, simplify
Resue earlier code
s, b, r symbols r')
eql (g 548)
eq2 Eq (b, 2*r)
tootalttttttte time 0.5/s 30/b 8/r
eq3 Eq (total__tim,, 4.25) # Tetal time is 4.25 hours Substitute r and in terms of to simplify the equation eq3 [gg__subtitutted = eq3. subs (I 5+s), (b, 1]+s) eq3_simplified simplify susbitubstt) print ("Simpli equation:", aq3_simplified)
>>> output:
Simplified equation: Eq(5.1/s, 4.25)
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps.
Step 4: Solve for the swim speed (s)
from sympy import symbols, Eq, sclve, simplify
Redefine variables
s, b, symbols r') # s: swim speed, b: bike speed, speed Simplified equation from previcus step
eq3__impliit
1/. 4.25)
Solve for s
solutionn solve (eq3 simplified, print ("Swim speed (s):", s_solutica00))
>>> output
Swim speed (s): 1. 20000000000000
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 5: Calculate the bike speed (b)
From sympy import symbols, Eq, sclve, simplify
Redefine variables
s, b, r = symbols('s b r²) # s: swim speed, b: bike speed, r: run speed
Swim speed from previous step
s_value 1. .000000000000000
Calculate bike speed
46

---

Published as a conference paper at ICLR 2025
value value 2r 2(5s) 10s print ("Bike speed :b):", b_value)
>>> output
Bike speed (b) 12.0
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps.
Step 6: Calculate the time spent bicycling
From sympy import symbols, Eq, sclve, simplify
Redefine variables
s, b, I s'm r') # s: swim speed, b: bike speed, r: run speed Bike speed from previous step
b_value 12.0
Calculate time spent bicycling
bike_distance 30 miles
time bike bike_distanee distance value hours
Convert to minutes
bike_time minutes bike_time 60
print ("Time spent bicycling (in minutes): ;", bike_time_minutes)
>>> output
Time spent bicycling (in minutes): 150.0
### END OF CODE
The final answer is 150
Example Problem: Let be the increasing sequence of positive integers whose binary representation has exactly 8 ones. Let N be the 1000th number in S Find the remainder when N is divided by 1000
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step Import necessary libraries and define function to count ones binary representation
From sympy import symbols, binomial
import math
Function to count the number of 'i's in the binary representation of a number def count_cnes(n) (n)
return bin count((1)) Convert to binary and count occurrences of '1"
print ("Function to count ones in binary representation defined.")
>>>>>>>>>>>>>>>>>>>>>>>>>> output:
Function to count ones binary representation defined.
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 2: Find the smallest number in the sequence
from sympy import symbols, binomial
import math
Reuse the function to count the number of '1's in binary
def count_cnes(n))
return bin(n) (annnnnccum count
smallest number the sequence is represented by consecutive '1's in binary
sma smalleest maaallest__mbbee int (111 8, 2) # Convert "1m.i (binary) to
decimal
print (T" smallest number in sequence is: [mmal lest_number)""
>>> output
The smallest number in sequence S is: 255
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Step 3: Find the largest number in the sequence
47

---

Published as a conference paper at ICLR 2025
From sympy import symbols, binemial
import math
Reuse the function to count the number of ' in binary def count_enes (n)
return (inuuttc
count
smallest_number int ('1'
The largest number in sequence is represented by consecutive 1i's followed by 24 '0's
dargestt int ('1' 011100000000000
'0' 24, 2) f Convert
to decimal
(f"The largest number in sequence is: (largest_nurber)
>>> output
The largest number in sequence S is: 4278190080
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 4: Calculate the total number of elements in sequence S
from sympy Import symbols, binomial
import math
def count_ones
return (n) count (111)
smallesttnumber = int (11' 8, 2)
'0' 24, 2) porgest__uubbe ('1'
Calculate the total numper elements sequence using binomial (32, 8)
This represents the number of ways to choose 8 positions out of 32 bits to set to '1'
totae_elements binomial (32, 8)
print (f"The total number of elements in sequence is: (total_elementa)
>>> output
The total number of elements in sequence is: 10518300
Let's continue with the next steps solve this problem and leveraging outputs of previous steps. Step 5: Find the 1000th number in sequence S
from sympy import symbols, s, nimomi
import math Reuse all earlier code
def fount_ones_t (n)
return bin (n) count ('1')
mmallest_number
int ('1' 8, 2)
largest_number int (ot('1 ('1' * 8 '0' * 24, 2)
total_elements binomial (32, 8)
Find the 1000th number in the sequence count Initialize a counter for the sequence smallest_number # Start from the smallest number in
Iterate through numbers and check for 8 '1's in the binary representation
while count 1000:
if count_ones (N)
# If the current number has exactly 11's count +- 1 # Increment the count
if count -- 1000: Stop if we've found the 1000th number
break += Move to the next number
print (f"The 1000th number (N) sequence is: (NF")
>>> output
The 1000th number (N) in sequence is: 7032
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Step 6: Calculate the remainder when N is divided by 1000
From sympy import symbol 8, bi nomi
48

---

Published as a conference paper at ICLR 2025
import math
def count_ones (n)
return bin(n) count ('1')
N = 7032 # From the previous stop
Calculate the remainder when N is divided by 1000
remainder N 1000
print (f"The remainder when divided by 1000 is: (remainder) ")
>>> output
The remainder when is divided by 1000 is: 32
### END CODE
The final answer is 32
Example Problem: Two geometric sequences d1 02 03, and b,, b2, bz.... have the same com-
mon ratio, with 27 bj 99, and 415 b1 Find ag
Example Solution: Let's solve this problem step by step using SymPy-based Python code.
Step 1: Define the variables and set up the initial conditions
from sympy import symbols, s, Eq, sclve
Define symbols
symbols ('r') I common ratic
al, bl 27, 99 # first terms of sequences and b
print (f*al (al), bl (bl)")
print ("Common ratio is represented by r")
>>>
output al 27, b1 99
Common ratic is represented by
Let's continue with the next steps to solve this problem and leveraging the outputs of previous steps. Step 2: Set up the equation for al5 bl1
From sympy import symbols, Eg, solve
Redefine symbols symbols('r) ('r))
al, bl 27, 99
Set up the equation a15 b11 eg Ec (al r*a14, bl r++10)
print ("Equation: a15 = b11")
print (f"Symoolcalll: (eq)")
>>> output Equation: a15 = b11
Symbolically: 144 99*r++10)
Let's continue with the next steps to solve this problem and leveraging outputs of previous steps. Step 3: Solve the equation for
from sympy import symbols, Eq, solve
Redefine symbols (ymbols('r!)
al, bl 27, 99
up the equation a15 b11 eq Eq (al r*414, bl r++10)
Solve for r solution = seqve(eq, print ("Solutions for r:")
print (solution)
>>> output
Solutions for
[0, -11* (3/4)) 11. (3/4) /3]
49

---

Published as a conference paper at ICLR 2025
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps.
Step 4: Identify the correct value for
From sympy import symbols, Eq, sclve, N
Reuse earlier code
symbols('r')
al, b1 27, 99
eg Eq (al r*414, b1 r.+10)
solution = seqve r
The correct value for is the positive real solation
t_value max (sxl for sol in solutien if sol.is_real. print (h" conmon ratio r is: (Nie_value, 6))*)
>>>
output
The common ratic r is: 1.38378
Let's continue with the next steps solve this problem and leveraging the outputs of previous steps. Step 5: Calculate a9
From sympy import symbols, Eq, sclve,
Reuse earlier code
symbols('r")
al, bl 27, 99
eq Eq (al I++14, b1 + r++10)
solution = seq,
r_value max for sol in solutien if sol. is_real]
Calculate a9
a9 al r_value*48
print (E"a9 (N(, 10))")
>>> output
a9 063.0000000
### END CODE
The final answer is 363
A.13 RESULTS WITH STANDARD DEVIATIONS
We present the main table results along with standard deviation scores Table I7
Table Benchmarking SBSC against different math reasoning methods across three datasets. We report average accuracy over runs with standard deviation within parentheses. Best result in each setting is highlighted bold and second best is underlined. Absolute improvement in performance by SBSC over the previous best method in each setting is indicated in subscript.
MaihOdyssey
Olympiad Bench
greedy
maj@7
greedy
maj 0
greedy
maj maj@7
greedy
maj @
Claudo-3.5-- S-S-nntt
COT
31.16
35.79
9.09 (2:10)
10.91
11.89 (10.6)
16.89
39.35 (110.47)
42,46
PAL
35.79 (+100)
36.42
27.48 (+0.6)
28.79
27.23 (+0.6)
31.01
41.07 (+0082)
44.44
TIR-ToRA
38.59 (+0.6)
24.64 (+3.2)
26.67
27.23 (+0.6)
32.43
47.69 (+0047)
SBSC (Ours)
49,33 (+:3.1)
-06.2
35.45 (+117))
-18.7
39.86 (+1.0)
53.31
(2:0.91)s.
-12.7
GPT-4o
COT
35.94 35.94 (.10.))
37.47
10.39 (112.))
12.12
13.51(1100)
17.57
41,80 (.11.8))
47,22
PAL
36.48 (2:0.6
38.11
24.63 (10.6)
26.97
15.74 (20.6)
20.27
41.67 (#:2.16)
46.43
TIR ToRA
37.33 (122.5)
40.42
22.42 (111.)
25.45
19.59 (12.6)
23.64
43.32 (.11.70)
49.61
SBSC (Ours)
44.55 (2:0.6))a.a
*54.3
30.7 (=1.1)aa
719.7
26.55 (11.1)m
712.8
48.74 (1).))))ss.
*jo.87
14 LEAST-TO-MOST PROMPTING
Least-to-Most (L2M) (Zhuu et al. 2022) is two-stage prompting strategy where the aim is: in first stage, to break down a complex problem into a series simpler subproblems and then, in second stage, solve these predefined subproblems, PAL (Gao et al. 2022) reported 1.2M version
50

---

Published as a conference paper at ICLR 2025
of PAL in their work. We follow the reported prompts and replicate it by designing exemplars for both the stages. We find L.2M-PAL inherits the same issues that PAL & TIR-TORA has. L2M-PAL comes up with entire sub aub-problems at once and also its uses single program-block block to solve those sub-problems. SBSC dynamically generates the next sub-task and the corresponding program to solve it leveraging the previous turns results. In Table T8] show the results obtained from 1.2M + PAL using Claude-3.5-Sone
on our AMC and AIME test datasets. Even after allowing sell-correction for stage with max turns n-15, L2M-PAL approaches PAL scores. Hence for main results, we stick to PAL & TIR- TIR-TORA along with self- ceff-consisteney (Shao et al. 2024) due to resource optimisation and wide adaption of those prompting strategies for math-pree tott-problem solving.
Table 18: Least-to-Most Prompting results on AIME and AMC
Method
AMC
AIME
greedy
maj
greedy
maj@7 @7
COT
31.16
35.79
9.09
10.91
PAL
35.79
36.42
27.48
28.79
L2M-PAL (n=l)
33.47
38.53
25.45
28.79
L2M-PAL (n=15)
34.32
25.45
TIR ToRA
38.59
43.16
24.64
26.67
SBSC (Ours)
49.33
16.2
35.45
16.7
51


Reviews:
[{"review_id": "35hcXaGDG8", "reviewer": "Reviewer_f1zY", "summary": "The paper introduces a step-by-step prompting technique for solving math problems using large language models (LLMs). In this approach, the model is prompted to generate sub-tasks in code format at each step, which are then executed to produce intermediate outputs. These outputs, along with the code from previous steps, are used to prompt the model again to solve the problem sequentially. The proposed method is tested a few benchmarks, including MC, AIME, and MathOdyssey, demonstrating improvements over existing prompting techniques.", "questions": "I believe that randomly sampling 4 examples out of a set of 10, even multiple times, may not provide sufficient variety to effectively test the sensitivity of the prompts. It's unclear how diverse the 10 exemplars are to begin with. Perhaps a more meaningful  test would be to have a larger pool of exemplars?\n\nIn figure 5, are the methods compute/token matched? What is the ratio of the tokens generated by the SBSC method compared to others such as TIR-ToRA? \n\nFigure 7 seems to have the wrong caption.\n\nWhat is the advantage of this step-by-step approach compared to other similar methods, particularly in cases where the model is unable to revisit and revise a problematic step (e.g., Step 1) once it has progressed to later steps?\n\nIn table 1, why is there no number for maj@7 for SBSC?\n\nHow does the method perform when used with open LLMs such as LLaMA?\n\nWhat types of errors are reported in Section 5.3? Are they primarily syntactic or semantic in nature?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "A key strength of this paper is the step-by-step approach, which leverages inference time compute to break down complex problems into manageable sub-tasks. For each sub-task, SymPy-based Python code is generated, and the model is instructed to include a print statement at the end of each code snippet so that intermediate execution outputs can be generated. \n\nThis structure allows the model to build on the results of previous steps, using the executions output/feedback to condition its generation of the next step and sub-task. \n\nThe step by step process with intermediate feedback provides the model with a chance to detect and correct errors or rewrite parts of the code. This improves the overall performance of the model.", "weaknesses": "The paper's approach relies heavily on the careful design of the step by step coding framework exemplars. This with the inclusion of detailed comments within the python codes in these exemplars seems to require considerable manual effort. While this strategy is effective for the test sets discussed, it may be overfitted to these specific benchmarks. This raises some concerns about the generalization of the method to broader problem sets or applications.\n\nAnother limitation is that the method is highly dependent on the coding capabilities of the underlying language model. While the authors acknowledge the correlation between the model's coding performance and the accuracy of the SBSC approach, the exact nature of this correlation is unclear. \nIn addition, the method relies heavily on SymPy which might further limit its flexibility and adaptability.\n\nThe approach demands a relatively long context length, which could be a constraint in certain settings. It would be helpful if the authors specified a rough minimum context length required for the given shots and the math problem. This would help get better insights into the method's scalability.\n\nLastly, the discussion of the related works section seems somewhat rushed. In my opinion, the first sentence feels awkward and reads more like filler rather than contributing meaningfully to the content. A more thorough comparison with existing literature on program or tool-aided generation (the section following POT and PAL) could be extended, as it is the most relevant part of the discussion. This would provide deeper insights into the novelty and strengths of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgDIhFpUlR", "reviewer": "Reviewer_EW8i", "summary": "The paper proposes Step-by-Step Coding (SBSC) for solving complex mathematical problems for large language models. The authors conduct extensive experiments comparing SBSC against other state-of-the-art methods like COT, PAL, and TIR-ToRA on datasets like AIME, AMC, and MathOdyssey.", "questions": "* Has SBSC demonstrated improvements across all domains of mathematical reasoning? Are there instances where multi-step code decomposition is not feasible or effective?\n\n* Given the authors' claim regarding enhanced performance in Olympiad competitions, it would be beneficial to include benchmark results from:\n    * JEE-Bench\n    * OlympicArena\n    * Omni-MATH\n    * Additionally, consideration should be given to experiments involving GSM8K and MATH, as many problems in MATH derive from AMC and AIME questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The paper is easy to understand, with a well-defined problem statement that introduces multi-turn TIR as a solution for complex multi-step issues.\n* The main experimental results are sufficiently thorough, providing reasonable evidence for the improvements attributed to SBSC.", "weaknesses": "* SBSC introduces an additional step of decomposing a large problem into smaller sub-problems during the decoding phase. How to ensure that this decomposition is both correct and rational? Could this lead to the introduction of additional errors?\n\n* Existing methods, such as TIR-ToRA, output a planning-like COT before code generation. Does this serve the same function as the multi-turn approach proposed in this paper? Is it merely a modification that incorporates multiple compiler feedback loops?\n\n* The introduction of multi-turn in SBSC may result in decreased efficiency. The manuscript should include comparisons under equivalent token costs.\n\n* I strongly recommend that the authors review their writing for numerous typographical errors, particularly the inconsistent use of citet and citep.\n    * line 157: \"Our inference procedure is inspired by ToRA Gou et al. (2023)\"\n    * line 196: \"AIME, AMC, MathOdyssey Fang et al. (2024) and OlympiadBench He et al. (2024)\"\n    * line 199: \" MathOdyssey Fang et al. (2024), a popular benchmark...\"\n    * ....", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L9yxniWPBe", "reviewer": "Reviewer_BqFn", "summary": "The paper presents Step-by-Step Coding (SBSC), a framework that guides Large Language Models (LLMs) to solve advanced math problems by producing a sequence of programs. SBSC operates iteratively, using the outcome of code from previous steps to inform the generation of subsequent sub-tasks and solutions. This approach is shown to be more detailed, adaptable, and accurate than existing methods. The efficacy of SBSC is supported by extensive testing on competition-level math problems. For the Claude-3.5-Sonnet model, improvements using SBSC with greedy decoding are significant, with performance gains of 10.7% on AMC12, 8% on AIME, and 12.6% on MathOdyssey over current state-of-the-art strategies. Additionally, when comparing SBSC's greedy decoding to self-consistency decoding of current methods, there are notable increases of 6.2% on AMC, 6.7% on AIME, and 7.4% on MathOdyssey.", "questions": "Apart from the issues expressed in the weaknesses section, I have no other concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The empirical results demonstrate that SBSC achieves superior performance over state-of-the-art methods, suggesting that its iterative nature allows for more accurate problem-solving in the context of advanced math problems.\n- The paper's experiments cover a variety of math competitions, indicating that SBSC is not tailored to a specific type of problem but is instead broadly applicable to Olympiad-level mathematics.\n- The paper is well-structured and clearly written.\n- The ablation and analysis section offers a deep analysis of the results, providing insights into why SBSC performs better than existing methods and under what circumstances it might be most beneficial.", "weaknesses": "I appreciate the efforts made in this paper to apply step-by-step partial code generation and execution within the challenging domain of Olympiad-level mathematics and the promising results. My query pertains to the aspect of novelty concerning this approach. The paper states in Line 62 that \"Fundamentally, both PAL & TIR-ToRA generate a single program block to solve the entire problem\", highlighting step-by-step code generation and execution as a key contribution of SBSC. However, given that the concept of language models incrementally generating code to interface with external tools has been explored in prior research, such as ReAct (https://arxiv.org/abs/2210.03629) and ToolFormer (https://arxiv.org/abs/2302.04761), and the idea is highly related to OpenAI's code interpreter and assistant APIs, I am curious about the distinctiveness of the approach presented in this paper. May I kindly suggest that the authors could provide a more explicit comparison with these precedents? An elaboration on how the proposed SBSC framework diverges from or advances these existing methods, particularly in the unique setting of mathematical problem-solving, would be enlightening. This addition would help the readers and reviewers to better understand the specific innovation and value of the proposed work in the broader context of step-by-step code generation research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RcxlnnYQRk", "reviewer": "Reviewer_uQkW", "summary": "This paper introduces Step-by-Step Coding (SBSC), a multi-step reasoning framework where LLMs are prompted to break down complex questions into sub-questions and solve them iteratively with the help of generated code. The experiments demonstrate that SBSC achieves higher accuracy compared to Chain-of-Thought (CoT) and Program-Aided Language (PAL) on competition and Olympiad-level math problems, regardless of problem topics or code errors. The authors also show that self-consistency further improves SBSC's performance.", "questions": "1. The mainly question is whether using code to solve oplymipcs-level math problem is a reasonable approach. Both baselines PaL and ToRA are not designed for such difficult problems, which often require more than basic operations or listing equations, such as some extra theorems cannot be solved by code. In Figure 1, there are several points where the LLM's use of Python code seems unnatural. First, the code is strongly based on some black-box high-level function provided by some libraries, like `summation` in `sympy` library to get a lemma \"$1^3+\\dots+n^3=\\frac{n^4}{4}+\\frac{n^3}{2}+\\frac{n^2}{4}$\". It is not a natural usage of codes (at least Python codes) because for more challenging tasks, the LLM could not find a propriate library provides the information. Similarly, the program's brute-force traversal from $n=1$ to $1000$ is not a typical way to solve such problems. A more natural way would involve algebraic manipulation like $$ \\frac{n^4}{4}+\\frac{n^3}{2}+\\frac{n^2}{4}\\equiv17(\\mod n+5)$$ $$ n^4+2\\times n^3 + n^2\\equiv68(\\mod n+5) $$ $$ (-5)^4+2\\times(-5)^3+(-5)^2\\equiv 68 (\\mod n+5) $$ $$ 400 \\equiv 68 (\\mod n+5) $$ so that $(n+5)\\mid 332 \\Rightarrow n +5 =83 \\text{ or } 166 \\text{ or } 332 \\Rightarrow n = 78, 161, 327$ then we final check these candidates and get the final answer $n=78\\text{ and }161$. The process shown in Figure 1 shows that using Python code is not a natural and reasonable way to solve these problems. The authors should further compare SCSC's approach to human-like reasoning on a subset of problems or some examples, or to analyze which types of Olympiad problems are most/least suited to SBSC's code-based approach.\n\n2. While step-by-step reasoning is a natural approach, writing and displaying code in steps is not common practice. Programmers often organize their code by defining classes and functions first, then calling them in a main function. How can we ensure that LLMs are capable of generating effective step-by-step code? I believe that the authors should add some experiments to support current LLMs have this ability. Additionally, some steps in the reasoning process may not correspond to clear intermediate results in code. For instance, what happens if a step only defines a function with no immediate output to display? I'm curious whether this decomposition approach remains effective across different problem types.\n\n3. Does the method's effectiveness depend on delicate prompt engineering or careful selection of few-shot examples? Although the experiments in Section 5.1 suggest the method is not overly sensitive to the few-shot candidate selection, I would like more clarity on the details and principles behind crafting these 10 examples. When generalizing the method to other tasks, how should the steps be divided to ensure each step yields clear, intermediate results that aid subsequent reasoning?\n\n2. Minor questions:\n    1. The authors say \"similar to NuminaMath, we remove all the answer choices from each AMC-12 question and modify the question to ensure an integer answer\". But I cannot find evidence that the NuminaMath modifies questions to ensure an integer answer. Rather, it is a preprocess in original AIME dataset to keep answer as integer and Numina focuses on this problem only for decontamination. The authors should clarify this point.\n    2. In Figure 4, how is the x-axis \"error steps per problem\" determined? A more detailed explanation of this experiment would be helpful.\n    \nI would raise my rating to 8 or higher if the authors can address my two main concerns (questions 1 and 2).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The concept of step-by-step, multi-turn coding is both intuitive and innovative. Unlike previous approaches that focus mainly on validation after the final generation or self-debugging, SBSC’s use of intermediate results from an executor provides more actionable and informative feedback. This method effectively combines task decomposition from CoT with the program-assisted capabilities of PAL, offering a promising approach.\n\n2. The experiments on Olympiad-level math problems clearly show SBSC’s effectiveness, outperforming the baseline by a large margin.", "weaknesses": "1. While the idea has potential for broader applications, the authors only tested it on Olympiad-level math problems. Additionally, while SBSC appears promising, is it appropriate to apply this method to such difficult problems? (See more detailed discussion in question 1.)\n\n2. The baseline comparisons could be more comprehensive. Although the comparison with CoT and PAL is sufficient to demonstrate SBSC's effectiveness, it does not fully showcase its strengths or weaknesses relative to other complementary methods, such as multi-turn program validation, self-debugging, the use of additional tools, or expert models. Even if such comparisons might be unfavorable to SBSC, they would provide valuable context for understanding the broader landscape and gauging SBSC's relative contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)