CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
Authors: Yongchao Chen
Yilun Hao
Yueying Liu
Yang Zhang
Chuchu Fan
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Code Interpreter
Code/text generation
Symbolic computing
Model fine-tuning
Model reasoning and planning
Primary area: deep_learning->large_language_models
TL;DR: Method CodeSteer to augment LLM capabilities by guiding LLM code/text generation, and SymBench for evaluation of symbolic tasks.

Abstract: Existing methods fail to effectively steer Large Language Models (LLMs) between textual reasoning and code generation, leaving symbolic computing capabilities underutilized. We introduce CodeSteer, an effective method for guiding LLM code/text generation. We construct a comprehensive benchmark SymBench comprising 37 symbolic tasks with adjustable complexity and also synthesize datasets of 12k multi-turn guidance/generation trajectories and 5.5k guidance comparison pairs. We fine-tune the Llama-3-8B model with a newly designed multi-turn supervised fine-tuning (SFT) and direct preference optimization (DPO). The resulting model, CodeSteerLLM, augmented with the proposed symbolic and self-answer checkers, effectively guides the code/text generation of larger models. Augmenting GPT-4o with CodeSteer raises its average performance score from 53.3 to 86.4, even outperforming the existing best LLM OpenAI o1 (82.7), o1-preview (74.8), and DeepSeek R1 (76.8) across all 37 tasks (28 seen, 9 unseen). Trained for GPT-4o, CodeSteer demonstrates superior generalizability, providing an average 41.8 performance boost on Claude, Mistral, and GPT-3.5. CodeSteer-guided LLMs fully harness symbolic computing to maintain strong performance on highly complex tasks. Models, Datasets, and Codes are available at https://github.com/yongchao98/CodeSteer-v1.0 and https://huggingface.co/yongchao98.

Chuchu Fan CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
Yongchao Chen 12
Yilun Hao
Yueying Liu
Yang Zhang
Chuchu Fan
Abstract
1. Introduction
While the reasoning and planning capabilities of LLMs have improved significantly (Wang al., 2024; Chen et al., 2024c; Li et al., 2023), they still fail in ostensibly simple tasks (Zhou al., 2024a). Crucially, many tasks in exist- ing benchmarks-suue as Blocksworld (Valmeekam et al., 2024) and Game 24 (Zhou et al., 2023b) can be completely solved with code solutions. based reasoning excels at semantic understanding and commonsense inference less suited for exact computation, symbolic manipulation, optimization, and algorithmic processing (Valmeekam et al., 2022). In contrast, symbolic computing via code generation is adept at handling rigorous operations and can easily lever- age specialized tools (c.g.. equation solvers), In many tasks, prompting LLMs to generate and execute code outperforms purely textual reasoning (Madaan et al., 2022: Liang et al., 2022; Chen et al., 2022).
Existing methods fail to effectively steer Large Language Models (L.LMs) between textual reasoning and code generation, leaving sym- bolic computing capabilities underutilized. We introduce CodeSteer, an effective method for guiding LLM code/text generation. We construct a comprehensive benchmark SymBench comprising 37 symbolic tasks with adjustable complexity and also synthesize datasets of 12k multi-turn guidance/generation trajectories and 5.5k guidance comparison pairs. We fine-tune the Llama-3-8B model with newly designed multi-turn supervised fine-tuning (SFT) and direct preference optimization (DPO). The resulting model, CodeSteerl.LMM augmented with the proposed symbolic and self-answer checkers, ef- fectively guides the code/text generation of larger models. Augmenting GPT-4o with CodeSteer raises its average performance score from 53.3 to 86.4, even outperforming the existing best LLM OpenAl ol (82.7), ol opreeviww (74.8), and DeepSeek R1 (76.8) across all 37 tasks (28 seen, 9 unseen). Trained for GPT-4o, CodeSteer demonstrates superior generalizability, providing an average 41.8 performance boost on Claude, Mistral, and GPT-3.5. CodeSteer-guided LLMs fully harness symbolic computing maintain strong performance on highly complex tasks. Models, Datasets, and Codes are available at https://githi: cww wwt.e/oo/myya.oe CodeSteer-vi. yongchaogccch////o/o
A key challenge is guiding LLMs to decide when rely textual reasoning versus programmatic solutions, given that most input questions lack explicit cues about which approach is best. Recent OpenAl GPT models address this by providing Code Interpreter module, allowing the model to iteratively generate and exccute code, then further rea- son with the output (Achiam al., 2023). Multi-agent frameworks like AutoGen (Wu al., 2023) adopt spe- cialized system prompt to steer LLM for code generation when needed. However, recently Chen et al. (2025) finds that all these existing methods struggle to effectively steer between textual reasoning and code generation, failing to fully leverage symbolic computing capabilities.
Our work tries bridge this gap by developing an assistant framework (CodeSteer) guide the code/text generation of the LLM solving the task (TaskLLM). By fine-tuning small model (Llama-3-81 (Dubey et al., 2024)) to be the assistant, we enable large models (GPT-4o (Achiam et al., 2023)) to fully leverage symbolic computing via code gener- ation while preserving other capabilities. Recognizing that iterative "executing and exploring" the most effective way to solve tasks, we build CodeSteer to generate prompts that guide the TaskLLM through multiple turns of interaction before finalizing answers.
and https:
/ //huggingface haeeeeeooooooooo co/yongcha098.
'Massachusetts' Institute of Technology, Boston, MA, USA 'Harvard University, Boston, MA, USA *University of Illinois Urbana-Champaige
Urbana, IL, USA 'MIT-IBM Watson AI Lab, Boston, MA. USA. Correspondence to: Yongchao Chen <yongchaochen@faan har haw.aaaddd..aaaaa.eaaaa.aaaa.a harvard.edu>, <chuchu eit.edu..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To achieve comprehensive evaluation, we gather and de- velop a benchmark with 37 symbolic tasks, referred as Sym-

---

CodeSteer: Symbolic-Augmentedd Language Models via Code/Text Guidance
Game 24
BoxLift
Unseon Tasks
Question: Usee numbers and basic arithretic operations 1+ /1 to obtain
Question: You are given list or boses with the following veights: (312,,. And with
Inputi 3, 4,
lifters (iftinn capocities: (68., 1891. Your task is assign the Lifters to lift all the z00es in suttiple steas, following rutes- CodeSteee Guidance python coding block) Suppent pythen execute.
Symieechh Score
CodeSteer Guidance 1: Utilirs your progracming expertise address tasks-
70-
TaskILIM Ansver 1: To solve this -nasitted texts.
(8 4) (3 4)
TaskLLM Answer 1: To solve this prootes, se nees co assign Lifters to boxes, -neiitte texts.
T python d 3,
30-
X
_roeitsed codex..
stops boxes:
Calcslate the result result
[]
X
30
6F AonCode 30 Uttrppetee Culkeeer
OpepSaekk 01 previce
* Print the rital snswer in the required forest
printi (03) a (fc) (b)) (resulthoo")
B while Len((boces):
rentining Lifters(s]
CodeSteer Guidance 2: The cuerest upprosch is incorrect becasse... To salve the problee, cansider core explare possible rueiiations.... Ieplesent search atgeritte that...
liffers
Number Multiply, (1)
approace
tota Caapacity
for J. Litter in
Question: What is the result 35+277
(emeerate(reeatt1rg capacity lifter box: used iifters.apeend
CodeSteer Guidence 1: Acquire the final answer with step by step textual anolysis-
TaskLLM Answer 2: .ranttted
tyyhonn
.conitted codes..
multiplicatin lieaston
TaskLLM Answer
step, .ccmitted
break
frox itertcols Import persutations, product
ffllmers: find_24.1
The current code rens into infinite Losp. Try to use core efficiert searching.
CodeSteer Guidence 2:
The tinel ansver 1st -595
CodeSteer Guidence Return Answer
def find 24nuubrin:::
aps = [1=', (-1, he', 1/1]
TaskLLM Answer 2: understand, _nceitied coden..
toxts.
for nun nun_pern in persutation
s0 pem in productlops, repeat=3):
Number Multiply, (2)
parenthesize the expression expressiinn a
all
CodeStoar Guidance This code still ruts ieploceeting
Question: shat is the result -338225433433327
"c(tuu (e11 <op por_peri)]>>
TasKLLM Answer 3: Here is the, s008stter toxt> python
CodeSteer Guidence 1: Goterate code to solve it.
(og_sera(2)) Irun, 13111
(nux ppn_p1e top pooperrtll). (onu_pemml1)
-ooiited codon..
TaskLLM Answer 1:
"" persiel) (co_perstd
python
(nux_pemm333]] (og_perat3l) (nun_permi21l)
CodeStoer Guidance 4: It seees cansot write the correct code. Directly use textual reasoning to.
(og_peru(2)) 1311"
-5343235
e"ctuu (e11 (og_pernl
TaskLLM Answer Let us anelyze step by step, texto-
num2 143433327
[(uusserrt11)>> (oo_peratal) (nun_perm (211)) (oo_scra(2) (run_porm133)> (nun_pera(e)>>>>>> (op_pemtal>
resuitt sunt AUR2
Step [1400, 13, 1, 9)), 1231,
print("The resuit -5343235 143433327 151",
-soxitted coccx.
Step 2: [(359, 14, 1, 01), 1238, [2
Step 31 wconttted textr..
19 11
CodeSteer Guidance Reture
CodeSteer Guidance 3: Rature Answer
CodeStear Guidance
Raturn
Figure 1: Examples and performance of CodeSteer on guiding LLM code/text generation to integrate symbolic computing, At each interaction with LaskLLLM, it reviews current and previous answers, then provides guidance for the next turn. CodeSteer returns final answers when it deems them ready. With CodeSteer, GPT-4o outperforms OpenAl Code Interpreter, o1, and ol -prevview models.
Bench. On SymBench, augmenting GPT-4o with CodeSteer greatly improves its average performance score from 53.3 to 86.4, even outperforning the current leading pure-text model, OpenAI ol (82.7) (Jaech et al., 2024) and DeepSeek RI (76.8) (Guo et al., 2025). Although trained for GPT-4o,, CodeSteer shows great generalizability, delivering an aver- age 41.8 performance gain on Claude-3-5-Sone --Snnet,, Mistral- Large, and GPT-3.5. By fully leveraging symbolic comput- ing, CodeSteer Sodeetter-guided guided LLMs maintain strong performance on highly complex tasks even when ol fails in all testing cases. Our key contributions are:
2) New methods for dataset construction and model fine-tuning of SFT and DPO: We fine-tune Llama-3- 8B with the synthesized datasets of 12k multi-turn guid- ance/generation trajectories (SFT) and 5.5k guidance com- parison pairs (DPO). Unlike standard multi-step settings, in CodeSteer's 'ss multi-turn guidance, the TaskLLM outputs complete answer each turn rather than only at the end. Consequently, we introduce novel components to both the dataset construction and training processes for SFT and DPO, such as data synthesis of dynamic guidance adapta- tion, emphasis on the final two turns in SFT, comparison score design, and efficient answer sampling in DPO. These modifications result in better performance. Both the final CodeSteer model and created datasets will be released.
1) Developing and publishing SymBench: Prior works by Chen et al. (2025) and Gui et al. (2024) gathered and de- veloped 14 and 31 tasks, respectively, targeting challenges in computation, symbolic manipulation, logic, optimization, spatial reasoning, and constrained planning. However, nei- ther study published the complete code for question/solution synthesis or the full datasets. From these 45 tasks, we se- lect 37 that remain challenging for GPT-4o and redevelop their generation code to produce samples with adjustable complexity. We refer to this newly published benchmark as SymBench.
3) Symbolic checker and self-answer checker: Observing that TaskLLM frequently produces text-like code that hard- codes answers, neglecting efficient symbolic computation, we introduce Symbolic Checker help LodeSteerLLMM evaluate code complexity and efficiency. Since most reason- ing and planning tasks can be better verified with coding, we add a correctness of CodeSteerLLMM These two new checkers have been proven to significantly improve the efficiency of
Checker for better judgment of answer
2

---

CodeSteer: Symbolic-Augmentee Language Models via Code/Text Guidance
dataset synthesis and CodeSteerL.LM fine-tuning.
3) Answer Finalization When Ready
4) Proposed CodeSteer Outperforms Nine Baselines and ol: CodeSteer's superior performance highlights the im- portance of enhancing LLM reasoning and planning with symbolic computing. This also demonstrates the potential for steering large models to generate smarter code and text by leveraging specialized smaller models.
The main components of CodeSteer are as follows: CodeSteerLLMM is the primary model fine-tuned and used to guide TaskLLM in code/text generation. The input prompt formats for the first and subsequent turns are pre- sented in Appendix Sec. D. To facilitate answer evalua- tion, CodeSteerLLM is equipped with two checkers -eekf answer and Symbolic-whose design is inspired by the inherent features of symbolic tasks.
2. Symbolic Tasks and SymBench
Self-answer Checker re-queries TaskLLM to generate and execute code for verifying its current answer, then returns the evaluation results and explanations to CodeSteerL Since many symbolic tasks benefit from code-based veri- fication, this approach often provides a more reliable per- spective. The prompt format for the Self-answer Checker provided in Appendix Sec. E.
Challenges in Code/Text Choices For tasks requiring computation, symbolic manipulation, logic, optimization, spatial reasoning, and constrained planning, coding- coding-based symbolic computing often more effective than text-based approaches. However, Chen et al. (2025) found that steering LLM code/text generation poses significant challenges, even in tasks with apparent symbolic characteristics. The main bottlenecks are: 1) Deciding whether code or text is sim- pler depends on task type, task complexity, and the LLM's capabilities, which is hard to judge (see Appendix Sec. A). 2) LLM-generated code often appears as text-like scripts that merely hard-code answers rather than enabling efficient symbolic computation, echoing the phenomenon described in Yang et al. (2024) (see Appendix Sec. B).
Symbolic Checker is a rule-based script to analyze the generated code for iteration, search, numeric handling, per- mutations, and combinations, then returns a complexity summary and score. This helps LodeSteerLLM determine whether the code sufficiently sophisticated for the task at hand. Since TaskLLM often produces text-like code prone to errors, the Symbolic Checker's complexity assessment aids, but does not solely dictate, CodeSteerL decisions. Further details on the checking code and prompt are in Ap- pendix Sec. F.
SymBench Chen et al. (2025) and Gui et al. (2024) col- lected 14 and 31 tasks with symbolic factors from vari- ous benchmarks such as Suzgun et al. (2022); Chen et al. (2024d); Yao et al. (2024); Cobbe et al. ); Hendrycks al. (2021), but their plete datasets remain private. We redevelop the generation code to automatically synthesize questions with adjustable complexity. Our resulting set of 37 tasks covers reasoning, planning, and execution, testing competencies in mathemat- ics, spatial reasoning, logic, order reasoning, optimization, and search. Details and categorization are provided in Ap- pendix Sec. C and Table 5.
Beyond enhancing CodeSteerLLM's performance, the Self- answer and Symbolic Checkers also streamline dataset syn- thesis for SFT and DPO fine-tuning, as discussed in the following sections.
code and com-
4. Fine-tuning the CodeSteerLLM
Among the three modules of CodeSteer, the CodeSteer- LLM needs to be fine-tuned to perform the complicated task of steering. The fine-tuning is performed on a subset of SymBench. Specifically, we randomly select 28 of the 37 SymBench tasks, using a distinet set of samples with- out overlap with the test samples. This setup allows us to evaluate CodeSteer on 28 seen tasks (with different test sam- ples) and on the remaining 9 unseen tasks. The fine-tuning consists of two steps. We first fine-tune the Llama-3.11 1-BB model with SFT, then further optimize using DPO. Both processes are fine-tuned with full parameter on 4*00 GPUs for 4-10 epochs. The detailed parameter and hard- ware settings for fine-tuning and inference processes discussed in Appendix Sec. H. We synthesize 12k multi-turn trajectories for SFT and 5.5k guidance comparison pairs for DPO. The specific data number for each task is in Appendix Sec. G.
3. CodeSteer Framework
Fig 1 illustrates how CodeSteer guides the LLM's code/text generation. At each turn, CodeSteer reviews the LaskLLM's current answer and the guidance/answer history, then de- cides whether to offer new guidance or finalize the response. It performs three key functions:
1) Initial Method Selection In the first turn, it chooses whether to solve the task with code text (c.g., use textual reasoning for small-number multiplication, and code for large-number multiplication in the task Number Multiply). 2) Dynamic Adaptation In subsequent turns, it refines guidance or switches methods if issues arise (e.g., encourag- ing more sophisticated symbolic approaches in Game 24, or switching to textual reasoning after multiple incorrect code attempts in BoxLift).
guidance/generatinn
3

---

CodeSteer: Symbolic-Augmentter Language Models via Code/Text Guidance
4.1. Multi-turn SFT
illustrates our framework for sampling DPO guidance pairs in a multi-turn setting. The main challenge is sampling and selecting guidance pairs that exhibit clear performance dif- ferences across various turns while minimizing the number of samples to conserve resources. We use a tree structure where each node represents a guidance, with a branching factor of or 3. To compare guidance pairs from the same parent node, we calculate their Performance Scores using the following equation:
To generate supervision data for SFT, we prompt the GPT-40 to serve both the guiding LLM (i.e., the L)))SteerLLM) and the TaskLLM to generate multiple guidance/generate trajectories. We then filter the trajectories keeping only those that produce correct answers. To improve success rates, CodeSteerLLM's prompt is more detailed and in- cludes pre-set knowledge or hints. To increase dataset diver- sity and enable dynamic adaptation of guided thoughts, this prompt also has different versions. For example, we may let GPT-4o choose all guidance styles, or enforce transitions from code to text or text to code. We set the maximum guidance turns to be 5 and return the final answer once that limit is reached.
1 -i
ending turn/correct, ending turn/incorrett, cGI Σjec(a) Score, otherwise.
Score;
(I) Here, Score; represents the score for a node at turn i, where is the current turn number, and C()) is the set of child nodes of node i. If the current turn is the final one, Score, is set to 15 i for correct answers and -i for incorrect ones. This incentivizes LodeSteerI.LM achieve correct answers in the fewest turns possible. For non-final turns, Score; is calculated as the average of its child nodes' scores. This ensures that each non non-terminal turn's score reflects the average performance of its potential subsequent actions, i.e., the expectation.
Multi-turn Gradient Cancellation Issue multi-turn trajectories, the SFT process incorporates gradients from each turn. This can lead to gradient cancellation in the early turns. For example, in one task, both [code, return answer] and [text, code, return answer] produce correct results, so if both trajectories are used for fine-tuning, the SFT cannot learn that code is the better first step.
Data Augmentation To mitigate this issue, we leverage the fact that the final two turns of guidance are most influen- tial, the TaskLLM produces new answers each turn while earlier turns primarily provide background. Consequently, we augment the SFT dataset by doubling the weights of the final two turns.
DPO data is collected from guidance pairs within the same parent node at each level that have score difference greater than 2. To prevent reward hacking (Skalse et al., 2022)-wherd CodeSteerL might bypass exploration and return incorrect answers quickly (e.g.. preferring score of -2 over -5)-ee we include only pairs where at least one guidance has positive score. To obtain diverse guidance answers, we set the inference temperature 1.5 for the SFT fine-tuned CodeSteerL and use three models fine-tuned at different epochs (6. 8, and 10) compare their guidance responses for the same parent node.
4.2. Multi-turn DPO
Question
6.5
13
3
7
6.5
5. Experiments
13
13
10
Experimental settings We use GPT-4o as the TasKLLM to test 28 seen and 9 unseen tasks, each with 100 samples of varying complexity. The samples for the 28 seen tasks are different from those used to train CodeSteerLLM. Addi- tionally, we evaluate other LLM types to assess CodeSteer's generalizability.
3
10
10
10
DPO data
Figure 2: Schematic of multi-turn DPO data sampling: blue squares represent intermediate (non-final) turns, and brown ovals mark finalizing turns. Guidance responses from the same parent node in LodeSteerLLM are compared gener- ate the DPO data.
We compare CodeSteer to six training-free and three training- baselines, with methods 1, 3-6, and 9 origi- nally proposed in Chen et al. (2025).
Training-free Baselines 1) No extra modifications but only input the original question (Only Question); 2) Our framework in Sec. 4.1 to synthesize SFT dataset, where GPT- 4o works as Lod with extra hints (Symbolic Agent): 3) Prompting LLMs to answer with only text with CoT (All Text + CoT); 4) Prompting L.MMs to first analyze
Because many correct trajectories in the SFT dataset are still suboptimal, we need to further fine-tune the CodeSteer- LLM on pairs of trajectories labeled with preferences. Here we use rule-based scores to assign preferences. Figure? 2
4

---

CodeSteer: Symbolic-Augmentte Language Models via Code/Text Guidance
Table Experimental results SymBench. Methods with the highest scores are highlighted blue.
Methods
CoT LLMS
Training-free Methods
Training-base -base Methods
Task success %
Code Text + Sum.
Code + Text Sam.2
GPT-4o CodeSteer
Code/Text Choice
Only Question
Symbolic Agent
All Text + CoT
All Code + CoT
AutoGen Conca.
Code Interpreter
DeepSeek RI
of preview
Ave. Norm., Seen
83.8
79.3
77.9
59.3
77.0
56.7
71.6
73.2
66.7
65.8
79.7
73.3
88.1
Ave. Norm., Unseen
79.4
69.1
65.1
34.5
67.9
37.9
63.2
59.5
51.9
51.7
72.1
61.9
81.3
Ave. Norm., Total
82.7
76.8
74.8
53.3
74.8
52.1
69.6
69.9
63.1
62.4
77.9
70.5
86.4
Seen
Tasks
Game 24
80
65
78
37
23
11
33
43
43
18
93
Plan
74
43
66
61
73
BoxLift
95
92
85
69
58
56
68
20
65
60
73
49
77
BoxNet
45
43
54
37
30
30
12
23
21
23
37
29
Blocksworld
100
100
77
43
60
52
32
50
50
48
44
42
52
Date Understanding
87
88
87
89
88
72
86
84
86
76
87
Web Lies
100
100
98
99
86
91
77
80
98
94
Logical Deduction
100
98
97
89
93
91
83
82
94
90
94
82
92
Navigation
100
100
100
98
93
95
99
91
96
94
92
98
99
GSM-Hard
79
77
71
78
76
80
83
81
78
77
79
77
MATH Geometry
94
73
77
76
73
MATH Count&Probab.
96
97
95
88
87
88
91
86
88
84
89
93
Logical Equation
100
100
100
52
50
52
40
48
30
33
56
71
78
New Operator
44
39
25
42
39
45
39
47
56
38
48
48
40
Pooling
46
40
42
46
60
43
47
40
49
46
Light Puzzles
100
56
92
73
Mahjong
96
98
93
66
77
73
80
94
72
74
96
64
90
Statistical Counting
25
72
78
34
93
32
95
93
93
86
95
89
97
Matrix Transformation
87
100
98
94
96
76
97
97
96
92
97
90
98
Logical Puzzle
58
41
44
44
Cons. Linear Arrange.
74
62
81
71
60
72
71
77
72
Pattem Recognition
100
100
100
70
90
44
89
100
56
60
94
100
93
String Insertion
96
49
72
100
100
100
67
75
100
89
100
Letter Logic Diagram
50
54
28
30
21
31
45
String Deletion& Modifi.
60
90
51
String Synthesis
2
0
2
20
11
7
5
16
12
29
Reversi
46
29
28
36
15
49
60
20
23
45
23
52
Standard Sudoku
0
0
98
100
94
12
14
100
100
100
Unseen Tasks
Letters
61
52
49
12
91
11
100
93
84
87
89
89
96
Eight Queen
79
73
40
44
Number Multiply
43
46
28
11
87
100
100
68
65
100
75
95
Cryptanalysis String Splitting Combinatorial Calcul.
60
21
49
20
15
24
20
13
16
20
27
0
24
96
91
90
28
52
25
48
47
37
35
48
43
56
57
98
35
45
60
55
70
67
80
Synthesis Decompo.
57
53
44
69
2048
52
0
37
44
43
40
28
37
25
20
39
49
56
Permut. and Combina.
100
100
100
66
89
48
64
60
40
46
80
75
93
5

---

CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
the question with CoT and then output the code answer (All
our method can still outperform all main baselines obviously, even more on unseen tasks.
Code + C) 5) Concatenating the input question with AutoGen's 's original system prompt in Appendix Section N (AutoGen Conca.): 6) Implement multi-agent framework that first queries L.MMs to answer the question with All Text + CoT and All Code + CoT methods, respectively. Then the final solution is obtained by combining and summarizing both versions of the answers by the same LLM but prompted differently (Code + Text + Sum.1).
Apart from the task performance, in later sections we also discuss the costs of token lengths and runtime for each method.
5.1. Overall Better Performance
Score Distribution 01 and GPT-4a CodeSteer
Training-bar based Baselines 7) Fine-tune Llama-3.1 as a summarizer based on the Code + Text + Sum. I method us- ing SFT on correct summary data (Code + Text + Sum.2); ;): 8) We fine-tune Llama-3.1 3--8B- as a one-step evaluator to choose between text or code generation (Code/Tex) Choice); 9) OpenAI GPT Code Interpreter with the original input ques- tion (Code Interpreter). Method 7 and 8 are fine-tuned on the same data number and task types as CodeSteer.
01
20
GPT-40 + CodeSteer
Frequency
10
Comparison with CoT LLMs We also compare with the current best models: OpenAl ol and o op-previww (Jaech et al., 2024) and DeepSeek RI (Guo et al., 2025). These models enhance reasoning and planning by using textual search, reflection, and exploration during answer generation. However, our analysis shows that these CoT LLMs have not yet integrated code-based symbolic computing to further improve their performance.
0.0
0.125
0.25
0.5
0.625
0.75
0.875
Scores
Normalized score distribution of CodeSteer+GPT- 4o and ol in 37 SymBench tasks.
Figure 3:
Table I presents the full results of all methods on SymBench, including individual task scores and the Average Normal- ized Score. The key findings are:
Evaluations Answers are evaluated using predefined rules, with GPT-4o assisting in adjusting formats as needed. Beyond the Code Interpreter method, some approaches have the LLM output code as the final answer. We extract and execute this code using predefined algorithms to obtain the final result or facilitate further reasoning. To prevent infinite loops, code execution is limited to 30 seconds. If this limit is exceeded, the task is marked as failed or returns errors for subsequent turns. We utilize success rate as the metric for each task. To compare each method, we calculate the Average Normalized Score over all the tested tasks by the following equation:
1) CodeSteer maintains similar relative performance on seen and unseen tasks, indicating no overfitting.
2) Augmenting GPT-40 with CodeSteer significantly boosts its performance, raising the Ave. Norm. Total Score from 53.3 to 86.4-outprformiig all 9 baselines (best base- line: Code/Text Choice at 77.9).
3) GPT-4o + CodeSteer surpasses ol (82.7), R1 (76.8), and ol op-evviww (74.8), highlighting the importance of in- tegrating symbolic computing into LLMs. Figure com- pares the score distribution of GPT-4o CodeSteer and ol, showing that CodeSteer reduces instances of extremely low scores (near 0), demonstrating its robustness to varied tasks. 4) Compared to other training- based methods (Code + Text + Sum.2 and Code/Text Choice) with the same data number and tasks, CodeSteer's 's better performance validates the framework's 's effectiveness (further discussed in See. 6).
N
Sij AveNorm, N max(s)
where AveNorm, is the Average Normalized Score for method j. Sij is the score of method j for task i, ms, is the maximum score for task i, NN is the total number of tasks. This equation normalizes each score relative to the maximum score in the respective task, and then averages the normalized scores over all tasks. We use the normalized metric to better compare the relative performance among methods and prevent any single task from disproportionately influencing the overall evaluation. As shown in Appendix Sec I, to ensure the robustness of our conclusions against changes in the evaluation metric, we recalculate the average score without normalization. From the result in Appendix Table 7, we can observe that for both seen and unseen tasks,
5.2. Scalability and Generalizability
To assess the impact of symbolic computing, Fig. 4 tracks the performance of five methods across four tasks of increas- ing complexity. As critical task-specific properties escalate, ol, ol op-preve and GPT-4o fail in highly complex cases, while symbolic-augmented methods (CodeSteer, Code In- terpreter) sustain performance. Notably, CodeSteer proves more robust across tasks than Code Interpreter.
In our study, LodeSteerLLL is fine-tuned on synthesized datasets where T is always GPT-4o. To assess its
6

---

CodeSteer: Symbolic-Augmeenter Language Models via Code/Text Guidance
Table 2: Experimental results of (eaude3---soonet-2000 Mistral-Large, and GPT-3.5 with or without augmented CodeSteer. Methods with the higher scores of the same model are highlighted blue.
Methods
Claude
Claude CodeSteer
Mistral
Mistral + CodeSteer
GPT-3.5
GPT-3.5 CodeSteer
Combinatorial Caleu.
48
66
25
34
29
Eight Queen
87
60
41
16
Reversi
0
45
0
33
32
Cons. Linear Arran.
73
90
47
48
25
9
Standard Sudoku
0
100
0
100
95
Ave. Norm. Score
29.1
92.0
31.0
59.8
42.3
(a)
Combinatorial Calculation
(b)
Number Muttiplication
[8]90 90
(b)e0
CookStoer
Success Rate (%)
Cocerine: Chacee
01
Symaaa Agann)
Sympak Ol-pprdde
60
60
01-proxiow
Score
Score
Success
GPT-to
40
40
Codo
(Cad
20
20
80
80
AB Not CaT
AB Bed
(3,10)
(5.78)
(6,100) 3001 (20,500)
(4.4)
(8.8)
(52,13) (26.19)
(20.20)
1500
3000 6000
(Num, numbers, Target value)
(Digit_num, Digit_num)
Token Length
Runtime
(c)
Game 24
(d)
Lotters
Figure 5: Score vs. token and runtime costs for each method, highlighting CodeSteer, RI, ol, and ol-preview in red. We display CodeSteer results separately for inferences using single or four H100 GPUs. Specific values are in Table 8.
Success Rate (%)
60
Success
40
20
20
0:
Num.5
Num
Num.
Num,
Num,
5
(20.30)
(s0
(150.200)
(350.700)
(150 (00)
(Str min Str max, lan)
5.3. Cost of Tokens and Runtime
Figure 4: Method performance across four representative tasks as task complexity increases from left to right on the x-axis controlled by value scales. C.S. and Inter. represent CodeSteer and Interpreter.
Figure 5 shows Score versus Token Length (including input and output tokens) and Score versus Runtime (covering both LLM inference and code exccution) for all methods. Complete data provided in Appendix Table 8. Token counts include only those used by TaskLLM,, excluding small and open-source models fine-tuned on Llama-3. 1-BB. For the ol and ol -pevveww models, only runtime is plotted since their thinking chains are unavailable. While achieving superior performance, CodeSteer uses more tokens than baseline methods due to its multi-turn generations. Most of these tokens are consumed by multiple interaction turns that ultimately fail. CoT LLM R1 consumes more tokens than CodeSteer due to the inefficient textual iteration.
transferability and generalizability, we test it with three pop- ular models: Claude-3-5-Sonnee Mistral-Large, and GPT- 3.5-Turbo. We evaluate them on five representative tasks based on GPT-4o's 's results in Table 1: two where text out- performs code and three where code is superior. CodeSteer has shown apparent effects when guiding GPT-4o on these tasks. The results in Table 2 confirm that CodeSteer gener- alizes well across other LLMs types. This is expected, as its core mechanisms-cooeer tion-are essential to all ceneral-purposs LLMs. Notably, observe that CodeSteer is particularly effective when applied to stronger LLMss, such as Claude. This is likely be- cause more powerful models possess superior self-reflection capabilities and can generate complex code with greater pre- cision. Thus, they benefit more from CodeSteer's 's additional structured guidance, unlocking their full potential.
In terms of runtime, CodeSteer is faster than ol and RI while delivering better performance. Additionally, since most CodeSteer's runtime comes from the inference of the 8B CodeSteerLLM on our workstation, hardware and system optimizations can significantly reduce it. For exam- ple, running LodeSteerLLM on four H100 GPUs instead of one decreases the average runtime from 63.8 to 45.4 seconds. CoT LLMS consume excessive runtime and to- kens due to their extensive and often redundant reasoning chains. Textual iteration is inherently inefficient for search. Appendix Sec. shows examples of text answers of RI and GPT-4o, which both models attempt to find the cor-
guidance and dynamic adapta-
7

---

CodeSteer: Symbolic-Augmenter Language Models via Code/Text Guidance
rect equation for the Game 24 task by listing all possible combinations, leading to uncontrolled iterations and end- less generation. This highlights the importance of symbolic computing through code generation.
Data Augment., which means this extra attention on the last two turns does enhance the SFT process.
Symbolic and Self-answer Checkers We evaluate the effects of the Symbolic and Self-answer Checker in two parts: 1) Dataset Synthesis Efficiency: Comparing Group 6 with Groups 7 and in Table shows that integrating these two checkers increases the Symbolic Agent's 's success rates, thereby enhancing the efficiency of the dataset synthesis process. 2) CodeSteer Performance: Comparing Group with Groups 4 and 5 demonstrates that augmenting with these two checkers improves CodeSteer's final performance.
5.4. ol + CodeSteer
Here we test whether CodeSteer can improve the perfor- mance of reasoning models like ol. As shown in Table 3, the performance of ol will improve notably when augmented with CodeSteer on 5 randomly chosen unseen tasks, further verifying the effectiveness of CodeSteer.
Multi-turn Guidance CodeSteer uses a multi-turn inter- action strategy with TaskLLM. In contrast, the Code/Text Choice method in Table relies on single-step guidance and performs worse than CodeSteer. This demonstrates that the multi-ture turn design enhances guidance effectiveness, align- ing with the common intuition that the best methods for many tasks emerge from iterative "executing and exploring" processes accompanied with dynamic adaptation.
5.5. Comparison with Heuristic-based Methods
To better evaluate CodeSteer and provide deeper compar- isons, we include three prompt-based baselines. Few-Shot: Uses five example-base prompts to guide the TaskLLM in mimicking the 'code/text' switching reasoning pro- cess. Code-FFrst-RRun A rule-based approach where the TaskLLM is prompted to use code for the first three rounds (with increasing complexity) and then switch to text-based reasoning. Gode-First-Agent Employs GPT-4o as the LodeSteerLLM to guide the TaskLLM using the same code- first-then- strategy as in Gode-First-Rule As shown in Appendix Table 9, the three prompt-based methods per- form significantly worse than CodeSteer, underscoring the effectiveness of training with our synthesized data. Upon analyzing failure cases, we identify two main reasons:
Guide Not Summarizer CodeSteer primarily serves as the guidance generator for TaskLLM rather than directly generating answers, summarizing, or selecting among multi- ple answers. This design choice accounts for the limitations of the open-source LLM we use compared to the more capa- ble closed-source LLM that supports TaskLLM. By focusing on guidance, CodeSteer reduces task complexity and data space requirements. The Code + Text + Sum.2 approach in Table attempts fine-tune tunettneeeee an answer summarizer using the same data volume but fails, highlighting that summariza- tion imposes a significant burden on Llama-3, 1-BB due to the unique characteristics of each task.
1)
Lode guidance often includes problem- specific coding knowledge (e.g., suggesting A* or DFS) and how to formalize the problem, which purely prompt- based methods struggle to capture.
2) Switching between code and text can be advantageous, as later code generations can build on insights from prior tex- tual reasoning. For instance, in Path Plan, a text-generated trajectory may be partially correct; subsequent code can refine directly, reducing the search space.
7. Related Work
Code Generation and Symbolic Computing in LLM Tasks LLMs are widely used for general agent tasks, such interacting with softwares and websites (Zhou et al., 2023c; Hao et al., 2024a;b; Xu et al., 2024), planning robot actions (Chen et al., 2024d; Ahn et al., 2022), and inferring with logic (Suzgun et al., 2022). Literally, many test tasks in previous works can be solved with direct coding (Suzgun & Kalai, 2024; Gao et al., 2023). Some recent works also further extend the applications of coding into tasks involv- ing commonsense reasoning and semantic analysis (Li et al., 2023; Weir et al., 2024). Most of previous works mainly uti- lize text (Yao et al., 2024; Ahn et al., 2022; Lin et al., 2023) or code (Liang et al., 2022: Bairi et al., 2024; Zhou et al., 2023a) as the only output modality. Chen et al. (2025) high- lights the importance of smartly switching between code and text generation in LLMs but notes current methods have clear drawbacks.
6. Ablation Studies
The CodeSteer framework comprises SFT and DPO dataset synthesis, CodeSteerLLM fine-tuning, a symbolic checker, and a self-answer checker. Here we do the ablation studies on these components and their related modifications. The added experimental results are shown in Table with the whole result table of 37 SymBench tasks in Append Sec. M.
DPO Effects In Table 4, 1. I.CodeSteer outperforms 2. WO DPO, showing the effectiveness of the DPO process.
SFT Data Augmentation As discussed in Sec. 4.1, we do the data augmentation of the last two turns in each trajec- tory to prevent multi-tura gradient cancellation. In Table 4, 2. WO DPO achieves higher score than 3. WO DPO WO
LLM Self-reflection and CoT Models LLM-generated
8

---

CodeSteer: Symbolic-Augmenter Language Models via Code/Text Guidance
Table 3: Per-task success rate (%) for the text-only baseline (ol) versus ol augmented with CodeSteer. For each task, the higher score is highlighted in blue.
Task success rate %
Cryptanalysis
Synthesis Decomposition
2048
Eight Queens
Combinatorial Calculation
ol ol CodeSteer
60
57
52
84
57
73
94
72
97
95
Table 4: Ablation studies on CodeSteer. WO DPO: CodeSteer with SFT but without DPO fine-tuning, WO DPO WO Data Augment: Same as WO DPO, but without data augmentation in the last two turns. Agent represents the Symbolic Agent.
Methods
1.Code Steer
2.WO DPO
3. WO DPO
WO Symbolic Checker
5.WO
6.
7.Agent WO Symbolic Checker
8.Agent WO Self-answer Checker
WO Data
Self-answer Checker
Agent
Task success rate %
Augment.
Ave. Norm., Seen
88.1
80.0
79.7
80.1
78.5
77.0
71.9
70.1
Ave.
Norm., Unseen
81.3
76.2
70.9
68.6
64.2
67.9
62.0
57.4
Ave. Norm., Total
86.4
79.1
77.6
77.3
75.0
74.8
69.5
67.0
feedback via self-evaluation can improve performance
writing preserves generalization across tasks.
variety of tasks (Yang et al., 2022; Welleck et al., 2022:
We note that CodeSteer encounters failures under the following conditions, insisting the further research to overcome these bottlenecks.
Limitations
Madaan et al., 2023). The OpenAI ol (Jaech et al., 2024) and DeepSeek RI (Guo et al., 2025) models demonstrate the potential of agentic LLMs that use Chain-of-Thought (CoT) text generation to explore and self-reflect, enhancing reasoning and planning. However, they lack symbolic com- puting and code generation capabilities, leading to weaker performance on complex symbolic tasks and consuming substantial tokens and time (Chen et al., 2024a; Ai et al., 2025).
1) Insufficient Capability of TaskLLM: In some cases, the capabilities of the Thetherr through coding textual reasoning- not sufficient solve the given problem.
2) Suboptimal Code Generation: The generated code may not use the most efficient method, which can lead to time- outs. For example, as shown in Fig. 4c, CodeSteer's success rate decreases when the target values increase, due the exponential growth in search complexity.
LLM Fine-tuning with Multi-step SFT and DPO SFT (Chen et al., 2024e) and DPO (Rafailov et al., 2024) are extensively implemented for LLM fine-tuning. To en- hance LLM's capability in multi-step agent tasks, these methods are further modified with multi-step goals and re- wards (Zhou et al., 2024b; Zhai et al., 2024; Zhang et al., 2024). LLM self-generated data have become increas- ingly important for model improvement when combined with search algorithms and rejection sampling (Zhou al., 2023b; Guan et al., 2025).
3) Lack of Robustness to Task Complexity: CodeSteer is not yet robust enough across tasks with varying complex- ity. As shown in Fig. 4a, performance drops in medium- complexity samples. In these cases, CodeSteer sometimes selects textual reasoning over coding and ends up producing incorrect answers.
8. Discussion
A Acknowledgments
Our work underlines the significance of augmenting LLM reasoning and planning capabilities with symbolic comput- ing and shows great potentials of steering large models for smarter code/text generation with specialized small models. We introduce novel modifications dataset synthesis and fine-tuning (SFT/DPO) to support multi-turn guidance framework, which has proven effective. Unlike CoT LLMs like OpenAl ol and DeepSeek RI, which rely solely on tex- tual reasoning for exploration, symbolic computing offers greater efficiency, robustness, and scalability. Since coding a core L.MM capability, generating symbolic tools via code
This work was supported by ONR under Award N00014-22- 1 1-2478 and MIT-IBM Watson AI Lab. However, this article solely reflects the opinions and conclusions of its authors.
Impact Statement
This paper aims to advance the field of Foundation Models. Steering the generation from language models has the great potential to improve safety and performance to better align with human preferences. Any such work is inherently a

---

CodeSteer: Symbolic-Augmentted Language Models via Code/Text Guidance
double-edged sword; the same techniques used to generate samples from harmless distribution of text could, with single sign change, be repurposed for generating samples from harmful distribution of text. Our method improves language model capability by integrating symbolic comput- ing, which may also be misused for harmful purposes.
Chen, Y., Arkin, J., Hao, Y., Zhang, Y., Roy, N., and Fan, C. PRompt optimization in multi-step tasks (PROMST): In- tegrating human feedback and heuristic-based sampling, In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3859-3920, Miami, Florida, USA, November 2024c. Association for Computational Linguisties. doi: (0.18653/v1/2022 ennlp-main.22 URL https: / /aclanttooonoyy..... www.::44...mmmm-mm......2.....
Overall, we believe the potential positive social benefits of our work in evaluation and steering language model output towards desired target distributions outweigh the potential negatives stemming primarily from misuse.
Chen, Y., Arkin, J., Zhang, Y., Roy, N., and Fan, C. Scalable multi-robot collaboration with large language models: Centralized or decentralized systems? In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311-4317. IEEE, 2024d.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Chen, Y., Jhamtani, H., Sharma, S., Fan, C., and Wang, C. Steering large language models between code execution and textual reasoning, 2025. URL https://arxiv org/ / abs/ / 241 0 03524.
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. Do as i can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv:2401.01335, 2024c.
Ai, M., Wei, T., Chen, Y., Zeng. Z.,, Zhao, R., Varatkar, G., Rouhani, B. D., Tang, X., Tong, H., and He, Resmoe: Space-efficient compression of mixture of ex- perts Ilms via residual restoration. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining V.I, KDD '25 pp. 1-12. ACM, July 2025. doi: 10. 10.1445/3390624337799966 URL http: 3709196.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 100 14168, 2021.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
//dx. wow.doi..ooo www.dooi..oooo... 10.
Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., and Shet, S. Codeplan: Repository-level coding using lims and planning. Pro- ceedings of the ACM on Software Engineering. ((FSE): 675-698, 2024.
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided lan- guage models. In International Conference on Machine Learning, pp. 10764 10799. PMLR, 2023.
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F,, and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.0455 19, 2025.
Chen, W. Ma. X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211. 125588 2022.
Gui, J., Liu, Y., Cheng, J., Gu, X., Liu, X., Wang, H., Dong, Y., Tang, J., and Huang, M. Logicgame: Benchmarking rule-based reasoning abilities of large language models. arXiv preprint arXiv:2408. 15778, 2024.
Chen, X., Xu, J., Liang, T., He, Z., Pang, J., Yu, D., Song, L., Liu, Q. Zhou, M., Zhang, Z., et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. arXiv preprint arXiv:2412.21187, 2024a.
Guo, D., Yang, D., Zhang. H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepscek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint ar arXiv:2501. /2948, 2025.
Chen, Y., Arkin, J., Dawson, C., Zhang, Y., Roy, N., and Fan, C. Autotamp: Autoregressive task and motion planning with llms as translators and checkers. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6702. IEEE, 2024b.
Hao, Y., Chen, Y., Zhang, Y., and Fan, C. Large language models can plan your travels rigorously with formal veri- fication tools. arXiv preprint arxvv22404. /891, 2024a.
10

---

CodeSteer: Symbolic-Augmentte Language Models via Code/Text Guidance
Hao, Y., Zhang, Y., and Fan, C. Planning anything with rigor: General-purposs zero-shot planning with lim-based for- malized programming, arXiv preprint urXiv:2410.121111 2024b.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery: A., Le, Q- V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain- of- ofothhtttt can solve them. arXiv preprint arXiv:2210.0926/ 2022.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambham- pati, S. Large language models still can't 't plan (a bench- mark for lims on planning and reasoning about change), In NewrlPS 2022 Foundation Models for Decision Mak- ing Workshop, 2022.
2021.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., Ei-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Valmeekam, K., Marquez. M., Olmo, A.. Sreedharan, S., and Kambhampati, S. Planbench: An extensible bench- mark for evaluating large language models on planning and reasoning about change. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Langley, P. Crafting papers on machine learning. Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, 2000. Morgan Kaufmann.
Wang. J., Wang, J., Athiwaratkun, B., Zhang, C.,, and Zou, J. Mixture-of-agente enhances large language model capabilities. arXiv preprint 14XXv::2406.04692 2024.
Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., and Ichter, Chain of code: Reasoning with a language mudel-ang code emulator. arXiv preprint arXiv:2312.044:: 2023.
Weir, N., Khalifa, M., Qiu, L., Weller, O., and Clark, P. Learning to reason via program generation, emulation, and search. arXiv preprint ar arXiv:2405. /6337, 2024.
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
Welleck, S., Lu, X., West, P., Brahman, F., Shen, T., Khashabi, D., and Choi, Y. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2022.
Lin, K., Agia, C., Migimatsu, T., Pavone, M., and Bohg, Text2motion: From natural language instructions to feasible plans. Autonomous Robots, 1745511345 2023.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, I.., Zhang, X., and Wang, C. Autogen: Enabling next-gen Ilm applications via multi- agget conversation framework. arXiv preprint 230i::2300.08155.5 2023.
Madaan, A., Zhou, S., Alon, U., Yang, Y., and Neubig, G. Language models of code are few-shot commonsense learners. arXiv preprint arXiv: 20.020000222222 128, 2022.
Xu, T., Chen, L., Wu, D.-J., Chen, Y., Zhang, Z., Yao, X., Xie, Z.,, Chen, Y., Liu, S., Qian, B., et al. Crab: Cross- environment agent benchmark for multimodal language model agents. arXiv preprint 2XXx::400.0151/1 2024.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang, Y., al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.176611 2023.
Yang, K., Tian, Y., Peng, N., and Klein, D. Re3: Generating longer stories with recursive reprompting and revision. In Proceedings of the 2022 Conference on Empèrical Meth- ods in Natural Language Processing, pp. 4393-4479, 2022.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Yang, Y., Xiong, S., Payani, A., Shareghi, E., and Fekri, F. Can lims reason in the wild with programs? arXiv preprint arXiv:2406. 23664444066 2024.
Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D.
Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-995 9471, 2022.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. 2024.
Suzgun, M. and Kalai, A. T. Meta Meta-prompping: Enhancing language models with aask-agnostic scaffolding. arXiv preprint XrXiv:2401. 12954, 2024.
Zhai, Y., Bai, H., Lin, Z., Pan, J.. Tong. S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. Fine-tuning large

---

CodeSteer: Symbolic-Augmentted Language Models via Code/Text Guidance
vision-language models decision-making agents via reinforcement learning. arXiv preprint arX1v:2405.10292 2024.
Zhang, X., Du, C., Pang, T., Liu, Q. Gao, W., and Lin, M. Chain of preference optimization: Improv- ing chain-of-thought reasoning in llms. arXiv preprint arXiv:2406. 09666, 2024.
Zhou, A., Wang, K., Lu, Z.,, Shi, W., Luo, S., Qin, Z Lu, S., Jia, A., Song, L.. Zhan, M., et al. Solving challenging math word problems using gpt-4 code in- terpreter with code-based self verificciii.tiion. arXiv preprint arXiv:2308. 07921, 2023a.
Zhou, A., Yan, K., Shlapentokh- Shlapentokh-Rothman Rothmmn, M., Wang, H., and Wang, Y--.. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint 2rXiv:2310.0444/ 2023b.
Zhou, L., Schellaert, W., Peartinez--lumedd. F., Moros-Daval. Y., Ferri, C., and Hernández-Orallo, J. Larger and more instructable language models become less reliable. Na- lure, pp. 1-8, 2024a.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint urXiv:2307. 13854, 2023c.
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. / /9446, 2024b.
12

---

CodeSteer: Symbolic-Augmennted Language Models via Code/Text Guidance
Appendices: CodeSteer: Symbolic-Aummented Language Models via Code/Text Guidance
A. Impacts of task types, task complexities, and LLM capabilities on code/text choices
14
B. Varied code versions of the same LLM
16
C. Description of SymBench tasks
17
D. Prompt for CodeSteerLLM
21
Prompt for Self-answer Checker
21
F. Code for Symbolic Checker
22
G. Synthesized dataset number of each task for SFT and DPO
23
H. Parameter and hardware settings of SFT/DPO fine-tuning and inference processes
23
Score comparison different methods without normalization across tasks
24
J. Score-cost table for each method
24
K. Comparison with heuristic-based methods
24
L. Example text answer of DeepSeek RI and GPT-4o in Game
25
M. Full experimental results of ablation studies
27
N. System prompt of AutoGen
288
13

---

CodeSteer: Symbolic-Augmenttr Language Models via Code/Text Guidance
A. Impacts of task types, task complexities, and LLM capabilities on code/text choices
The phenomenon and challenges of steering LLM code/text generation are first proposed by Chen et al. (2025). Here we discuss these phenomenon in details for the motivation of our work. Fig 6 presents two typical examples of the recently popular topics of "9.11" and "9.9" numerical comparison and 'r' letter count in 'strawberry' that the ChatGPT of GPT-4o makes mistakes by direct textual reasoning but easily solves the problem after prompted to use code. Meanwhile, Fig 7 displays the example that GPT-4o makes mistakes to solve the question by code generation but partially solve the question by textual reasoning. The above two examples show that whether code or text is simpler highly depends on the task types and LLM own capabilities and characteristics.
The OpenAl GPT-4o Code Interpreter trained to steer LLM code/text generation. However, the study of Chen al. (2025) finds many limitations of this method. In Fig 8, they observe an intriguing property GPT Code Interpreter: its decision use code depends on the complexity of the task, as shown in Fig 8. GPT-4o Code Interpreter chooses handle simple Number Multiplying questions with text and complex questions with code, resulting in correct answers. However, it fails in medium ceddium-difficlly questions since it tends to be overconfident and chooses answer the question via textual reasoning, which sometimes is wrong. Hence, whether to implement symbolic computing depends on task complexities even for the same type of the task.
(a)
X
9.11 9.9, bioger?
(b)
X
Hov many dhowberyy pooiioos?
The word "srawberry contains ner Thai pootions are
9.11 bigger
abiictar
Use code the following question
9.11 and 9.,, which bigger?
Instoweem poollous
To determine which number is bigger, can compare the two numbers directly in Pythors
postions:
cop coo
Furtoc
e Capy Goide
Eucbund courc
numbors
t.aositions
letter ewweratteeened) Letter
Ruml 9-11
Compane
Outpel:
bigger Number
Hulber
Postndons
Let's executt this code find out.
Expleration
three "atawberr"
8.9 is bigger than 8.11.
Thai
indeding)
Figure 6: The cases that GPT-4o makes simple mistakes by direct textual reasoning but can reliably solve the problem with prompted to use code.
14

---

CodeSteer: Symbolic-Augmenteed Language Models via Code/Text Guidance
BoxLift
Corrial-Correct answer from
Wrong answer from All code + CoT
All text CoT
To solve this probles, we need to assign Lifters to baxes in such a say that we edrimtze the namer of stess pready
(315,
411
3oop 2: [[083:
8. 211 (84). 1ke 3111
2. Sort the Lifters descerding arder of their casacittes.
Step 8: [1266,
211 (386. B. 1331
ste0, single Lifter Lift a canture mulliple Lifters Lift Als
heavlest
ssing
Litters-
d):
[i),,
5:09
[[33)
[43). 1224, [3ll, (323, D4S3
untal bures are Lorted. napeat this
5:4p
[1)). 188h, [211, (93, DD1)
[(87,
(2)),
(42, (413)
teaes 182, 251, 196, 218, 247, 20, 253, 256, 280, 136, 145, 323, 113, 186, 58, 59, 132, 48, 87, 2361
tooms. www.errtttrroraiilo TraTr Liffers.sartireee
veite
stea
lifterstil
bes baccellA) [A]
roral
3. Locparsty Lifter box:
liftersie
usse_ ap
Lifter
tetall capacity boxt
if total coppocityyy boxt
Latteral)
for ondex In sortaclueed 1ithers, reversentraal:
boses, deptll
stops apperdiatsp!
pranth printt enStep [step_nan): (step)")
Achieve score of 0.5!
Run into infinite loop!
print)""so"")
Figure 7: Representative answers of BoxLift task. The left figure is the partially correct answer of GPT-4o with All Text + CoT method. The right figure is the wrong code answer from All Code + CoT method. The text and code parts are colored in blue and green, respectively. The All Code + CoT method generates the wrong code that runs into an infinite loop.
Simple question Text as answer
what the result 12*56?
The result of 12 x 56 672.
Hard question Code as answer
what is the result of 124954436--60772266222
124354536 5607425632 697,308, 812, 2211,66,, 752,
Medium question Text as answer
what is the result of 1243
6969501
result 1243 5607 6,969,
X
Figure 8: GPT-4o Code Interpreter tends to handle simple Number Multiplying tasks with text and complex tasks with code. However, it often fails with medium-difficlly questions, where it is overconfident and chooses not code when needed.
15

---

CodeSteer: Symbolic-Augmenttd Language Models via Code/Text Guidance
B. Varied code versions of the same LLM
Game 24
Correct code answer of GPT-4o
Wrong code answer of GPT-4o
To solve the problen womitted text>-
To solve the problen of using the numbers 3, 5, 8, and to obtain 24, we need to find conbination of arithretic operations that will result in 24, Each number nust be used exactly once, Let's break down the approach:
Let's write a Python script to perfore these steps.
python
fitenare: find 24.py
fron itertools import pormutations, product
Identify possible operations: We can use addition, subtraction, sultiplicatin licationn,,, and division.
def find_24 24(numbersl
ops ['+", '=', 'e', 'y']
for nun, pem in perutatiions(uuterrs))
Try different conbinations: Me need to experinent with different groupings and operations to which combination ylelds 24.
Tor 0p. in product(ogs, repeat-31:
# Generate all possible
to parenthesize the expression
expressions
f"((mun_per (op_per((l)) (nun_pera(21)) (op_pem(11))
Ensure all numbers used: The solution rust incorporate all numbers exactly
((uun_perm(2)>>
(op_perm[2]) (oun_pern(3)))'.
""'[(nun_pernn (op_pempppmm__pp (0)) (nun_perm(1))) (op_pem(1])
(nun_pert) perp (21))
[op_pem[2]] (nun_pern[3))",
various conbinations, find that following equation
["[[mu_pper (op_perm(0) ((mun_pern(11) (op_pern(t))
(op_pem[2]) (nun_perm(3))),
("(munpperr()) {op_perm(())
[[{mun_per((1)> {op_per([1))
(3x8)-113-5)=24
(nun_perm
(op_perm(2) [2]) (nun_perm(3)))"'
[=(mun_per(ee] (op_pern(o]) ((nun_porx(1))> {op_per[[1)>
Nov, let's the Python code compute display this result:
((nun_porm(2]) (op_pem(2]) (nun_pere(3)))))
for expr in expressions: try: if absteval(expr] - 24) < 1e-6:
python
Define the numbers
Check if the result is 24
=
return PPee<eeppr = 24099"
b
except ZoroBivisionError: continue
8
13
return "No solution found."
Calculate the result using the identifled equation result = (a 4 c) (a b)
numbers [3, 13] print(find__a) (4(umbe))))
Print the result in the required fornat print("eac", result, "sas")
.onitted texts.
Figure 9: Representative code answers Game 24 task. The left figure is the correct code of GPT-4o with extra AutoGen prompt in Appendix Sec. N for guiding code/text choices. The right figure is the wrong code after prompting GPT-4o to answer with code "Think of an algorithm to solve the task and implement it in python'. The text and code parts are colored in blue and green, respectively. In both cases, GPT-4o is prompted to solve this task with code. The only difference is the guiding prompts. However, GPT-4o answers with different types of codes, with or without efficient symbolic computing. This phenomenon shows that LLM code generation is unstable under varied prompts, tasks, and LLM types.
16

---

CodeSteer: Symbolic-Augmentted
Language Models via Code/Text Guidance
C. Description of SymBench tasks
Here we describe the 37 testing tasks. They require strong symbolic, mathematical, logical, geometrical, scientific, and commonsense reasoning capabilities. The first 14 tasks originate from Chen et al. (2025), while the last 23 are from Gui al. (2024). Note that both these two previous works do not release the full question datasets and codes for these 37 tasks. The released question dataset in Gui et al. (2024) only contains 8 or 16 questions for each task. Hence, we develop codes to automatically synthesize the questions for each task with tunable complexities. Both our developed codes and question datasets are released.
Number Multiplying This task involves querying LLMs to compute the product among integers. It represents a classic problem that LLMs are not able to solve through pure textual reasoning.
Game 24 This task involves querying LLMS use a given set of integers to generate an equation that evaluates to 24. This task is tested in previous work Sfee-of-Thougeee (Yao et al., 2024).
This task involves querying LLMs to plan the robot trajectory waypoints based on human task instructions and environments. This task originates from AAMP (Chen et al., 2024b).
Path Plan
Letters This task involves querying LLMs to count the total number of specific letters in a long word and specify their positions. An example question can be 'How many r's in the word strawberry and what are their positions?". This task has recently gained significant attention because current LLMs struggle to perform it effectively and accurately.
BoxLift This task involves coordinating robots of various types to lift boxes of different sizes and weights. Each robot has specific lifting capacity and can collaborate with others to lift single box. A box can only be lifted if the combined lifting capacity of the robots exceeds the box's 's weight. The objective is to lift all the boxes in the minimum number of time steps. This task originates from Scalable-Robots (Chen et al., 2024d).
This task involves coordinating robot arms to move colored boxes (squares) into corresponding colored goal locations (circles) in the fewest time steps. Each robot arm is assigned and restricted to a cell indicated by the dotted lines. The arms have two possible actions: (1) move box within their cell to neighboring cell, or (2) move box within their cell to a goal location within the same cell. The objective is to ensure all boxes are placed in their matching goal locations efficiently. This task originates from Scalable-Robots (Chen et al., 2024d).
BoxNet
Blocksworld In Blocksworid, the objective is to stack a set of blocks (brown) according to a specific order. The robot can perform four actions: (1) pick up a block, (2) unstack a block from the top of another block, (3) put down a block, (4) stack a block on top of another block. A robot can only pick up, unstack, or stack a block it is clear, that is, the block has no other blocks on top and not currently being held. This task originates from PlanBench (Valmeckam et al., 2024).
Date Understanding Given small set of sentences referring specific date, the task involves querying LLMs to answer a provided question based on the information in these sentences (e.g., 'The concert was scheduled for 06/01/1943, but was delayed by one day to today. What was the date yesterday in MM/DD/YYYY?"" This task originates from BIG-Bench-Hard (Suzgun et al., 2022).
Web of Lies This task involves querying LLMs to determine the truth value of a random Boolean function presented as a natural-language word problem. This task originates from BIG-Bench-Haid (Suzgun et al., 2022).
Logical Deduction This task involves querying LLMs to deduce the order of a sequence of objects using clues and information about their spacial relationships and placements. This task originates from HIG-Bench-Han (Suzgun et al., 2022).
Navigate This task involves querying LLMs to determine whether the agent would return to its initial starting point after following a series of navigation steps. This task originates from BIG-Bench-Hart (Suzgun et al., 2022).
GSM-Hard (Gao et al., 2023) This is the more challenging version of GSM8K (Cobbe et al., 2021) math reasoning dataset, where the numbers in the original questions of GSM8K are replaced with larger, less common values.
MATH-Geometry This is the math reasoning dataset from MATH dataset (Hendrycks et al., 2021), with specific focus on geometry questions.
MATH-Count&Proon Probabiity This is the math reasoning dataset from MATH dataset (Hendrycks et al., 2021), with specific focus on counting and probability questions.
17

---

CodeSteer: Symbolic-Augmeented Language Models via Code/Text Guidance
The following 23 tasks originate from LogieGame (Gui et al., 2024).
Logical Equation The task is to assign a specific numeric value to each letter from given set, using predefined range of numbers and a set of inequalities. Each letter corresponds to unique number, and the relationships between the letters are defined by mathematical equations or constraints.
New Operator This task introduces custom mathematical operations involving two numbers, defined with unique formulas. The goal is to use the given definitions of these operations to compute the result of a specific expression.
Pooling This task involves applying a pooling operation on a numerical N x N grid. The pooling operation uses an n X n sliding window (n < N) that moves across the grid from left right and top to bottom. The results from each window are then arranged based on their positions to create new output matrix.
Light Puzzles In this task, you are given an n x n grid representing a network of lights, where a lit light is represented by "1" and an unlit light by "0". Several buttons control the state of these lights by turning them on or off in certain positions. The state of each light can be affected by multiple buttons. The task is to follow a series of button presses and determine the final state of the grid.
Mahjong Given an initial set of letter cards, in each turn, new card is added and one card is removed. Some effects may happen when specific combinations of the cards appear after introducing the new card. A result is determined based on these specific conditions. The goal is to determine a result based on series of turns
Statistical Counting Calculate the total score of a string by scanning it from left to right, where consecutive identical letters carn points (for example, two or more consecutive A's add point, B's add 2 points, etc.). The task is to start with a score of and return the final summing value.
Matrix Transformation Rotate a given matrix of characters based on given instruction (e.g.. 90 degrees clockwise), preserving each character's position relative to others in the transformed output. The input matrix can be of any size and contain any character.
Logical Puzzle
The task involves querying LLMs to select a specified number of different values from a grid of numbers, ensuring that certain mathematical constraints (sum or product) are satisfied for selected numbers for each row and column.
Constrained Linear Arrangement In a two-player card game, the task is to deduce your opponent's 's moves based on the game's rules, your played cards, and the announced results of each turn. Each card can only be used once, and the game follows specific interaction rules between different card types, where certain cards can defeat, be defeated by, draw with others according to predefined relationships.
The task involves querying LLMs to find all squares in a character matrix where each square consists of identical characters and has a side length of at least 3.
Pattern Recognition
String Insertion The task is to transform a string by scanning it from left to right and inserting specific characters after certain character patterns (c.g., each pattern WXYZ requires inserting W immediately after it occurs). All operations are performed simultaneously the original string.
Letter Logic Diagram The task is to complete an incomplete grid by selecting from a list of letters, where each row and column must contain each letter exactly once, and all cells on the minor diagonal (top-right to bottom-left) must contain the same letter. Some cells are already filled in as constraints.
String Deletion and Modification The task to transform a string by repeatedly applying a set of ordered string manipulation rules until no more changes are possible, where each rule modifies the string based on specific patterns or conditions present in the current string state. For example, a modification rule can be "If the string ends with "ba", replace it with "a
String Synthesis Given an initial set of blocks and a set of synthesis rules that combine different types of blocks, the task is to determine the final block(s) after repeatedly applying these rules in order until no more combinations are possible.
Reversi In this game similar to Reversi, players take turns placing pieces on an n x n grid. After placing a piece, any of the opponent's pieces located between two of the player's pieces (in the same row, column, or diagonal) will be flipped. The task to determine the state of the board after rounds, starting from given configuration.
Standard Sudoku Given partially filled Sudoku grid, the task is to fill the remaining empty cells with numbers between
18

---

CodeSteer: Symbolic-Augmentte Language Models via Code/Text Guidance
1 and 9, ensuring that number repeats in the same row, column, or 3 subgrid.
Eight Queen Given grid with some queens already placed, the task is to place the remaining queens such that no two queens share the same row, column, or diagonal, while avoiding positions with obstacles in the grid.
Cryptanalysis In this task, you are provided with combination lock consisting of numbers and letters, where neither the numbers nor the letters repeat. Using a series guesses and feedback, the goal is to deduce the correct password based on the given conditions.
String Splitting A dismantling engineer has old machines and can obtain machine parts through set of predefined methods. By continuously cycling through these methods in specific order, the engineer dismantles machines combines parts to create new components, and the task is to determine the total number of parts and remaining machines after all possible cycles.
Combinatoral Calculation Given a set of integers, the goal is to use arithmetic operations (addition, subtraction, multiplication, division) and parentheses to arrange the numbers such a way that the final result matches a specified target value. Each number must be used exactly once, and the order of the numbers cannot be changed.
Synthesis Decomposition A farmer grows various crops and can exchange them for agricultural products. Using set of methods, he can trade specific combinations of crops for products, following cyclic pattern until no further exchanges are possible. The goal is to determine the synthesis result for each round.
2048 Similarly to the 2048 game, in a grid, numbers representing powers can move in any direction, combining when they encounter a matching number to form the next power 2. Given a starting position and a sequence of movements, the goal is to determine the resulting grid after executing the moves.
Permutation and Combination Given of objects with specific positioning constraints, the task is determine the correct arrangement of the objects on shelf. Each object must be placed in position according to the rules provided, ensuring that the conditions adjacency, order, and specific positions are met. For example, rule about adjacency could be 'Book A must be adjacent to book l'.
19

---

CodeSteer: Symbolic-Augmentted Language Models via Code/Text Guidance
Table 5: The evaluated capabilities of all tasks, classified as Execution, Planning, and Reasoning tasks.
Categories
Tasks
Mathematics
Spatial Reasoning
Logical Reasoning
Order Reasoning
Optimization
Search
Number Multiplying
X
X
X
New operator
X
X
X
Pooling
/
x
X
X
Light Puzzles
X
X
Mahjong
X
X
X
X
Statistical Counting
/
X
X
J
X
Matrix Transform.
X
X
X
X
Execution
Pattern Recognition
X
X
XX
String Insertion
String Deletion &oodi.
X
String Synthesis Reversi
X
X
String Splitting
Synthesis Decomposition
2048
Game 24
Path Plan
X
Letters
X
BoxI.ift
X
BoxNet
x
Blocksworld
Logical Equation
Planning
Logic Puzzle
Const. Linear Arrange.
X
Letter Logic Diagram
*
Standard Sudoku
Eight Queen
x
Cryptanalysis
X
Combinatorial Calculation
Permutation and Combina.
X
Date Understanding
X
Web of Lies
X
Logical Deduction
X
Reasoning
Navigate
X
GSM-Hard
MATH-Geomee Geometry
MATH- Cere--ooon&&&raabiaaaie
X
20

---

CodeSteer: Symbolic-Augmentted Language Models via Code/Text Guidance
D. Prompt for LM
The input prompts of LodeSteerLLM follow multi-turn dialogue, i.e., previous turns prompts and responses will be included history prompts for following generation of response guidance. Since we the maximum turns of guidance be 5 for each task, the total addition prompt and output lengths of LM does surpass maximum context window 8k. The formats for the first turn of prompt and following turns of prompts as follows. Note that 'The summary of generated code complexity is: (code_complextty_sue summmary) is not included if the generated answer by TaskLLM does not have code.
Turn prompt to CodeSteerI.LM
You are guiding another TaskLLM to solve task. You will be presented with a task that can potentially be solved using either pure textual reasoning or coding. Your goal is to determine which method will be most effective for solving the task. Follow these steps:
**Respond** with the chosen approach but the solution. You can choose between the following options: you choose coding, explain the reasons and respond the final returned guidance with the format <<<gui danguddance dance prompt content>>> in the end of your response.
If you choose textual reasoning, explain the reasons and respond the final returned guidance with the format s<<guidance prompt content>> in the end of your response.
Now, here is the task:
Following Turns of prompts LodeSteerLLM
The response from TaskLLM is: {response)
The feedback from the checking agent is: {check_result)
The summary of generated code complexity is: {code_complen cowplexxtt _uummrry))
The final returned guidance prompt should be of the format <<<quidance prompt content>>>.
E. Prompt for Self-answer Checker
Prompt for Self-answer Checker
Given the following question and the answer from other LLMs, write a python code block check the correctness of the answer. Try to generate the code to check the correctness of the answer. Try your best to check whether the answer satisfy all the constraints of the given question. If the answer is correct, return the text "Correct". If the answer is incorrect, return the reason why the answer is wrong, like what condition or constraint is not satisfied. Question: {question] Answer: {answer]
21

---

CodeSteer: Symbolic-Augmentee Language Models via Code/Text Guidance
F. Code for Symbolic Checker
The following code checks the factors iteration, search, numeric, permutations, and combinations the answered code by LaskLLM and returns the summary of code complexity and the complexity score. We directly return the summary of code complexity as wwwpleottpyysssseeeeeeceooossese the returned cooee.oompeeee summary' concatenates with "The generated code may not be complex enough to carry symbolic computing for solving the task.'
to Lod for further guidance. If the complexity score less than 2.0,
analyze_comd code_and and galain(code_stringg """Analyzes Python code for conputational. approaches and provides explanation. import ast,
tree pprse(code_stringg
analysis !'search': False, 'synbolic': False, 'numeric's False, 'expl's (,, "score": d3
Check patterns
ast.walk(tree);
if isinstance (ast.for, ast.Mhile)): ana lysis ('search") True lysis isinstance (ast.Inport, est.InportFrom) and 'itertools' node.names ana lysis ("serrch" True ana lysis "Bgend"". itertoots for conbinetoriat search")
loops for systeratic search")
elif isinstance (node, ast.Binūp):
lysis ['nunericc] ana saalysssseen "eexot"".ageend((oontee direct nathenatical operations") ana (ellyys("score ("score'] ] +* 8.25
Check symbolic patterns 1f any(x in code_string for ['eval(', "exec(', "f" "f'S, '.forma(("]) analysis( symbolic") True
analysis exppe' Ilaappendd string operations for expression handling"))
Calculate complexity
analysis( scare') += Sun(Len(re.FF) (innn(reeiinallpaateen code string])
for pattern in ['forjwhile', 'ifjeláfjelse', 'try|exceptt]
explanation
result = [] for conp_type, detected in [('SEARCHING', analysis ('search") }],
('SYMBOLIC', analysis( syybboll" ("NUMERICAL', ana malysis| ('huneric') []];
detected:
result. (""nncoopp_tyyii type) APPROACH detected:")
result.extens [f"- (exp)" for any(x lower()
snallysss'"expt)
[comp_type. (o)er) "operation', 'toop')
return "'n".join(resutt + [f"\nComplexity score: (snoe)s analysis ('score""
Exception
return I"Invalid Python code: (str(e)}",
Figure 10: Code for checking the symbolic factors of the generated code by TaskLLM.
22

---

CodeSteer: Symbolic-Augmentte Language Models via Code/Text Guidance
G. Synthesized dataset number of each task for SFT and DPO
Table 6: Synthesized dataset number of each task for SFT and DPO fine-tuning processes.
Dataset number
SFT success trajectory numher
DPO pair number
Game 24
792
320
Path Plan
442
215
BoxLift
345
163
BoxNet
330
186
Blocksworld
406
248
Understanding
497
238
Web of Lies
492
204
Logical Deduction
489
241
Navigation
503
170
GSM-Hard
332
125
MATH Geometry
342
115
MATH Count&Proh.
346
127
Logical Equation
396
213
New Operator
394
189
Pooling
404
187
Light Puzzles
406
259
Mahjong
421
230
Statistical Counting
402
223
Matrix Transform.
391
214
Logical Puzzle
454
148
Constrained Linear Arrangement
432
155
Pattern Recognition
414
135
String Insertion
409
128
Letter Logic Diagram
500
226
String deletion&
504
230
String Synthesis
397
185
Reversi
403
194
Standard Sudoku
400
212
Total
12043
5480
H. Parameter and hardware settings of SFT/DPO fine-tuning and inference processes
We utilize four H100 80GB GPUs for hull-parameter fine-tuning of the Llama-3.1-88 models. The model is trained for 10 epochs in the SFT stage and 6 epochs in the DPO stage. The learning rate is set to x 10-5 for SFT and 5 x 10-6 for DPO. We use a batch size of 4 for training. In DPO, the loss function follows the standard sigmoid loss (Rafailov et al., 2024), with the hyperparameter 3 set to 0.1.
In most cases, we perform the inference of CodeSteerl using a single H100 80GB GPU. However, to analyze the impact of hardware configurations on CodeSteer runtime, as shown Fig. 5, we also conduct inference using four H100 GPUs for comparison.
For the generation of guidance answers in the DPO dataset creation, utilize three different SFT fine-tuned Llama-3.1 1-8B models, trained for 6, 8, and 10 epochs, respectively. For each question and stage, we query all three models and compare their generated guidance answers.
23

---

CodeSteer: Symbolic-Augmentee Language Models via Code/Text Guidance
I. Score comparison of different methods without normalization across tasks
Table 7: Average (non-normalized) score (%) for each method. The best score in every is highlighted in blue.
Avg. Score
o1
DeepSeekR1
Symbolic Agent
Code/Text Choice
Code Interpreter
GPT-4o + Codesteer
Seen
73.7
70.4
67.5
70.0
64.9
Unseen
67.8
64.8
60.9
56.0 64.9
72.2
Total
72.3
69.0
65.9
68.8
62.8
75.2
J. Score-cost table for each method
Table 8: Score-cost table for each method.
Average Norm.
Average score (t)
Average token length (4)
Average runtime (s) (L)
Baseline Methods
Only Question
53.3
566.
8.2
Symbolic Agent
74.8
1192.5
27.3
All Text + CoT
52.1
1110.7
15.3
All Code + CoT
69.6
949.8
8.9
AutoGen Conca.
69.9
1295.9
10.6
Code + Text + Sum. 1
63.1
3931.6
24.2
Code + Text + Sum. 2
62.4
2808.6
32.4
Code/Text Choice
77.9
587.4
20.1
Code Interpreter
70.5
1175.9
23.8
CoT LLMs DeepSeek RI
76.8
6396.6
68.6
82.7
N/A
70.5
ol- ol-preview
74.8
N/A
37.7
Proposed Methods
CodeSteer, 1*H100
86.4
4693.3
63.8
CodeSteer. 4*11100
86.4
4693.3
45.4
K. Comparison with heuristic-based methods
Table 9: Task success rate for CodeSteer versus three surrsticcbased methods. The highest score for each task is shown in blue.
Task Success Rate %
CodeSteer
Few-Shot
Code-First-Rule
Code-First-Agent
Game 24
933
28
68
76
Path Plan
75
54
57
Eight Queen
47
Combinatorial Calculation
86
58
47
59
2048
56
49
40
48
24

---

CodeSteer: Symbolic-Augmenter Language Models via Code/Text Guidance
L. Example text answer of DeepSeek R1 and GPT-4o in Game 24
Okey, let's need to the numbers 7, 13, to aake 24, tach number has to used exactly the. Let's start by listing out all the possible operatiors and combirationn.
First, nusbers
3, 23, 13, wast, there are 23., Sh, but the problem says each number has to used once, But since both 13s are sane, neybe that gives
flexibility, So let's is think. Naybe I can combine the 135 in some way. Like 13 divised by 13 is Is But then I already have 1. Nen, 5o if I do 12/13 el, then have and and 3. which proteo by ton't useful tere.
nhe hos to get 24
That wight not help, Raybe
then? -13
25
but
(3-1)
=24, so that would be (13v13) (3 -31 -24, Lot no veraty: 13123 26, 3-3 is 2. 26-2 is 26, Yeat But walt, have used all the nussers? Let's check, 15, 15,33,3,1, Yes- -1) -26, Let ne write that with pareatheses cornectiys 1013 -13) 13 -131 -26, Alternatively, 2t'5 (13x131-13-
Eacs is ssed once. So the coustion would be 143 1131
works, Let check
13433025, 3-10d, something.
Yeg. that's 340224. So
sotutionn Let with
there other ways.
Alternat wyly,
numbers 1,13,13, letts see. 13 (13/3)
eb, des telpful- 15 -1-12 12 415-25. Not n. tre- Alternatiell, 13 [35 -1) -1, stall a.
than what if I do 13 a Estmething] to close to 24, 1322226, 26- 2624, But vhere does 2 coee fros? 3-102, So 1342 -2024, whit, but hov to structure that Let's see: 33-13-1
shat about divided by
Than 1eled. Mo, that's not cnough. Maybo 3-13 ..... Lot no think, Be(8) -24, hav to get Let's soc. Kaybe
2). kelt, maybe not, Let's see. use 3-122, then 13x2-26, Then 20- 133 al where in the reastning number, But the reeaining number is alrendy
13w(5-33,
-in
Alseration lod 13w soectning divided by sonething. Let think. 13 (3) is 39. Thas 30 (13 121 -25, Net 24, Close. 2583 -13 w50 14-25. Not enough. Hee.
wait, and idea: 13 13 =26, That subtract and ads 1. 28-3 28-3-1e24, So that's 28-3 28-3-le24, But how to arrange that, The expression would be 113-131 -3 12. That uses all proples 13,13,3,1 spnetires
Let's compute: perentheses. Let's parentheses needed. to 23413-342-24, but coss that fit the requirnt rnd formtt? The precious eceeples used perenthesms, but raybe is's ailowe.. Let check the previous answers. ecample the first
once. (13+15)
23, plus
Yes, that works, another possible solution 19+13-3-1-24, But 13+33
is
right,
arover (130-0)((11133) uses all numbers once, So 's valid solution. not wait, raybe the prob er expects sore grouping, Let ne see, write (13+13) -3 -1, bit that's the sare as 13+13-24]. Atternatively, (13 +13) (3 -1) which we thought
but another mcmpll, anover was (2-16- (18128)) which simplifin was B315--52222,,
-28
That
compute
(136131 (3-1)
that
there be poss la se solationn
But the problew says "coch sumber has to be used is the oquttian" 5o as long as all four nurbers are used once, either solution is okay, Let's check both. Numbers used 13,13, Correct.
First solutioni (13+13) (3-1) =24, Kimbers so soth are valid. which one to choose? Nayze the second one uses parentheses, which more in Line with the exmeles giver, where parentheses are used. for rocmple, the farst is was (18-21e111-135) so grouping operat
Correct.
The wecordl unmple: ((4v11)-(2442) (18118)). So pertups the arserr the wrilten with pareniheves alov porhaps secord färst case, left to right warks, let's verify If the first solatton is correct without parentheses. Let ne seel 13415-34), Since andition and subtraction have the cane precedence, itts evatusted left to right. 26 -3 is 23, plus 1 is 24, Correct. 5o that's is a valid expressson. So the answer could be eitter (13-13-341) or (113-13--33
could acceptable.
Another approsch: Swits (12/1)). Let's compute that. 12/2 in 13, 13-13m). 30ama. No good. Alternatively, (13 -1) [3 - (23/13)1. 13/13 is 1. 3-202, 12 42-24, th, walt, that's a possible solution. Let's break it down
113 (12/1311: 13/13s1, 3-102,
Thes 12 =2 :24, So the equation as 133 -1) x (3 - (13/131) #24, 'st's check the nusters: 12,1,3,13, ALL used ence. Correct, So that's another vaiid solutson. Let me be better solution because it usleiplication and division, which wight be necessary some other combinattonns 5o this could be the interded answer, let ne check againt
13-1-12, 13/13-1, 3-1-2, 12x2-24,
So that's is another
[33 11 x 13 133 1311 <12 2-1224222-24. Correct. 5o that uses all rumbers once. So this is another valid sotution..
possible solutions:
+13 -3 41 -24
2- (13 +33) 13
-1) -24
-31 vaiie.
(13/133) -24
All
stould choose? The third involves euttiptication and diviston, which is stettar some of the examptes. for instance, the third example: 115 (13/131) -61 a 15-1)40e24, So the thtrd solution sern is ssatlar in structure, So naybe that the nxpected answer, Letts ser, Let nn check the nimbers
to with, again for solution 3: 15, 33,3,3,3,, All used orce. Tes, Each number used exactly ence, So yea. to is valid. third solution
could
exarples
grouping,
properity:
I013 -11 a 13 113/13313 =24
Breaking it down:
3 -2n2
12-1-12
Alternatvelly another write it: 133-11413 (33/131) =24, That's correct.
Alternat toe maybe even without the tener parenthesee: (33-1)+(3 -13/13)- But divistin has higher precedence, no thts the nome as (3 - (18/131). =2x24, whith doesn't help. Woit, 13-1 i5 12, 26- (12) -14, No. Not helsful. 5o that approsch night not vork.
-1)
1311 -24, Which
third solution.
Let's 13613-11
fron and (1)
5o the third solution seeus salid. Let creck if theee are any other possible contiratiors.
Jeind.
No, not
Alterntiveely (13
13-1-14. 1-1-2. 2692-18. Not 24.
was. shat about [15 3) (13 illsooechhnng" Let's ses. 15- 2-18- 3-1222, 10912-120. 126 divided by sceething. 128/5-24, But we don't have a 5, So that doesn't he
Altematively, ly, (13
+il, -14-25.
Close.
Another
113/133 Let's
13/1341, 3-1e4, 4-1e3, 1343:39, No.
Alternat uue
[13 [13 -11/soeething ingl- Not sare. let's sees 13-119, 13-271. bel-.. Net helpful.
Alternatine1/ 1y. (13 vi)
-3) +10m23. Not
/131317
3 1-2, 102ed, 13ed-33, Not helaful.
Has. Let's go back to the earlier solutions. The third solution seens the most plausiate, using Lication and divisson, which are coneon in the examples, Far exaple,, the first exapple uses wittppiccaioon, the second subtraction and andition, the third uses division and auttiolication, etc. So the third 50 solution fits that
pettern. ansorr (115 -1] 13 (13 /13131 +24, Let on conf again:
13/15-1
-1-2
12 >2:26. correct.
Yea. valid equat using all four nubbers ance. Each number used esectly once. is used in (13-1) and is (13/13), is used in (3 .... ), and froe input
The fürst 13 used In 113 -1)
- The used in [13 -1]
The used [3 in (13/13)
second
Yea, each number is used once. The tvo 13s are both used, each is different parts. So that's is correct. the sime nutier and can't used twice, but probice says each number should be used once. solution is
Alsemotivell,
problce then that the two are (including
Since
Therefore, the answer shouls (113 -1) x (3 (13 /1331> =24.
Figure 11: Example text answer of RI in the task Game 24. RI searches possible answers with the continuous back-and-fort textual reasoning process. This search process still fails in the end.
25

---

CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance
Game 24, GPT-40 text answer
soloe the 3, 3, a1d
Aoother attanpe: a 4.11
try cifferent conbinatiote
speratiera. Let exptore posibitttie::
itimen 28
(This reese's vark)
2=20
Antter eilemps
Atiies 35 U Cliss dReSeTC warK)
Lette try scetter contiratior:
2. Atttee 5 28 1
1. 11 12 vel 2 6.33
5. V1 10 50 23 12 (Thee soeee't vart)
1. 12 12-3--3 1)
Another attampt:
5. 21 x2- (Tiss docse't varkl
2: 1 Viiess 2 34
Letts try eitferert espreachs
attempts
2: stttess 2- 15.)
I. X1 12- 2 18
cTiss deese'e verks
3. 12
(Thee coesets vare)
Ieentee atreept:
1- Vittes (Thas dossn't wark!
Another atteapts
I.
3.
2. itiaes (This coeee's vart)
Lette try contiratior:
7
Amother ettempts
2- 14 CiRss doese'c warkl
1. valv 2.13
3.
Inotter atteape:
2. 10: 3
1. 11 12 vil 2.1)
3. M 10 5 22 12 (Thee soeee'e vart)
1. 11 12 selv 4-44 4-1
Aoother attempt:
5. 20 2 1 (Tias docan't work!
2: M V/asss 2- 24
Another ottempts
2: vol =
1. 12-
cless work)
3.
(This soese't vore)
(eonter 1- M. 2-6
Woother atteapts
stttime 3-30
3.
(This dossr't vart!
2.
(Thes cesen's vart)
Inether ettempt:
12 2: Vitees
Another attempts
1713 Cikss decan't world
1.
2.13
3.
caus sesse't
(nother otteeps)
2. 16 13
1. MI 12 vil 6.1)
3. XI 10 + 22 (This soeee't vort)
1. 1.31 12 Vete 4-4.)
Another atteept:
3. 20 2 22 (hiss dosett vertl
2. 1 1iles 2- 34
Another ottempt: 2: Vels -
1. XI 18
clsss wir))
3. 12
(Itis seese't vors)
Inonter: L-A1 A1 Vets
Another atteapts
3.
(Thss doser' vart!
2. (tites (This coeee's vart)
Inetter etcempt: 12 7
Another estempt:
2: 14 3 (his desse't werkl
34
1.
Another otterps: 2. 10 12 1
3.
taks desstt work)
1. XI vil 2.1.)
3. 1I 18 = 22 (This soeee'e vert)
1. 11 12 Nete 4.4
Another atteapt:
3. 1...... 20
(THss deseet verkl
2. M 7 1itees 2 14
1:31 12 2
I. MI 2 18
3. Atiies 3.6.) 2185 deess work
S.
13 (Itis seese't worl)
Another atteapts
(nottee:
1. A1 vete 2
2. (timee
3. 3-18
2253 (Thee coeen's vart)
(Thss dosrr't vart!
Another ettempo:
1. valv
Another ottempt:
3.
Atims
(Tts desset wars)
VAless 2 34.33 3.
(Iiss doese't werk!
Notter
1. Mil 6.1
fnotter atteege:
3. 1I
16 5 25 12 (This C0PER vort)
2. 18 3
V
Another atteapt:
Figure 12: Example text answer of GPT-4o in the task Game 24, GPT-4o continues the textual reasoning process until reaching the maximum token generation length but never returns the answer.
26

---

CodeSteer: Symbolic-Augmenter Language Models via Code/Text Guidance
M. Full experimental results of ablation studies
Table 10: Full experimental results of ablation studies the components in CodeSteer framework.
Methods
C.Code Steer
2. 2.WO
A.WO DPO
4.WO
5.WO Self-answer Checker
7.Agent WO Symbolic Checker
8.Agent WO
DPO
WO Data
Symbolic Checker
Agent
Self-answer
Task success rate %
Augment.
Checker
Ave. Norm., Seen
88.1
80.0
79.7
80.1
78.5
77.0
71.9
70.1
Ave. Norm., Unseen
81.3
76.2
70.9
68.6
64.2
67.9
62.0
57.4
Ave. Norm., Total
86.4
79.1
77.6
77.3
75.0
74.8
69.5
67.0
Game 24
93
93
46
62
57
37
41
28
Path Plan
75
74
43
41
29
BoxLift
77
65
72
58
47
39
BoxNet
29
21
31
13
17
30
24
15
Blocksworld
52
50
50
54
51
60
45
41
Date Understanding
87
83
80
83
89
84
92
Web of Lies
98
92
95
Logical Deduction
92
92
95
89
93
91
87
Navigation
99
90
95
85
80
93
94
88
GSM-Hard
77
74
72
79
74
76
73
70
MATH Geometry
75
69
73
68
70
MATH Count&Prob.
93
81
85
82
Logical Equation
78
58
56
56
50
52
56
New Operator
40
38
40
24
52
39
28
20
Pooling
46
43
51
45
46
44
52
Light Puzzles
68
52
56
Mahjong
90
88
92
95
77
85
79
Statistical Counting
97
98
92
95
84
93
90
96
Matrix Transform.
98
100
97
96
95
96
92
96
Logical Puzzle
70
58
52
44
58
53
54
Const. Linear Arrange.
86
81
64
Pattern Recognition
93
96
95
95
93
90
92
100
String Insertion
100
100
100
100
100
100
100
100
Letter Logic Diagram
45
20
35
35
35
30
25
23
String Modition&Modi.
93
88
86
76
String Synthesis
29
26
20
12
14
Reversi
52
49
39
52
24
36
28
36
Standard Sudoku
100
100
95
100
100
98
100
100
Letters
96
85
88
87
84
91
79
75
Eight Queen
78
74
72
72
52
73
64
52
Number Multiply
95
90
95
87
80
74
Cryptanalysis
24
12
12
String Splitting
56
56
31
43
41
52
42
40
Combinatorial Calculation
86
76
88
65
76
45
60
56
Synthesis Decomposition
66
62
64
60
53
56
44
2048
56
44
32
Permutation and Combina.
93
86
56
89
82
78
27

---

CodeSteer: Symbolic-Augmentted Language Models via Code/Text Guidance
N. System prompt of AutoGen
System prompt of AutoGen (Wu et al., 2023)
You are helpful AI assistant. Solve tasks using your coding and language skills, In the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute. 1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself. 2. task with code, the code perform the task and output the result. Finish the
When you need perform task smartly. Solve the task step by step if you need If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill. When using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't 't modify your code. So do not suggest incomplete code which requires users to modify. Don't use code block it's not intended to be executed by the user. you want the user to save the code in a file before executing put filename: filename inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print" function for the output when relevant. Check the execution result returned by the user. If the result indicates there is error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption. collect additional info you need, and think of different approach to try. When you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible. Reply "TERMINATE" in the end when everything is done.
28


Reviews:
[{"review_id": "1RLnIXx1W9", "reviewer": "Reviewer_AJhL", "summary": "This paper introduces **CodeSteer**, a model fine-tuned to enhance reasoning abilities between text and code. The authors propose a synthetic dataset and a new evaluation suite of 37 symbolic tasks to demonstrate the model’s performance on complex reasoning tasks. They claim that by leveraging both code-generation capabilities and natural language generation, CodeSteer shows significant improvements on reasoning benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful feedback. We've clarified several misunderstandings raised by the reviewer and also incorporated new experiments and analyses. Hope the reviewer could reconsider our work based on the responses below.\n\n***Question 1:*** *Direct comparison with models that use text or code alone.*\n\n**Response 1:** **We have already compared methods that use only text or only code in the original paper. As shown in Table 1**, the results for **All Text + CoT** and **All Code + CoT** represent cases where GPT-4o is prompted to generate solely text or code to solve the task in a single round (see Lines 209–213 for details). The superior performance of GPT-4o+CodeSteer further demonstrates the effectiveness of CodeSteer.\n\n***Question 2:*** *Process to generate synthetic data: How are prompts, code fragments, and textual explanations created? Are there automated processes?*\n\n**Response 2:**\n\n1. Apart from Symbench and dataset, we also propose novel training techniques such as **multi-round DPO, SFT data augmentation, Symbolic Checker, and Self-answer Checker**.\n\n2. **All training data are synthesized and validated automatically using predefined rules, without costly human annotation.** Details of dataset synthesis for SFT and DPO are provided in **Section 4.1 (Lines 157–170)** and **Section 4.2**. The data generation process, including textual reasoning components, is handled by GPT-4o guided by detailed prompts with preset knowledge or hints. The implemented prompts are shown in **Page 3 (Lines 160–163, first column) and Appendix Section D**. Generated code is extracted using predefined scripts.\n\n***Question 3:*** *Checks for correctness, diversity, and relevance to the targeted symbolic tasks. Filtering steps to remove low-quality samples. Metrics for the dataset’s comprehensiveness.*\n\n**Response 3:** **In both SFT and DPO, we include only answer trajectories that lead to correct solutions**. To encourage diverse responses, we use varied prompts for SFT (Section 4.1, Lines 163–168) and different model checkpoints for DPO. The selected training tasks span a wide range of symbolic reasoning types, as detailed in Appendix Section C and Table 4. Each task includes samples of varying complexity to promote diverse reasoning. Training data is dynamically updated based on task performance—tasks with lower performance receive more data in subsequent rounds.\n\n***Question 4:*** *How exactly were these tasks sourced or created? What cleaning and filtering procedures?*\n\n**Response 4:** **As noted in the original paper (Lines 90–100 and 128–140, first column), we collect and redevelop all the tasks from former works since their datasets and codes are not open-sourced**. The specific sourced work of each task is explained in Appendix Section C. Among 45 tasks, we select 37 tasks since the left 8 tasks are not challenging to GPT-4o.\n \n***Question 5:*** *With 37 tasks, the benchmark appears relatively small. Plan to scale it further in future work or to combine it with existing reasoning datasets?*\n\n**Response 5:** **SymBench is not small since it covers nearly all types of symbolic tasks appearing in the current research domain.** SymBench also comprises more tasks (37) than other reasoning and planning benchmarks like BIG-Bench-Hard (23 tasks), PlanBench (6 tasks), LogicGame (31 tasks). Definitely, we will add more tasks and combine with other existing reasoning datasets for a more comprehensive benchmark in the future.\n \n***Question 6:*** *Additional metrics: Exact Match, CodeBLEU*\n\n**Response 6:** Since many SymBench problems have multiple correct solutions, Exact Match is not ideal. Similarly, CodeBLEU relies on reference code and assumes a single ground-truth solution, making it less suitable since TaskLLM outputs may be in text or code and correct solutions are not unique. **The reviewer’s suggestion inspired us to explore alternative metrics that capture different aspects of generation**. For example, the following table shows the average code complexity increases with more rounds, supporting the idea that TaskLLM refines code progressively under guidance.\n\n| Code complexity score | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 |\n|------|----------------|-----------------|-----------------|-----------------|-----------------|\n|| 9.32 | 11.44 | 12.94 | 13.31 | 13.54 |\n\n***Question 7:*** *Prompt GPT-4 directly with a few-shot approach, guiding it to replicate CodeSteer’s “code-then-text” reasoning procedure.*\n\n**Response 7:** Here we **include three prompt-based baselines: Few-Shot, Code-First-Rule, Code-First-Agent**. Due to word limitation, please refer to our response to Reviewer RBSg for the full experimental results and detailed discussion. In summary, **these three prompt-based methods significantly underperform compared to CodeSteer, highlighting the value of training with our synthesized data**.\n\n**We will include all of the above contents in the revised paper. Happy to answer any further questions.**", "claims_and_evidence": "- **Claim**: The fine-tuned model (CodeSteer) performs well on symbolic reasoning tasks by switching contexts between text and code.\n- **Evidence**: The experimental results on the proposed benchmark demonstrate CodeSteer’s effectiveness, supporting the claim that it leverages code generation to assist with reasoning.\n\nWhile this claim is promising, more details on the following would strengthen the evidence:\n1. **Comparison with standard baselines**: A direct comparison with models that use text or code alone would highlight the advantages of the interleaved method.", "methods_and_evaluation_criteria": "### Dataset Synthesis\nOne of the central elements of CodeSteer is the **synthesized training dataset**. However, the steps to create this dataset are not fully detailed in the main body. Questions that arise include:\n1. **What is the precise process to generate synthetic data?**  \n   - How are prompts, code fragments, and textual explanations created or derived?\n   - Are there automated or semi-automated processes for generating these examples?\n2. **Data Evaluation**  \n   - How is the quality of the synthesized data validated?\n   - Are there checks for correctness, diversity, and relevance to the targeted symbolic tasks?\n3. **Data Quality Considerations**  \n   - What filtering steps are taken to remove low-quality or repetitive samples?\n   - What metrics (e.g., coverage of different symbolic reasoning types) ensure the dataset’s comprehensiveness?\n\n### Proposed Benchmark\nThe paper introduces a benchmark consisting of **37 symbolic tasks**:\n- **Data Collection**: How exactly were these tasks sourced or created?  \n  - Were they adapted from existing symbolic reasoning benchmarks or newly constructed?\n  - What cleaning and filtering procedures were applied?\n- **Benchmark Size**: With 37 tasks, the benchmark appears relatively small.  \n  - Is there a plan to scale it further in future work or to combine it with existing reasoning datasets for broader coverage?\n- **Placement in the Main Text**: Since the benchmark is a significant contribution, it would be beneficial to include more details and examples in the main paper (instead of primarily in the appendix).", "theoretical_claims": "There are no explicit theoretical claims beyond empirical demonstrations of CodeSteer’s effectiveness.", "experimental_designs_or_analyses": "In the current evaluation:\n- **Reported Metric**: Table 1 uses a single metric to show CodeSteer’s performance.  \n  - **Recommendation**: Consider adding additional metrics that assess different aspects of generation, such as:\n    - **Exact Match** (for correctness of final answers).\n    - **CodeBLEU** (for measuring code-generation quality and similarity).\n- **Baselines**: While CodeSteer’s performance is the focus, it would be illuminating to compare with:\n  - **GPT-4 (Few-shot)**: Prompt GPT-4 directly with a few-shot approach, guiding it to replicate CodeSteer’s “code-then-text” reasoning procedure, but without the fine-tuning. This would reveal whether the gains come primarily from the interleaved prompting or from specific CodeSteer training data.\n\nBy incorporating multiple metrics and stronger baselines, the paper’s findings could be even more compelling.", "relation_to_broader_scientific_literature": "Code Generation\nReasoning", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "see above", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r7r2aSeXt4", "reviewer": "Reviewer_nEoc", "summary": "This work introduces a comprehensive benchmark SymBench comprising 37 symbolic tasks with adjustable complexity\nand also datasets of 12k multi-round guidance/generation trajectories and 5.5k guidance comparison pairs, and fine-tune a CodeSteerLLM using the introduced datasets, achieving improved reasoning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation with our work and helpful suggestions. We've incorporated new experiments and analyses based on reviewers’ questions. We kindly ask the reviewer to reconsider our work in light of the responses below.\n\n***Question 1:*** *The authors train the CodeSteerLLM to verify and guide the reasoning process of LLMs through supervised fine-tuning on the collected dataset. This is different from the recent surge of RL to incentivize such capabilities. I am curious if RL can further improve the potential of CodeSteerLLM. I expect more experimental results that explore this direction.*\n\n**Response 1:**\n1. **Both CodeSteer and recent reasoning LLMs, such as R1, begin with supervised fine-tuning (SFT) to initialize the model with basic answer formats and reasoning capabilities.** In the second stage, we apply DPO to further optimize CodeSteerLLM, while other methods typically use PPO or GRPO. Prior work has shown that DPO, PPO, and GRPO aim to learn the same optimal policy.\n\n2. **Why do we choose DPO rather than PPO/GRPO?**\n\n**The key distinction between CodeSteer and recent reasoning models is its two-LLM design: a fixed, non-tunable TaskLLM (GPT-4o) and a learnable CodeSteerLLM.** This setup introduces three main constraints of directly applying PPO/GRPO:\n- GPT-4o is closed-source and non-deterministic, producing varied outputs even with temperature set to 0, making it difficult to evaluate the effectiveness of a single CodeSteerLLM guidance using one answer trajectory.\n- Sampling from GPT-4o incurs significant API costs, unlike the virtually unlimited sampling budget available when training open-source models.\n- Existing PPO/GRPO approaches rely on final reward signals for optimization, but in CodeSteer, intermediate guidance steps are hard to evaluate directly.\n\nTo address these challenges, we propose a tree-based multi-round guidance sampling strategy (Section 4.2). Each intermediate guidance is scored based on its average downstream returns. Additionally, since DPO relies on preference comparisons rather than precise reward values, it may require fewer TaskLLM samples than PPO/GRPO.\n\nIn the following table, we compare the performance of DPO and PPO using the same number of answer samples (ranging from 0 to 5000) and starting from the same post-SFT model. After convergence, **DPO consistently outperforms PPO**, particularly in low-sample regimes. **This supports our decision to adopt DPO in this study. We will add the above discussion and results into the final paper version.**\n\n| CodeSteer Ave. Norm. Score | Num.0 | Num.1000 | Num.3000 | Num.5000 |\n|-------|----------------|-----------------|-----------------|-----------------|\n| DPO | 79.1 | 79.5 | 80.3 | 81.4 |\n| PPO | 79.1 | 79.0 | 79.6 | 81.1 |\n\n3. **A promising future direction is to apply reinforcement learning to enhance reasoning abilities in unified code/text LLMs.** While PPO and GRPO may not offer clear advantages over DPO in our current setup, they could be more suitable when applied to a single unified model that both guides and generates code/text. In such cases, PPO/GRPO can directly optimize the policy, which we plan to explore in future work.\n\n***Question 2:*** *Moreover, why not combine CodeSteerLLM with reasoning LLMs like o1 or r1?*\n\n**Response 2:**\n\n**We have added o1+CodeSteer to further verify CodeSteer’s effectiveness.** During the time this work is done, we find the o1 and r1 APIs always report errors when requiring multi-round answer generation, thus hindering testing CodeSteer with these reasoning LLMs. Now the OpenAI platform has fixed this issue. Here we add the experiments to test whether CodeSteer can improve the performance of o1. As shown in the following table, **the performance of o1 will improve notably when augmented with CodeSteer on 5 randomly chosen unseen tasks, further verifying the effectiveness of CodeSteer. We will include these results in the final version.**\n\n| Task success rate % | Cryptanalysis | Synthesis Decomposition | 2048 | Eight Queens | Combinatorial Calculation |\n|-------|----------------|-----------------|-----------------|-----------------|-----------------|\n| o1 | 60 | 57 | 52 | 84 | 57 |\n| o1 + CodeSteer | 73 | 94 | 72 | 97 | 95 |\n\n**We are glad to answer further questions from the reviewer.**", "claims_and_evidence": "The claims are decent and supported by the comprehensive experimental results.", "methods_and_evaluation_criteria": "The authors train the CodeSteerLLM to verify and guide the reasoning process of LLMs through supervised fine-tuning on the collected dataset. This is different from the recent surge of RL to incentivize such capabilities. I am curious if RL can further improve the potential of CodeSteerLLM.", "theoretical_claims": "There are no theoretical claims to be evaluated.", "experimental_designs_or_analyses": "Experimental designs are proper.", "relation_to_broader_scientific_literature": "The proposed method can improve the reasoning capabilities of LLMs and help the realm of ai4science.", "essential_references_not_discussed": "no issue to be discussed", "other_strengths_and_weaknesses": "My biggest concern is if RL can further improve the potential of CodeSteerLLM, and I expect more experimental results that explore this direction. Moreover, why not combine CodeSteerLLM with reasoning LLMs like o1 or r1?", "other_comments_or_suggestions": "no", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ETidkxofX1", "reviewer": "Reviewer_3Uky", "summary": "This paper targets the better performance of LLMs on symbolic reasoning tasks such as Game-24 with multi-round generations. The method contains many components, such as a small model fine-tuned for guiding the generation, a self-answer checker, and a hardcoded symbolic checker. \n\nThe guidance model (CodeSteer based on Llama-8B) is fine-tuned to guide large models (e.g., GPT-4o) during multi-round generation, e.g., deciding on if to generate code to solve the problem or not. The guidance model is trained on a synthesized dataset with GPT-4o solving 28 symbolic tasks. The method is evaluated in another 9 unseen tasks. \n\nThe self-answer checker always prompts GPT-4o to generate codes to verify the answer's correctness after each round of generation. The symbolic checker evaluates the complexity of the generated codes to reject codes that are not sufficiently sophisticated for the task at hand.\n\nWith all the modules, the proposed method outperforms GPT-o1 and Deepseek-R1 for 9 unseen tasks. Without any of the symbolic checkers, the proposed method performs worse while still outperforming the baseline that prompts GPT-4o for guidance.\n\n## update after rebuttal \n\nI still have concerns regarding the generalizability of such method, like if we can use it in more general domains/tasks. The authors’ replies are not that helpful.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation with our work. The following responses clarify several misunderstandings raised by the reviewer. We've also incorporated new experiments and analyses. We kindly ask the reviewer to reconsider our work in light of the responses below.\n \n***Question 1:*** *Test on 9 unseen tasks are not sufficient to evaluate the models' performance, especially when considering the symbolic checkers that are specifically designed for the tasks at hand. Are the symbolic checkers task/domain-specific?*\n\n**Response 1:**\n\n1. **The 28 seen and 9 unseen tasks are randomly divided without human bias.** The test sets are diverse enough as they cover all 6 types of reasoning capabilities we evaluated as shown in Table 4. Furthermore, in our work, the seen tasks are also valid and decent benchmarks to evaluate since the tested samples are all different from the training samples and the complexity range is large enough.\n\n2. **The relative improvement of CodeSteer to other methods are almost the same in seen and unseen groups, showing no overfitting is happening.** We include a table of CodeSteer’s performance improvement comparison over several main baselines for both seen and unseen tasks below. \n\n| Avg. Norm. Improvement | o1 | DeepSeekR1 | Symbolic Agent |  Code/Text Choice | Code Interpreter |\n|------------------------|----|-----------|--------------------|-----------------|--------------------|\n| Seen                   | 4.3 | 8.8      | 11.1        | 8.4             | 14.8               |\n| Unseen                 | 1.9 | 12.2     | 13.4     | 9.2             | 19.4               |\n\nFrom the table we can observe, GPT-4o+CodeSteer has compatible and even better performance improvements for unseen tasks. This result indicates that no overfitting happens during our fine-tuning process and our method is generalizable.\n\n3. **Symbolic checkers are not domain-specific**. Please refer to appendix F for the whole code, which is universal for all tasks we evaluate. Symbolic checkers evaluate the code complexity by checking all the symbolic computing characteristics, with no task-specific components. Thus, they do not weaken the generalization of our method.\n\n4. **SymBench is not small since it gathers and covers nearly all types of symbolic tasks appearing in the current research domain (refer Appendix Section C describing SymBench)**.\n\n***Question 2:*** *There is no validation set, and I haven't seen links to the hyperparameter tuning process.*\n\n**Response 2:** As explained above, we directly tune hyperparameters on 28 seen tasks without requiring a validation set. **We will add the explanation of the hyperparameter tuning process in the Appendix of the revised version.**\n\n***Question 3:*** *The name, multi-round data generation, was confusing though, as it could also refer to generating the datasets multiple times.*\n\n**Response 3:** Thank you for your helpful advice. Admittedly, this will cause some confusion. **We will change it to multi-turn data generation in the final version.**\n\n***Question 4:*** *The evaluation metric is less intuitive and stable with the normalization term as the maximum pass rates of all methods. I would recommend the raw average of pass rates.*\n\n**Response 4:**\n\n1. **We use the normalized metric to better compare the relative performance among methods and prevent any single task from disproportionately influencing the overall evaluation.**\n\n2. **To ensure the robustness of our conclusions against changes in the evaluation metric, we recalculate the average score without normalization.** From the result in the table, we can observe that for both seen and unseen tasks, our method can still outperform all main baselines obviously, even more on unseen tasks.\n\n**We will add this metric in the final paper.**\n\n| Avg. Score | o1 | DeepSeekR1 | Symbolic Agent | Code/Text Choice | Code Interpreter | **GPT-4o + Codesteer** |\n|------------|----|-----------|--------------|-----------------|--------------------|-------------------|\n| Seen       | 73.7 | 70.4 | 67.5 | 70.0 | 64.9 | **76.1** |\n| Unseen     | 67.8 | 64.8 | 60.9| 64.9 | 56.0 | **72.2** |\n| Total      | 72.3 | 69.0 | 65.9 | 68.8 | 62.8 | **75.2** |\n\n***Question 5:*** *It could be interesting to finetune a bigger model, such as GPT-4o.*\n\n**Response 5:**\n\nSince we do not have the access to finetune GPT-4o, here we add the experiments to **compare finetuning CodeSteerLLM on Llama-3-8B, Llama-2-13B, and Llama-3-70B (LoRA) on the same dataset of SFT and DPO, as shown in the following table**. We find the performance will not improve apparently with larger models. The current main bottleneck is still the lack of high-quality dataset.\n\n| Ave. Norm. Score | Seen | Unseen |\n|-------|----------------|-----------------|\n| Llama-3-8B (full-parameter) | 88.1 | 81.3 |\n| Llama-2-13B (full-parameter) | 87.3 | 82.2 |\n| Llama-3-70B (LoRA) | 88.6 | 80.0 |\n\n**We will add the above new experimental results and discussion in the final paper.**", "claims_and_evidence": "The authors claim that the proposed outperforms baselines including o1. However, it only shows evidence in 9 tasks. They are not sufficient to evaluate the models' performance in general, especially when considering the symbolic checkers that are specifically designed for the tasks at hand.", "methods_and_evaluation_criteria": "I like the proposed data generation and fine-tuning process for the guidance module in general. The model is fine-tuned to prefer actions that lead to higher returns whose expectations are estimated given the MCTS trees. The name, multi-round data generation, was confusing though, as it could also refer to generating the datasets multiple times (e.g., obtaining new MCTS trees using the current fine-tuned guidance model). \n\nThe evaluation metric is less intuitive and stable with the normalization term as the maximum pass rates of all methods. I would recommend the raw average of pass rates. This may not be too problematic, as I did check the other metric using the numbers reported in Table 1, and the ranking of methods looks similar to the reported one.", "theoretical_claims": "No theoretical claim in this paper.", "experimental_designs_or_analyses": "The test sets are small and may not be diverse enough. \n\nThere is no validation set, and I haven't seen links to the hyperparameter tuning process.", "relation_to_broader_scientific_literature": "Guided search is important for test-time-compute scaling. There are multiple works investigating it for various domains, such as multi-round code refinement. The recent reasoning modules such as o1 and R1 seem to have similar abilities as well.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "It could be interesting to finetune a bigger model, such as GPT-4o, for guidance, but I am not sure if the performance will improve a lot and surpass o1 without the symbolic checkers.", "questions_for_authors": "* Are the symbolic checkers task/domain-specific? Can they generalize more general reasoning tasks?\n* Are there results in more domains and tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Cl6Gn2xr4", "reviewer": "Reviewer_RBSg", "summary": "The paper introduces CodeSteer, a method to guide large language models (LLMs) in making optimal choices between textual reasoning and code generation. The authors propose a multi-round supervised fine-tuning (SFT) and direct preference optimization (DPO) approach using a newly created dataset. A benchmark, SymBench, is introduced, containing 37 reasoning tasks. The model CodeSteerLLM, trained on Llama-3-8B, significantly enhances GPT-4o’s symbolic reasoning capabilities. Experiments demonstrate that CodeSteer improves the performance of GPT-4o from 53.3% to 86.4%, outperforming OpenAI’s o1 and DeepSeek R1. The framework generalizes across different LLMs, improving performance on Claude-3, Mistral, and GPT-3.5.\n\n## update after rebuttal\nThe rebuttal address my questions. I remain positive on the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation with our work and helpful suggestions. Here are our responses and newly added experiments based on reviewers’ questions. We kindly ask the reviewer to reconsider our work in light of the responses below. We are happy to further communicate with the reviewer.\n\n***Question 1:*** *The evaluation could point out failure cases where CodeSteer struggles. This will help provide insights on how to further improve on the work.*\n\n**Response 1:**\nCodeSteer encounters failures under several conditions:\n1. **Insufficient Capability of TaskLLM**: In some cases, the capabilities of the TaskLLM—whether through coding or textual reasoning—are not sufficient to solve the given problem.\n\n2. **Suboptimal Code Generation**: The generated code may not use the most efficient method, which can lead to timeouts. For example, as shown in Figure 4c, CodeSteer’s success rate decreases when the target values increase, due to the exponential growth in search complexity.\n\n3. **Lack of Robustness to Task Complexity**: CodeSteer is not yet robust enough across tasks with varying complexity. As shown in Figure 4a, performance drops in medium-complexity samples. In these cases, CodeSteer sometimes selects textual reasoning over coding and ends up producing incorrect answers.\n \n**We will elaborate on these issues, along with relevant examples, in the Appendix to guide future improvements to the system.**\n\n***Question 2:*** *In Figure 1, why not address the case 1 problem with coding approach as well?*\n\n**Response 2:**\n\nThank you for the great question. In our study, **solving Case 1 problems through coding is acceptable**. During training data synthesis, we do not explicitly account for differences in execution or time cost between code generation and textual reasoning. Our primary criterion for data selection is whether the answer is correct. If both coding and textual reasoning successfully solve a problem, both are included in the dataset. Since code execution costs and runtimes **depend on user-specific hardware**, it is difficult to define a consistent comparison metric between the two approaches. Instead, we incorporate the number of guidance/generation rounds as part of the DPO scoring to favor more efficient solutions. **We will include this discussion in the revised version of the paper.**\n\n***Question 3:*** *Can an alternative approach, where coding is preferred at the beginning and, if coding failed, switched to text-based reasoning, work? From Figure 1's examples, it seem this alternative approach could achieve similar results as the proposed method. I had the question mainly because it is hard to come up with examples where there would be frequent switching between code and textual based reasoning. Based on the examples shown in the work, the solution seem to be having the LLM first generating code-based solutions. And if the code-based solutions are hard-coded (detected from rules), then ask the model to do textual-based reasoning instead.*\n\n**Response 3:**\n\nThank you for the insightful suggestions. Based on the suggestion from you and Reviewer AJhL, To better evaluate CodeSteer and provide deeper comparisons, **we include three prompt-based baselines:**\n\n1. **Few-Shot**: Uses five example-based prompts to guide the TaskLLM in mimicking the 'code/text' switching reasoning process.\n2. **Code-First-Rule**: A rule-based approach where the TaskLLM is prompted to use code for the first three rounds (with increasing complexity) and then switch to text-based reasoning.\n3. **Code-First-Agent**: Employs GPT-4o as the CodeSteerLLM to guide the TaskLLM using the same code-first-then-text strategy as in **Code-First-Rule**.\n\nAs shown in the table, the three prompt-based methods perform significantly worse than CodeSteer, **underscoring the effectiveness of training with our synthesized data**. Upon analyzing failure cases, we identify two main reasons:\n- CodeSteerLLM’s guidance often includes problem-specific coding knowledge (e.g., suggesting A* or DFS) and how to formalize the problem, which **purely prompt-based methods struggle to capture**.\n- **Switching between code and text can be advantageous**, as later code generations can build on insights from prior textual reasoning. For instance, in Path Plan, a text-generated trajectory may be partially correct; subsequent code can refine it directly, reducing the search space.\n\n| Task Success Rate % | CodeSteer | Few-Shot | Code-First-Rule | Code-First-Agent |\n|-------|----------------|-----------------|-----------------|-----------------|\n| Game 24 | **93** | 28 | 68 | 76 |\n| Path Plan | **75** | 54 | 59 | 57 |\n| Eight Queen | **78** | 47 | 62 | 73 |\n| Combinatorial Calculation | **86** | 58 | 47 | 59 |\n| 2048 | **56** | 49 | 40 | 48 |\n\n**We will include the above added experiments and discussion in the final paper.**", "claims_and_evidence": "Yes, the claims are well-supported. \n+ Performance improvement demonstrated in Table 1 \n+ Ablation studies show the importance of the design choices", "methods_and_evaluation_criteria": "Yes, the methodology is well-founded and appropriate.\n+ Eval is performed on both seen and unseen tasks \n\nMinor suggestion: \n- The evaluation could point out failure cases where CodeSteer struggles. This will help provide insights on how to further improve on the work.", "theoretical_claims": "The paper does not rely on theoretical claims, but its methodology is sound.\nBoth DPO and SFT and well-studied in the literature, the work uses these approaches to build CodeSteer.", "experimental_designs_or_analyses": "Yes, the experimental setup is well-structured.", "relation_to_broader_scientific_literature": "+ The development of SymBench, which is open-sourced, unlike prior works. \n+ New methods for dataset construction and fine-tuning \n+ Strong empirical results", "essential_references_not_discussed": "The paper is well-positioned within the broader literature.", "other_strengths_and_weaknesses": "+ The paper present a multi-round guidance mechanism to help LLMs to select between producing code or text-based reasoning \n+ The design elements such as DPO, self-answer checking, symbolic evaluation seems to be helpful", "other_comments_or_suggestions": "- see questions for authors.", "questions_for_authors": "Overall, the paper is a pleasant read. The paper is well-structured and easy to follow. The empirical study supports the claims made by the paper. The methodology seems sound and the empirical results demonstrate the effectiveness of CodeSteer in assisting LLMs in solving reasoning tasks. Below are a few questions: \n\n- What are the failure cases where CodeSteer struggles?\n- In Figure 1, why not address the case 1 problem with coding approach as well? \n- Can an alternative approach, where coding is preferred at the beginning and, if coding failed, switched to text-based reasoning, work? From Figure 1's examples, it seem this alternative approach could achieve similar results as the proposed method. I had the question mainly because it is hard to come up with examples where there would be frequent switching between code and textual based reasoning. Based on the examples shown in the work, the solution seem to be having the LLM first generating code-based solutions. And if the code-based solutions are hard-coded (detected from rules), then ask the model to do textual-based reasoning instead.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper made several interesting contributions to the space of coding / reasoning, specifically in text vs coding based reasoning. Reviewers generally find the following contributions to be valid:
- the benchmark
- the data synthesis technique and process
- the framework that can leverage both the tuned llm and powerful model like gpt-4o
- and the empirical gain over benchmark datasets.

The reviewer AJhL found some aspects that can be further improved or clarified, and the authors promised to include that in the final revision. This decision of acceptance is based on that the authors will carefully incorporate the improvements and clarifications in their revisions.
Decision: Accept (poster)