CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning
Authors: Jiaxin Wen
Jian Guan
Hongning Wang
Wei Wu
Minlie Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Large langauge model for reasoning
planning
code-aided reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce CodePlan, a scalable paradigm that empowers LLMs to generate and follow code-form plans---pseudocode that outlines high-level, structured reasoning process.

Abstract: Despite the remarkable success of large language models (LLMs) on traditional natural language processing tasks, their planning ability remains a critical bottleneck in tackling complex multi-step reasoning tasks. Existing approaches mainly rely on prompting or task-specific fine-tuning, often suffering from weak robustness and cross-task generalization. To address the limitation, we introduce CodePlan, a scalable paradigm that empowers LLMs to generate and follow code-form plans---pseudocode that outlines high-level, structured reasoning processes. By leveraging the structured and versatile nature of code, CodePlan effectively captures the rich semantics and control flows inherent to sophisticated reasoning. Importantly, CodePlan allows the automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. This enables it to scale up efficiently and improve reasoning capabilities across diverse scenarios. To train CodePlan, we construct a large-scale dataset of 2M examples that integrate code-form plans with standard prompt-response pairs from existing corpora. With minimal computation overhead during both training and inference, CodePlan achieves a 25.1\% relative improvement compared with directly generating responses, averaged across 13 challenging multi-step reasoning benchmarks, spanning mathematical reasoning, symbolic reasoning, instruction-following, multi-hop QA, and decision-making tasks. Further analysis reveals CodePlan's increasing performance gains on more complex reasoning tasks, as well as significant data efficiency thanks to its generalization ability.

Published as a conference paper at ICLR 2025
CODEPLAN: UNLOCKING REASONING POTENTIAL IN LLMs BY SCALING CODE-FORM PLANNING
Jiaxin Wen 1, Jian Guan Hongning Wang', Wei 2 Minlie Huang"
'Tsinghua University AAt Group
(jiaxinwenthu, j jianguanthu, wang hongn @mmm com wwwe119850318eeee il. com, aituanget:singhua edu cn
ABSTRACT
Despite the remarkable success of large language models (LLMs) traditional nar- ural language processing tasks, their planning ability remains a critical bottleneck in tackling complex multi-step reasoning tasks. Existing approaches mainly rely on prompting or task-specific fine-tuning, often suffering from poor robustness and cross-task generalization. To address the limitation, we introduce CODEPLAN, scalable framework that empowers LLMs to generate and follow code-form plans pseudocode that outlines high-level, structured reasoning processes. By leveraging the structured and versatile nature of code, CODEPLAN effectively captures the rich semantics and control flows inherent to sophisticated reasoning tasks. Impor- tantly, CODEPLAN allows automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. This enables it to scale up efficiently and improve LLM's reasoning capabilities across diverse scenarios. To train CODEPLAN, we construct large-scale dataset of 2M examples that integrate code-form plans with standard prompt-response pairs from existing corpora. With minimal computation overhead during both training and inference, CODEPLAN achieves 25.1% % relative improvement compared with directly generating responses, averaged across 13 challenging multi-step reasoning benchmarks, spanning mathematical reasoning, symbolic reasoning, instruction- following, multi-hop QA, and decision-making tasks. Further analysis reveals CODEPLAN's increasing performance gains on more complex reasoning tasks, well as significant data efficiency thanks to its generalization ability.
INTRODUCTION
With the rapid progress in pre-training and post-training (Brown 2020 Chung al. 2024), large
language models (LLMs) have exhibited remarkable performance across wide range of natural language processing (NLP) tasks. However, as LLMs are tasked with solving increasingly complex problems that require multi-step reasoning, such as mathematical problems (Hendrycks et al. [202]] multi-hop auestion-answeriin
(Trivedi et al. (0022), and complex decision-making (Shridhar et al. 2020 Xie et al.) 2024), their limited planning capability has become critical bottleneck ( (Yang et al.
2023 As illustrated in Figure T LLMs often exhibit various failure modes in multi-step reasoning. such repetitive steps, incoherent logic, focus drift, and early answering (Yao et al.) 2022 Lanham
et al. 2023). Effective planning, i.e., generating a high-level abstraction in advance (Yang 2012
Russell Norvig 2016), can frame the subsequent reasoning procedure, thereby guiding LLMs
through the intricate low-level steps to ultimately solve the tasks (Wang et al. 2023).
Delving deeper, LLMs' planning deficiencies largely stem from the fact that massive pre-training
text corpora often do not explicitly exhibit the underlying reasoning structures, thereby obscuring the latent, high-level planning signals that LLMs should learn (Zelikman al. (2024). To remedy this challenge, current approaches mainly frame LLMs' reasoning procedures through either advanced prompting techniques (Wei al. 2022a Yao et al. (024) or task-speeific fine-tuning (Zelikman et al.)
2022 Guan et al. 2024). However, prompting approaches typically impose strict requirements on
"Equal Contribution
Corresponning Authors

---

Published as a conference paper at ICLR 2025
Table I: Comparison between CODEPI.AN and representative methods in LM planning, evaluated from two perspectives: (1) Expressiveness, including structuring for representing complex logic, versatility for diverse domains and interpretability; and (2) Learning, including data abundance, and training/infernnee efficiency.
Expressinnes
Learning
Method
Plan
Structuring
Versatility Interpeetability
Data Abundance
Efliciency
CUT [Wei al. 2022a
Steps Interwined with Surface Realization
X
N/A
Pho--ma-oaaee Hanrdd al. 2023
Free-Form Natural I. Tout
x
AMOR (Gusn
Expert Designed Finite State Machine
/
ProdictiiFPA [Cemlle
Lateni
Qulet-STaR Zenkoma
Learnalle Latent
CODEPIAN (This Woek)
Free-Form Programming Language Code
/
models' inherent capabilities as well as carefully designed prompts (Anagnostidis & Bulian 2024). while task-specific fine-tuning approaches limit the models' ability to generalize to new domains.
To surmount the aforementioned issues, we propose CODEPLAN, a novel, scalable framework that empowers LLMs to generate and follow code-form flans-pseudocode that serves as high-level, structured blueprints of the reasoning process. By leveraging the structured and versatile nature of code (Madaan et al.) 2022), CODEPLAN effectively captures the rich semantics and control flows inherent to sophisticated reasoning. As exemplified in Figure [ code naturally supports various common reasoning structures, including the hierarchical composition of multiple subtasks (function making and calling), iterative steps (for-loops), and conditional multi-branch steps (i f- statements). Importantly CODEPLAN allows the automatic extraction of code-form plans from massive, wide-ranging text corpora that naturally embed the planning signals, bypassing the need for curated, task-specific datasets. This enables CODEPLAN to scale efficiently and improve reasoning capabilities across diverse tasks beyond specific ones like mathematical reasoning Yu et al. 2023). Table summarizes the advantages of CODEPIAN against prior work.
To train CODEPLAN, we construct a large-scale dataset with 2M examples in the form of (prompt, code-form plan, response). We validate the effectiveness of CODEPLAN in multiple models, includ- ing Mistral (Jiang et al. 2023) and Llama (Touvron et al., 2023 Dubey et al. 2024). Extensive experiments show that CODEPLAN consistently and significantly outperforms directly generating re- sponses without planning, yielding a relative 25.1% % performance gain averaged across 13 challenging reasoning benchmarks spanning mathematical reasoning, symbolic reasoning, instruction instruction-folloiing multi-hop question answering. and decision-male for the models' enhanced ability to tackle complex multi-step reasoning problems. By leveraging code-form plans an intermediate representation during training, we pioncer a scalable framework for endowing LLMs with structured, versatile, and interpretable reasoning - a capability that has remained elusive when relying solely on natural language.
tasks. These results provide compelling evidence
In summary, this work makes several pivotal contributions:
L. We introduce CODEPLAN, novel, scalable framework that empowers LLMs to generate and follow code-form pauus-pseddooode that outlines high-level, structured reasoning processes. This framework unlocks new frontiers for structured reasoning with LLMs, transcending the limitations imposed by the obscured implicit planning signals in natural language text.
II. CODEPLAN allows efficient and cost-effective training data construction from massive, wide- ranging corpora, enabling promising data scalability. We exemplify this by curating a large-scale dataset comprised of 2M prompt-response pairs along with their corresponding code-form plans. This dataset also establishes a rich resource for future research on reasoning in L.MMs.
III. We demonstrate CODEPLAN'S remarkable efficacy and generality across 13 challenging reasoning benchmarks on multiple backbone models, scaling from 7B to 13B. Further analysis reveals its growing advantage over baselines as problem complexity increases, and its strong data efficiency.
2 METHODOLOGY
We formally define the multi-step reasoning task as follows: Given a prompt X that poses a problem, the goal is generate response Y that requires comprehensive solution through sequence of log-
2

---

Published as a conference paper at ICLR 2025
(1) Mathematical Reasoning
(1c) CODEPLAN
Prompt: How many digits are required to number book containing 240 pages?
Plan:
een pages
(1a)
extion
Response: You number dddllts, the number of book have diglts (EARcY Asswensw))
from 240, Since 240 has
(1b) Standard Training
reter
Response: find the number of digis required, we can use the formula:
Response:
the number right requined,
fallowing stepie
Number digits floorllog logt Ojnumberj)
1. Determine the number of requird each number
(becossect FoRmuAA)
For numbers 1-9,1 dight required.
where floor(s) (X) rounds down to nearest integer.
For numbers
clgisjare required.
the number 240, Number of digits = #oorllo210/2400
For numbers 100-240. digitslore required.
2. Calzulate the total number of dight required.
Bloor' 3802)
For 1% there are 9 numbers, so 9 dddiptsore required.
10:09, there n0mbrrr so 180 digtsslar required.
For 100-240, there are 341 numbers, so R23 diglisare required.
Therefane, diglts are required to number a book containng 240 pages.
lotal digit required 9 180 423 612
Therefore, 612 digits required to
containing 240 papes.
Instruction-Frn Following
(2c) CODEPLAN
Prompt: Compare and the diffrrent typos of renewable
energy sources available today and discuss their teasibility.
Response: Solat (2a) Base Model
(twerels ('oacatiiee E. cpoestlllligh... 9,5%,
abundant and most widely for
used is captured by
"slomassti ('Iesslaiisyy' M4, "mcenntbiilly"" 6.4)
electricity, Wind energy generated by turbinas used generate electricity Hydro energy generated. Geothermal generated.. ANAuse CoWnASSONs) Solar energy is the most abundant and the most widely used (Ateasrmve wmu eeer PVANGRAPH)
4f accce 2- 2.4
siLE sance "fessible"
5- dar soance acasms maagy axarces isecar: feantbil Loye Hi, cacoonal Aecomecttheeee
(2b) Standard Training
Response:
Response:
The feasibility of each renewabie energy source depends on various factors such as the avaliabillty.
Solar energy's is lighl bl it abundant and can be hamnessed various forms such
photovoltaic (PV) parels and solar thermal systems. However, access billty limiteed
Henewoble energy sources can be dessified into three main types (Thr Falowwi list is Not comperanooe AND Lacks Communsous) 1. Biomass.... Blomass is widely used and sustainable energy sounce.. BBowess Evency Resenvery Nor Wizerr Uszo IN FACT)
conditions the for large-scale
Wind energy mone sccessible than solar energyoot iequirs suitable wind corditionsood may opposition from local
Hydroelectric powsei....s ...s Broven tachnology bus can have signfficntt environmental Impacts including dispacement of communties
2. Hydropawer.... ..
fragmentation ecosyremss
Geathermal energy...
Eecthormal energy .is rollable source but (scuires specific goological conditions and can expensive
the feasibility of using them depends on factors such the inailability the
(ReACTITVE
PARAGZAPH)
Biomass energy... .. renewable source butt raises concerss about sustamsbilltt.....
Figure 1: Two examples for the mathematical reasoning task (Top) and instruction-followee task (Bottom) with Mistral-7B as the base model. Words highlighted in red: Unreasonable reasoning steps; Maroon words: Conditional branches in the plan; Blue words: Iterative loops in the plan; Purple boxes: Function making in the plan; Golden underlined words: Function calling in the
plan; Words highlighted in green: Essential reasoning steps in the response adhering to the plan.
ical reasoning steps. In this section, we elaborate on the method for solving the task by decomposing the generation process (22 into two stages: planning (22.) and surface realization (2.3)
2.1 FORMALIZATION
Typically, LLMs are trained to minimize the negative log- loglliiel of ground-truth outputs:
- log p( xY|
To imbue LLMs with structured and systematic reasoning capabilities, we propose decomposing the generation process into two stages: planning and surface realization (Reiter & Dale 2000). The planning stage aims to generate plan Z that outlines the control flow for solving the problem X, while the surface realization stage then translates this plan into the final natural language response Y, fleshing out the low-level reasoning details. The optimization objective L is then modeled as follows:
L logr(Y|X)) = log Ex(2))))(( 2),
(2)
where p(Z (Z and p(Y X, Z) refer to the planning model and surface realization model, respectively. Nevertheless, marginalizing over the latent variable Z is generally intractable, as the search space
3

---

Published as a conference paper at ICLR 2025
could be vast. To circumvent the challenge, we minimize variational upper bound of the loss with posterior q([|X, (Kingmal 2013). Hence, the objective can be formulated as:
C = log E pptz (YI X; 2) < Py(Z log P 2) + KI. (a(Z)x ()
(3) 2022a) that intertwine planning and realization steps, the
In contrast to existing approaches (Wei et al. formulation explicitly disentangles these two stages, allowing for systematic generation of structured plans that effectively guide low-level reasoning steps. Moreover, overcomes the limitations caused by the obscured planning signals in natural language data.
well-defined q( X, Y) is crucial for effectively optimizing Eq. 3 We provide two principled approaches to model this posterior distribution:
(1) Explicit Plans. straightforward assumption is that there exists a "gold annotator" who excels at summarizing a high-quality plan 2" for any given (X,Y) pair. Under the assumption, g g follows Dirac distribution: {Z] Y) = 8(2 Z*)
Xx, Y)
(2) Implicit Plans. Alternatively, plans can be modeled as learnable latent vectors that implicitly encode expert knowledge, styles, or other nuances shaping the reasoning process (Kingma 2013). In this context, [ Y) can be defined as continuous Gaussian distribution (Kingma 2013). discrete
multinomial distribution over a learnable codebook (Van Den Oord et al., 2017; Cornille et al.)
2024) pre-defined vocabulary (Zelikman et al.) 2024). Compared to explicit representations, latent vectors might capture more subtle planning aspects. However, this approach often introduces computation overhead and may suffer from posterior collapse (Bowman et al. 2016).
In this work, we adopt the simple and explainable "Explicit Plans" approach by setting (Z 6(Z = Z*), where 2" denotes the plan provided by "gold annotator." We reserve the exploration of the implicit approach for future work. This setting then yields the following upper bound for Eq. 3
È = log p (Y|X, Z²) - log (X)
(4)
For simplicity, we unify the planning model into a single LM architecture with shared parameters O. Crucially, this formulation implies that an optimal plan 7" should achieve delicate balance between informativeness for effective reasoning guidance and conciseness for efficient generation. This balance minimizes the combined difficulty of plan generation and subsequent realization, thereby optimizing overall model performance.
and the surface realization model P(Y|X, Z*)
2.2 PLANNING
Given prompt X, z" should capture the rich semantics and control flows inherent in the reasoning path Y. This motivates us to use programming languages-which are Turing complete (Boyer & Moore 1983 as general representation of 2", thereby framing planning code generation.
As shown in Figure D this code-form representation versatile across diverse scenarios with several compelling advantages: (1) It scamlessly incorporates conditional branching (i.e., i ff-stttements) to dynamically adapt reasoning steps to intermediate results or contexts. (2) It integrates iterative loops (i.e., for-loops) to handle sequential data perform repeated operations. (3) It defines and deeelleeeeeiiii composes modular tools (i.e., Python functions), enhancing LLMs' abilities to craft and use tools (Cai et al. 2024). This also potentially endows our approach with the ability build agents that can interact with external environments through specific APIS (Hausknecht et al. 2020), which is left for future work. (4) From a high-level perspective, it naturally supports hierarchical reasoning structure, by defining variables and attributes upfront, addressing tub-tasks via specific functions, and orchestrating the main procedure using rigorous formal logic. This fosters the breakdown of complex reasoning problems into modular sub-components, facilitating the effective transfer of planning knowledge and enhancing the model's 's systematic reasoning abilities (Yang et al., 2023).
Given a dataset of (X, Y) pairs, we obtain the annotation of 7" by prompting an LLM that has been extensively pre-trained on code data, as detailed in $3.1 Specifically, we instruct the model to generate Python-style pseudocode that outlines the reasoning structure for solving the problem X and arriving at the response Y. We do not mandate the generated plans to be executable in light of three considerations: (1) encoding reasoning logic using code is sufficient for generality and scalability, and execution might be unnecessary: (2) generating pseudocode is more liken-efficient than executable code; and (3) our pilot study finds that existing L.LMs still struggle to generate fully
4

---

Published as a conference paper at ICLR 2025
Table 2: Instruction for generating the code-form plan for given prompt-response pair.
Prompt: {(Prompt)) Respense: [Response)
Given zcompt-respont paic, your task is describe high -evell logic of the response using pseude Python code. Such that following this code, models can easily generate the response.
The code should bolance conciseneas and informativenesa.
should high-level, instead replicat low-level detaila response, The code should be less than 200 words (adjust its length based on responee lengthe).
executable code plans for various tasks, as this requires a comprehensive understanding of various libraries, APIs, and domain-specific knowledge. By circumventing the execution step. we can focus on the core challenge of generating structured plans that capture the underlying reasoning logic, without additional complexities. After obtaining the dataset of (X, Z" Y) triples, we optimize the model for plan generation by minimizing the second term of Eq. (i.e., log (X))
2.3 SURFACE REALIZATION
Next, we proceed to the surface realization stage, aiming to generate a comprehensive response Y to the prompt X under the guidance of the high-level code-form plan Za, To this end, we optimize the model by minimizing the first term of Eq. (i.e.. - log p X, 2*)).
As illustrated in Figure T this enables the controllable generation of Y that adheres to the human- readable plan Z. For instance, when realizing 1E-statements, the model is explicitly conditioned on the multi-branch conditions specified in the code. ensuring adherence to the intended logic. Similarly, when expanding for-loops, the model is guided to systematically process each element, following the encoded iteration logic, which is critical for tasks involving iterative reasoning or sequential decision-making making (Shridhar al., 2020 Xie et al., 2024). This tight coupling between the code-form plan and the final reasoning path enables the model to produce coherent, logically sound solutions that faithfully reflect the intricate reasoning structure.
EXPERIMENTS
3.1 TRAINING DATA CURATION
To facilitate effective learning of planning and surface realization, we curate a large-scale training dataset of examples in the form (X, Z", Y). We start from WebInstruct (Yue al.) 2024) that is automatically constructed from raw web data and contains diverse prompt-response pairs, and prompt Llama-3-BB-Instruee (Dubey
2024) to synthesize Z Table shows the prompt. We use nucleus sampling (Holtzman et al. 2020) (p 0.9) with temperature of 0.7. This approach enables efficient construction of large-scale datasets, and is readily extensible other existing corpora such as Li et al. 2024) and Cheng et al. (2024b). More details are presented in Appendix A.1
3.2 BASELINES
We evaluate the following baselines: (1) Plan-and-Solve (PS) Prompting: It prompts the LLM first devise natural language plan decompose the entire task into smaller subtasks, and then generate the response following the plan (Wang et al. (023). (2) Quiet-STaR: It automatically
learns implicit plans for generating each token from Weblnstruct (Zelikman al. 2024). (3) Vanilla
Training: It first trains the LLM on WebInstruct and then prompts it to directly generate the response.
3.3 EXPERIMENTAL SETUP
We select Mistral-7B aaag et al., 2023) and Llama-2-7B/13B (Tovroo et al. 2023 as our backbone models. For Mistral/Llama, use learning rate of 5e-6/1e-5 and global batch size of 512/256. respectively. We set the maximum training epochs to 2. During inference, models are instructed to generate CoT-style responses for all tasks. Unless stated otherwise, conduct evaluations under the
5

---

Published as a conference paper at ICLR 2025
fhw-shot setting (from 2-shot to 4-shot) without fine-tuning evaluation benchmarks. The above settings are also applied on baselines for fair comparisons.
3.4 EVALUATION BENCHMARKS
We evaluate CODEPLAN across a diverse range of tasks necessitating multi-step reasoning:
Mathematical Reasoning. It involves three benchmarks: (1) GSM8K (Cobbe al., 2021
collection of grade-school problems; (2) MATH (Hendrycks et al. 202I). a more challenging suite of high-school-lewe level problems; and (3) SVAMP (Patel et al. 2021p. a robustness evaluation benchmark.
Symbolic Reasoning. We use four benchmarks requiring multi-step logical deductions: (1) Boolean Expression from Big-bench-hard -aard (BBH) (Suzgum al., 2022). where the model infers the value of a complex boolean expression; (2) Coin Flipping (Wei et al. (022b)) which requires determining the final face after a sequence of flips. We use the challenging 4-f version; (3) Last Letter Concatenation (Wei al., 2022b), which requires concatenating the last letter of 4-word sequence; and (4) Dyck Language from BBH, which requires predicting the closing parentheses of a Dyck-4 word. This task often demands over 10 reasoning steps. We find all models struggle to cahieve non-trivial few-shot performance on this task due to model degeneration (Holtzman et al.] (020). To mitigate these confounding factors beyond planning capabilities, we synthesize 5K examples based official implementation to fine-tune each model before evaluation.
Eetruction-Following We assess the proficiency in following real-world instructions two benchmarks: (1) AlpacaEval 1.0 (Li et al. 2023) and 2.0 (Dubois et al.] (024), which capture representative user interactions: and (2) MT-Bench (Zheng et al. (024)20 meticulously curated set of high-quality queries spanning eight domains. Since our training data are not specifically tailored for cnerrcttionfolloiinge we follow prior work (Tunstall al. 2023) to continue fine-tuning each
trained model on 150K randomly sampled examples from UltraChat DDing et al. 2023).
Multi-hop Question-Answering (QA). We assess multi-step reasoning over complex information
dependencies on two becnhmarks: (1) HotpotQA (Yang et al. 2018), which comprises 2- questions requiring reasoning over two supporting passages; and (2) MuSiQue (Trivedi et al. 2002). which consists of 2-hop to 4-hop questions with intricate dependency structures. Since our method focuses primarily on enhancing planning capabilities rather than knowledge acquisition, we provide the gold reference passages in the context during evaluation. We report the exact match (EM)scores.
Decision Making. We use one benchmark to evaluate the performance in sequential preissonnmaking scenarios: ALFWorld (Shridhar et al. 2020). a text-based virtual household environment comprising six distinct task types. It necessitates the model to navigate through intricate sequences of actions and observations, posing rigorous test of the models' planning capabilities in dynamic environments. We use ReAct-style reasoning steps (aao et al.) 2022) during evaluation.
3.5
MAIN RESULTS
As highlighted in Table CODEPLAN shows consistent and substantial performance improve- ments across all backbone models and most benchmarks against the baselines, underscoring the efficacy of incorporating code-form plans in enhancing LLMs' systematic reasoning capabilities. Specifically, CODEPLAN generally outperforms the PS Prompting baseline, often by a substantial margin. This indicates that the benefits of learning plan in code are beyond what can be achieved by planning in natural language with careful prompt engineering alone. Quiet-STaR demonstrates inferior performance even compared to the backbone model despite significant computation over- head"] illustrating the difficulty of learning implicit plans through latent variables. Notably, the vanilla training approach, which solely learns low-level reasoning steps, does not always improve backbone models' performance on downstream tasks. For instance, the performance of Mistral-7B drops by 9 points on several symbolic, multi-hop QA and decision-making tasks after vanilla training. We attribute this to the distribution gap between the massive, wide-ranging corpus and downstream benchmarks, thereby leading to suboptimal adaptation of the model's 's reasoning capabili- ties. In contrast, CODEPI.AN consistently improves upon the initial backhone model across all benchmarks, demonstrating its ability to develop more generalizable and transferable planning
We use the official implementation for Quict-STaR, which only supports Mistral and not. Llama series.

---

Published as a conference paper at ICLR 2025
Table 3: Main results five types of reasoning tasks. A means the margin between vanilla training (Vanilla) and CODEPLAN. On average, CODEPLAN yields relative 25.1% performance gain against vanilla training. We highlight the best results in bold. We report accuracies for mathematical reasoning, symbolic reasoning, and decision decision-making tasks, the EM and F1 scores for multi-hop QA tasks, the win rate for AlpacaEval, and the GPT-4 score for MT-bench.
Mathenatical Reasoning
Symbolic Reasoning
Model
GSM8K
MATH
SVAMP
Bookan
Cain Flip
Last Letter
Dyck Langunge
Mistral-78
46,9
18.8
47.5
71.2
84.1
39.5
73.0
+ PS Prompting
45.5
17.3
58.5
75.6
79.5
41.5
70.1
Quiet-STaR
45.6
15.9
47.5
66.0
57.2
1.5
$4.4
Vanilla
54.1
31.5
55.2
85.6
86.1
37.5
CODEPLAN
59.5
61.4
92.6
57.5
87.2
45.4
+2.8
46.5
+20.0
415.2
(Relative Gain)
(+10.0%)
(+8.9%)
(x11.296)
(+5.196)
(+7.5%)
(153.3%)
(x21.196)
Laama-2-7B
16.5
34.9
58.8
60.0
2.0
71.6
PS Prompting
12.0
4,7
27.6
61.2
61.4
1.0
7044
Vanilla
30.7
19.6
36.6
75.2
54,4
0.0
63.6
CODEPLAN
338
20.8
41.5
79.2
63.0
5.0
88.0
A
+3.1
+1.2
+4,9
+4.0
+8.6
+5.0
+16.4
(Reintive Gain)
(+10.1%)
(+6.19)
(+13.498)
(45.3%)
(+15.8%)
(NIA)
(+25.856)
Llama-2-131
30.2
85.5
1.0
Prompting
22.0
24.5
Vanilla
41.3
59.7
15.0
CoDEPLAN
49.5
27.4
53.4
86.4
100.0
23.5
88.0
A
+5.2
+38
+7.5
+0.9
440.3
48.5
+16.8
(Relative Gain)
(+11,7%)
(-16.1%)
(+16.1%)
(+1.15)
(+67.59)
(+56.79)
(+23.6%)
Instruction- Following
Multi-hop QA
Decision Decision-Makingg
Model
AlpacaEvall
MT-Bench
MuSiQue
HotpotQA
ALFWorkd
Mistral-7B
35.4
23.2
Prompting
Quiet-STaR
25.0
10.5
Vanilla
69.9
6.0
6.9
33.7
33.0
14.1
CODEPLAN
71.9
10.7
37.2
40.4
23.2
+4.7
41.8
47.4
(Relative Gain)
(+2.9%)
(478.355)
(426.19%)
(-10.45)
(+22.49)
(461.59%)
Llamn-2-7B
Prompting
10.6
10.1
16.0
12.1
+ CODEPLAN
65.1
6.2
6.2
27.4
27.4
14.1
A
+7.1
+2.4
+0.4
+2.4
+11.4
+2.0
(Relative Gain)
(.12.22)
(163.28%
(46.9%)
(69.6%)
(+71.396)
(x16.59))
lamm--2-3B
66.7
6.5
6.1
26.8
38,0
23.2
Prompting
$2.8
5.0
5.6
31.3
25.8
19.2
Vanilla
66.7
6.4
5.9
28.3
34,0
21.2
CODEPLAN
73.9
12.2
34.8
40.4
33.3
+7.2
+5.8
41.2
46.5
46.4
+12.1
(Relative Gain)
(+.10.8%)
(474.5%8)
(420.39))
(-23.09)
(+18.8%)
(457.195)
knowledge during training. The superiority may result from the inherent structured, concise, and unambiguous semantics encoded within code-form plans. In this way, LLMs can more effectively extract and internalize the underlying planning signals in natural language data.
3.6 ANALYSIS
We analyze two key benefits of CODEPLAN: (1) increasing superiority on more complex prob- lems (Finding and (2) improved training data efficiency (Finding 2). Furthermore, compare
7

---

Published as conference paper at ICLR 2025
CODEPLAN against two variants: planning in natural language (Finding 3) and reasoning through executable code (Finding 4), offering empirical evidence for the merits of leveraging code as inter- mediate plan representations. Appendix B also discusses the influence of plan annotation models, the efficiency of CODEPLAN, and case studies highlighting the strength and limitations of CODEPI.AN.
Nistral78
Vendii
EnceiPan
Finding I: CODEPLAN Yields Amplified Ben- efits for Increasingly Complex Reasoning Tasks. To gain deeper insights into the mer- its of CODEPLAN,
01.-
conduct experiments on
MusiQue, which encompasses questions span- ning various levels of complexity. As illustrated in Figure 2 CODEPLAN yields increasing per- formance gain as the task becomes more com- plex. The relative improvements in EM scores increase from 6.3% for 2-hop questions to re-
2.15
20.8
4-hoo
130 3-h00
4-hop
markable 43.8% for 4-hop questions.
Figure 2: EM (Left) and Fl (Right) scores on the MuSiQue benchmark. N N-hop means that the ques- tion requires reasoning steps to answer based on knowledge in Wikipedia passages.
This finding highlights key insight - as reason- ing challenges grow more intricate, the ability to generate and leverage structured code-form plans grows more valuable. For simple tasks, LLMs' inherent language understanding capabilities often suffice. But as task complexity increases, the limitation of vanilla training-thh the ambiguity and obfuscation of planning signals in natural language data becombs more clear. In such scenarios, CODEPLAN enables LLMs to systematically understand and frame the reasoning process, thereby navigating solution pathways more effectively.
(teoumey))
Finding CODEPLAN Improves Data Effi- ciency. We analyze the performance trajecto- ries of CODEPLAN and vanilla training. As de-
200
ViriEl
Verill
Crddef
Casteitan
picted in Figure S the LLM trained with CODE-
PLAN almost always outperforms its counter- part trained on the same raw prompt-response data, evincing superior knowledge acquisition and generalization to out-of- ofstiiibiitttiionn reason- ing challenges. Moreover, CODEPLAN achieves stable and consistent performance as- cent, maintaining its supremacy throughout the training process. This showcases the advantage of CODEPLAN in developing transferable high- level reasoning skills from wide-ranging data, as illustrated by the case in Appendix B.4
173-
47.
2.50
1.200
2200
$000
Taining Steps
Figure 3: Performance trajectories on two down- stream tasks via vanilla training and CODEPLAN, "4-Hop" denotes evaluating on the 4-hop subset.
Natural Language Han
Vart
sing
Coderan
Finding 3: CODEPLAN Outperforms Plan- ning in Natual Language. To investigate how code-form plans compare to natural language plans, we conduct comparative experiments by replacing the code plans in our dataset with nat- ural language counterpart Figure 4 shows the results. Training with natural language plans also leads to moderate improvement over the vanilla baseline. However, CODEPLAN consis- tently outperforms its natural language coun- terparts across all benchmarks by substantial margins. The performance gaps are particu- larly pronounced on complex reasoning tasks requiring structured planning. such as mathemat- ical reasoning. symbolic reasoning, instruction-
fallowing
Reacoringg
Dechton
Maidnn
Figure 4: Comparing natural language planning with CODEPLAN. The scores cach type of task averaged across all corresponding benchmarks.
We generate natural language plans using the pipeline in 3.1 with modified prompt Appendix A.3
8

---

Published as a conference paper at ICLR 2025
Table 4: The negative log-likelihood of generating responses in different plan forms. Stage 1/2 refers to planning/surface realization, respectively. "Overall" is calculated by summing up the two parts.
Method
Stage 1: (o'g(("
Stage 2: loggp(Y 2*)
Overall
Vanilla
0.
0.689
0.689
Natural Language Plan
0.351
0.337
0.688
CODEPLAN
0.237
0.347
0.580
following, and decision-making tasks, with relative improvements of 4%, 27.2%, 6.5%, and 27.5%, respectively. This validates the superiority of code representations over natural language.
Additionally, we are curious about why planning code outperforms natural language. We evaluate the performance of the two-stage process: planning and surface realization. For each plan form, Table reports the model's negative log-likelihood (NLL) on 10K subset of the training data. The vanilla baseline without explicit planning exhibits high overall NLL,, reflecting its difficulty in directly modeling the complex mapping from prompts final responses. While natural language planning substantially reduces NLL for surface realization, it introduces significant challenge in planning compared to CODEPLAN, We attribute this to two reasons: (1) code provides a more structured and precise representation of complex reasoning logic compared to natural language, thus offering more concise and easier-to- le-r plan labels. (2) by framing plan generation as code generation, LLMs can better leverage their pre- wee-training knowledge, code is far more prevalent in pre- pre-training corpora than natural language plans. Consequently, CODEPLAN achieves a more substantial overall NLLL improvement, providing empirical validation for theoretical framework in Eq. 4 that requires minimizing the combined difficulty of planning and surface realization.
Finding 4: CODEPLAN Outperforms Exe- cutable Code-form Reasoning. Prior work primarily focuses on using executable code di- rectly solve mathematical reasoning tasks (Gao al. 2023). However, we posit that this ap- proach is inherently limited when tackling broad, multi-step reasoning challenges that necessitate deep natural language understanding capabili- ties. To substantiate this claim, we conduct com- parative experiments by replacing the code plans in our dataset with executable code solutions. Specifically, instead of constructing pseudocode plans, we directly translate each response into executable code that can output the answer af- ter execution with code interpreter, using the prompt in Appendix A.3 We refer this base- line as CODEREASON.
CadeRtasoo
60-
SVAMP
MATH
MusiQue
Figure 5: Comparing CODERFASON (i.e., exe- cutable code-form reasoning) with CODEPLAN.
As presented in Figure 5 while CODEREASON achieves the highest performance on SVAMP, benchmark comprised of relatively simple math
word problems, it consistently underperforms CODEPLAN across a diverse array of reasoning benchmarks that demand extensive natural language understanding capabilities, such as MATH, MuSiQue, and HotpotQA. Notably, CODEREASON even lags behind the vanilla training baseline on MATH and MuSiQue. In contrast, CODEPLAN'S 's novel framework of generating code plans as an intermediate representation scamlessly integrates robust planning capabilities with the rich language understanding abilities of LLMs, yielding superior performance on intricate multi-step reasoning challenges.

---

Published as a conference paper at ICLR 2025
4 RELATED WORK
LLMs for Reasoning. Endowing LLMs with robust reasoning abilities remains a formidable challenge. Existing approaches predominantly fall into three categories: (1) Prompting techniques, which use expert-designed prompts to elicit reasoning skills without training (Wei al. 2022b]
Press et al. 2023; Imani et al. [202]] Hong et al., (024) (2) Task-specific fine-tuning, which curates
tailored -uning data or rewards improve reasoning specific tasks like mathematical reasoning
Yu al. 2023 Mitra et al. 2024 Shao et al.] 2024, OpenAl| 2024), code reasoning (Le et al. 2022
Shen et al. 2023), ). instruction ciaruuttio-followinnn following (Cui et al. 2023). visual reasoning (Cheng et al. 2022aa) ) and
decision-making (Zeng et al. 2023 Guan et al. (024) tasks. However, these approaches often falter
in generalizing beyond their intended tasks. (3) Tool integration, which seeks to augment LLMs'
reasoning capabilities by incorporating external tools like calculators (Schick et al. (024), retrievers (Asai et al.] 2024). and code interpreters (Gao et al. (023). Compared to these works, CODEPLAN focuses enhancing LLMs' high-level planning capabiliiiess in diverse domains, without relying on task-specific prompts, fine-tuning data, rewards, or tools.
Deliberate Planning in LLMs.
Enhancing LLMs' planning capabilities is crucial for complex reasoning tasks (Yang et al, (2023). Prior work primarily focuses on teaching LLMs to plan in natural language via task-specific prompts (Wang et al.) 2023 Khot et al.) 2022) curated fine-tuning data (Yin et al. 2024] Guan et al.] (024)). In contrast, CODEPLAN innovatively introduces code as a
structured and versatile plan representation. Recent works also attempt to learn implicit planning
(e.g., latent code or verbal words) from wide-ranging text corpora (Zelikman et al. 2024]] Cornille
al. 2024). However, these approaches may struggle to automatically unveil effective planning signals from the vast space and often introduce significant computation overhead during training
due to online sampling over prior and posterior distributions. In contrast, CODEPIANN introduces neglectable computation cost as illustrated in Appendix B..3 Additionally, recent works also explore multi-pati planning ( Yao et al. (024) and iterative plan refinement (Shinn et al. 2024), which are orthogonal to our work. We leave integrating CODEPLAN with such techniques for future work.
Code-aided Reasoning. Recent works have explored leveraging code to empower LLMs for complex reasoning. One approach directly employs prompting techniques (Gao et al. 2023 Ye et al., 2023) or curated fine-tuning data (Gou et al. 2023] Zhou et al., 2023) to generate executable code as a surrogate for natural language response, subsequently utilizing a code interpreter to derive the answer. Despite the precision afforded by executing code, this framework suffers from limited data scalability and is significantly limited narrow domains such mathematical calculation. Beyond direct problem-solving, code has also been utilized to enhance LLMS' capabilities in handling structured reasoning tasks, such as graph generation (Madaan et al. 2022). event structure prediction (Wang et al. 2022), and decision-making (Wang et al.) (024)) tasks. Sharing a similar motivation, our work leverages code to represent intricate reasoning structures.
CONCLUSION
In this work, we introduce a pioneering framework to endow LLMs with robust planning capabilities through the explicit supervision of code-form plans. By reframing plan generation code generation task, CODEPLAN harnesses the structured and versatile nature of code to capture the rich semantics and control flows underpinning sophisticated reasoning processes. Importantly, CODEPLAN allows the automatic extraction of code-form plans from massive, wide-ranging text corpora without the need for curated, task-specific datasets. We demonstrate the effectiveness of this framework by training CODEPLAN on large-scale dataset comprised of 2M natural language problems paired with their corresponding code-form plans. Across an extensive evaluation spanning 13 challenging multi-step reasoning benchmarks, CODEPLAN demonstrates remarkable efficacy, consistently and substantially outperforming vanilla training by a substantial margin. In-depth analyses further corroborate CODEPLAN'S increasing performance gain on complex problems and generalization ability. This work paves the way for several promising research directions, including exploring diverse posterior distributions over plans, enabling multi- path planning, facilitating plan verification, reflection and refinement, and realizing agents that can seamlessly leverage external knowledge sources and APIs within their planning and reasoning processes.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTS
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604), and he Tsinghua University Initiative Scientific Research Program. This work was also supported by the Ant Group Research Intern Program.
REFERENCES
Sotiris Anagnostidis and Jannis Bulian. How susceptible are LLMs to influence in prompts? In First Conference on Language Modeling, 2024. URL http:::///ppeeeeieew htt./eeww.ee.oo.oeeeeeeeeeeeee forttforum?? ?id=
y IDcIa
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, 2024. URL htllsttps: http::::///peeeeeww.... wwtpe/wwwww.eeeeeeeo.eeeeeeeeeee forum?
Id=
Samuel Bowman, Luke Vilnis, Oriol Vinyals. Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, 2016.
Robert Boyer and Strother Moore. mechanical proof of the Turing completeness of pure LISP. Citeseer, 1983.
Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.1416 14165, 2020.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. In The Twelfth International Conference on Learning Representations, 2024. URL
net wttpeeew.wwweeeteeeeeeeee wow.mmiddggV8330ee
Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. From the least to the most: Building plug-and- play visual reasoner via data synthesis. arXiv preprint arXiv:2406. 199044946.994, 2024a.
Daixuan Cheng, Yuxian Gu. Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. Instruction pre- training: Language models are supervised multitask learners. arXiv preprint 1xXi::2406.14491 2024b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1-53 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Nathan Cornille, Maric-Francine Moens, and Florian Mai. Learning to plan for language modeling from unlabeled data. arXiv preprint arXiv:2404.00614 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhaneing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:240.22178. 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Longth-controlled alpacaeval: A simple way debias automatic evaluators, arXiv preprint a444v:2404.0445, 2024.
11

---

Published as a conference paper at ICLR 2025
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-are aided language models. In International Conference on Machine Learning, pp. 10799, PMLR, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint arXiv:2309. 177522 2023.
Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, and Minlie Huang. Amor: A recipe for building adaptable modular knowledge agents through process feedback. arXiv preprint MxXiv22402.01466 2024.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https
//openreview.oo net fott://orrrmm2dd
Sirui Hong. Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference Learning Representations, 2024. httpt:tt////pettt.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. 37-42,744 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernan- dez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Mcasuring faithfulness in chain-of-thought reasoning. arXiv preprint wtXii:2307.13702. 2023.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-2133 2022.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Conference on Learning Representations, 2024. URL. htt.:/ttsss//opeereeee..... wtt.:r/wwwwweeeeeefooereomm
id=loi jJJRRST
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023.
Aman Maduan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code few-shot commonsense learners. arXiv preprint arXiv:2210.07122. 2022.
12

---

Published as a conference paper at ICLR 2025
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of sims in grade school math. arXiv preprint arXiv:2402.14830, 2024.
OpenAl.
Introducing openai op-prvvee. ol prrvviewwwweewwee-eoe--e--- 2024.
https://openas com/ cow/index/////oo/oooooooccc index
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 2021.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 5687-5711, 2023.
pytorch. Pytorch profiler, 2023. URL bttps///sykocchh.rr////utttt//ae
reci recipes/pree pates/profllerreeeee.rr pe. html
Ehud Reiter and Robert Dale. Building Natural Language Generation Systems. Cambridge University Press, 2000.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. Pearson, 2016.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to tools. Advances in Neural Information Processing Systems, 36, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14903 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wxXv:2207.00228. 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguisties, 10:539-554, 2022.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Kei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2609-26333 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.11663//1/2222.addd 147. URL httpa://acllatttooooo..oo/////// acl -ong
147
Xingyao Wang, Sha Li, and Heng Ji. Code4struct: Code generation for few-shot event structure prediction. arXiv preprint arXiv:2210.12810) 2022.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better IIm agents. arXiv preprint arXiv:2402.01030, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint 200::2206.07682, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouugh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022b.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Traveiplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, 2024. URL hhttps://openreviers net/
Qiang Yang. Intelligent planning: a decomposition and abstraction based approach. Springer Science
& Bussness Media, 2012.
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportumities. arXiv preprint arXiv:2303.04129, 2023.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-23800 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large language models are versatile decomposers: Decompose evidence and questions for bable- reasoning. arXiv preprint arXiv:2301. 13808, 2023.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12380-12403, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang. James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.1228.. 2023.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548, 2024.
14

---

Published as a conference paper at ICLR 2025
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 95.15476-15488 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.0792. 2023.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 TRAINING DATA CURATION DETAILS
The curation of high-quality training data crucial for the success of our approach. To ensure the integrity and relevance of the generated code-form plans, we filter examples without valid terminations in the plans, as this is essential for maintaining logical coherence. Figure Gillustrates the pipeline for curating the training data.
Pre-trainee on Large-Scale Code Data
plans volid terminations
<Prompt, Response>
<Prompt, Code Code-form form Plan, Response>
Training Data
Figure 6: Training data curation pipeline.
Furthermore, explored several fine- filtering mechanisms to refine dataset. One notable approach involved assessing the information gain of each plan Z by comparing the perplexity of generating Y with and without the plan (i.e., X(Y experiments revealed that these additional filtering steps yielded only marginal improvements in the performance LLMs.. In the interest of efficiency and to avoid potential overfitting, we opted omit these more complex filtering mechanisms in our final process. Appendix B.2 about plan quality.
P(Y (")). However, our pilot
B.2 shows more discussion
A.2 DATA STATISTICS
Table 5 and Table present the detailed statistics of our training data and evaluation benchmarks, respectively. For mathematical reasoning tasks, use the full test set of GSM8K and SVAMP, and a non-calculationninneene
subset of MATH, including "Prealgebra," " "Geometry." "Countin & Probability" since focus planning. For symbolic reasoning tasks, set of Boolean and Dyck Language, and use the official code to generate a challenging 4-hop test the
the official test
for Coin Flipping and Last Letter Concatenation. For instruction-follloinn tasks, official test set of A AlaacaEval and MT-Bench. For multi-hop QA tasks, we randomly sample 200 questions from the 2/3/4-hop subset of MuSiQue (600 total) and randomly sample 500 questions from HotpoiQA. For the decision-makki task, we use four task types, including "pick&place," "clean&place," "heat&place," and "cool& place."
Table 5: Statistics of the curated training data.
Aspects # Exammlee
Avg. Prompt Length
Avg. Plan Length
Avg. Response Length
Statistics
2,335,072
36.2
45.7
134.7
Table 6: Number of test examples in the evaluation benchmarks.
Sombelic
Instrnction- Alpacalval M.VZIMT-HHett
Multi-hop
DecisionMMsking ALt
SSSHK/AATHSVAAI
Linguage
MuSi(aa/Holpaau HolppaAA
1.319/13311.000
NW01800100000200
305/80//90
600/500
100
A.3 PROMPTS
Prompt for Annotating Natural Language Plans. We present the prompt for annotating the natural language plans for the original WebInstruct data of prompt-response pairs in Table 7
16

---

Published as a conference paper at ICLR 2025
Table 7: Instruction for generating the natural language plan for given prompt-response pair.
Prompt: {(Prompt))
Respense: [Response)
Given prompt-respont paic, your task describe high- logic of the response using natural language. Such that following this logic, madels can easily generate the response.
The logic should balance conaisenass and informat veneas. logic should be high-level, instead of replicating low-level detaills in the response. The logic should be less than 200 words (adjust its length based on response lengths).
Prompt for Annotating Executable Code for Reasoning. executable codes for the original training data in Table
We present the prompt for annotating
Table 8: Instruction for generating the executable code for given prompt prompt-response pair.
Prompt: [(Prompt)) }
Respense: [Respome)]
Given prompt-respon pair, your task is convert natural language respense into executable Python code that can print the same response.
The execution output should be consistent with the natural language answer.
B ADDITIONAL RESULTS
B. EXPERIMENTS LLAMA-3 MODELS
B.2 ASSESSING THE ROBUSTNESS OF CODEPLAN TO PLAN QUALITY DRIFT
To ensure the reliability and generalizability of our approach, it is crucial to assess the robustness of CODEPLAN with respect to various factors that could potentially influence its performance. Specifically, we focus on the choice of plan annotation models. This analysis aims to provide insights into the stability of our method across different experimental conditions.
We initially used Llama-3-8$ 3-----Inntttutt to construct code-form plans from natural language responses, balancing efficiency and performance. To investigate the impact of label quality CODEPLAN, conducted an ablation study using two additional models for plan annotation: Gemma-2B-instie and 1aaeaa--------tttttee (Dubey et al.) 2024).. These models represent spectrum of capabilities, allowing us to assess how variations in label quality affect overall performance.
Table 9: Performance of CODEPLAN on three representative types of reasoning tasks when using dif- ferent models for annotating code-form plans. Scores averaged across corresponding benchmarks for each task.
Annotation Model
2e-ssa----Insrruct
Mlama-3-B-IInseee
3lama-3-70B--nnse
Mathematical Reasoning
45.1
51.7
50.8
Multi-hop QA
29.9
38.8
37.7
Table presents the results of this comparative analysis. While there is substantial performance gap between Gemma-2B. Innstrctt and Samma-3--B-Instreen the difference between Llama-3-8B -anma------Insttuuee and the more powerful Llama-3-70B-Insse Instrru is relatively small across all reasoning tasks. This suggests that once the annotation model reaches sufficient level of competence, CODEPLAN's performance becomes largely insensitive to further improvements in plan quality, underscoring its stability and resilience. The robustness of CODEPLAN to variations in plan quality offers several key advantages. First, alleviates the need for employing the most computationally expensive and resource-intensive models for plan annotation, enabling more efficient data curation and training.
https:: https::/huggingffae wtw.aaccc.ccooooooeooeeeee e/ ggemmm 2b-it
17

---

Published as a conference paper at ICLR 2025
Table 10: Efficiency of CODEPIAN compared with vanilla training.
Model
Memory (GB)
Time (Second)
Vanilla
30G
0.055
CODEPLAN
30G
0.058
Table 11: One example for mathematical reasoning task (from the MATH benchmark).
PrempL:
There is number The number C multiple of multiple number C' greater this mumber C??
What
Plan and Respoase From CoD22LAN:
def ()
multiples_of.7 [i for i in ra1ge 30) i
multiples_of.5 [i for i range121, 30) %6
result = for in of if not in multiples.ot. return result [0]
The multiples of between 20 and 30 are 21 and 28. The multiples of between 20 and 30 are 21, 24, 27, and 30. Therefore, the number C' is The answer
Reaponse Vanilla:
The number C is multiplc of 7, but is NOT a multiple of 3. This means that C' is multiple of 21, but NOT multiple of 27. The only
number this description 21.
Second, it enhances the generalizability of our approach, as CODEPLAN can maintain its effectiveness even when the plan annotations deviate from perfection due to factors such as domain shift or annotation noise.
B.3 EFFICIENCY ANALYSIS
For efficiency evaluation, Table 10 reports the memory usage, and average per-example inference time CODEPLAN and the vamilaa training baseline. During evaluating, we set the batch size to and use A100 GPU. We calculate memory consumption using PyTorch toolkits pytorch (2023). The per-example inference time averaged over 1000 generations.
The results demonstrate the comparable computational efficiency of CODEPLAN with vanilla training, with only negligible increase in inference memory usage. Moreover, the average inference time of CODEPLAN is only marginally higher (~5%) than that of vanilla training. These efficiency metrics highlight that CODEPLAN does not sacrifice much computation overhead to get substantial performance improvements, since pseudocode concisely outline the reasoning structures.
B.4
CASE STUDY
To provide deeper insights into how CODEPLAN enhances LLM reasoning capabilities, we present detailed case study. This analysis involves the testing benchmarks used in the main paper, the reusability of functions in the code-form plans to explain the high data efficiency of CODEPLAN, as well as the strong performance of CODEPLAN for two popular tests, including Value Comparison and Letter Counting, thus comprehensively illustrating the concrete benefits of our code- based planning approach across diverse reasoning challenges.
Cases for Benchmarks. respectively. Note that the example for the instruction-follovii task has been shown Figure the main paper. As exemplified, the code-form plan clearly outlines the step-by-step reasoning process:
present one example for each type of task in Table II
12
Mathematical Reasoning (Table II):
This example requires identifying a number that satisfies multiple constraints. The vanilla pre-trained model fails to correctly incorporate all conditions, yielding an incorrect answer. In contrast, CODEPLAN'S code plan methodically enumerates the relevant number ranges, applies the given criteria through logical operations,
18

---

Published as a conference paper at ICLR 2025
Table 12: One example for the symbolic reasoning task (from the Last Letter Concatenation bench- mark).
Prempt:
letters words in Aliza Betzy Rohan" concatenate
Reaponsa from CoRRPRAR:
def exiraci_ande wttpectanndccocceeeeeeeeeee concatenate Jast. letters penieessiingn (imput_stry str):
words input.str. split()
concatenated.resull
for word words:
letter word[-1] concatenated.resut
last.letter
concatenated result
iaput_sir- "Ole Aliza Betzy Rohan"
result www.nc..ndd.oncattnntte....
(imput.str)
Current word "Ofe", last letter is "e", concatenated resalt is "c", Current word is "Aliza", last letter "a", concatenated result is "ea", Current word "Betzy". lotter "y", concatenated unswer is eayn.
"cay". Current word "Roban", letter "n". concatenated result is "cayn".
Raspense From Vanilla:
Current word "Ofe", last istter concatenated resalt "e", Current word is "Aliza", last letter "a", concatenated resalt is "ea", Current word is "Betzy", last letter is "y", concatenated result is "ey". Current word is "Roban", , last letter is "n", concatenated result is "cya". The cacya.
Table 13: One example for the multi-hop QA task (from the MuSiQue benchmark).
Prompt:
The bcautiful country used as backdrop for the movic The Lord of the Rings was founded on what date?
and Reaponse from CORTPLAN:
def get get_answer
movic query (" "What movie used as backdrop") = "The Lord of the Rings" country query (" ("What country used backdrap") "New Zealand" event query ( What event is related to the country" ") "Treaty of Waitangi"
date query "What is the date of the event") "6 Fehruary 1840" answer = date return answer
The movie The Lord of the Rings was lilmed in New Zealund. The Treaty of Waitangi was signed in New Zealand on 6 February 1840. The answer is February 1840.
Reaponse Vanilla: be movic The Lord of the Rings was released in 2001. The country was founded in 1840. The answer is 1840.
and precisely identifies the correct solution. The code naturally captures the step-by-step reasoning process, breaking down the complex problem into interpretable sub-tasks.
Symbolic Reasoning (Table 12): Here, the task involves concatenating last letters of words in given string. While the vanilla model makes mistake in tracking the concatenation order, CODEPLAN'S code plan clearly delineates the iterative process of extracting each word's last letter and appending the result string. The structured nature of the code ensures precise execution of the required operations, leading to the correct solution.
Multi-hop QA (Table 13): Answering this question requires reasoning over multiple pieces of information and making implicit connections. The vanilla model struggles to synthesize the relevant facts, providing an incorrect response. However, CODEPLAN's code plan explicitly outlines the necessary steps: identifying the key entities mentioned, querying relevant information about each, and logically combining the retrieved knowledge to derive the final answer. This structured approach enables CODEPLAN navigate the multi-hop reasoning process successfully.
Decision-Making (Table In this task set in a simulated environment, the agent must execute a sequence of actions to accomplish the given goal. While the vanilla-trained LM struggles maintain coherent plan of action, CODEPLAN'S code-form plan clearly specifies the high-level steps required, such as locating and obtaining the target object, cooling it, and placing it in the designated location. This explicit planning stage allows CODEPLAN to systematically break down the complex task and execute the necessary actions to achieve the desired outcome.
19

---

Published as a conference paper at ICLR 2025
Table 14: One example for the decision-making task (from the ALFWorld benchmark).
Prompt:
Cool mug and pat coffeemachine
Plan and Response from COD3P1AR:
def wwt.lmmttui.n......m.. www.in_coffemachine))
for receptable in Iccept.to.cheek:
observation (ete(recellable)
if "clased" in observation:
observation (pen(psceptablee)
if "muy" observation:
found mug = take ("kgg","g", receptable) goto ("fridge")
cool ( (found "fridge")
gato (" "oofeemmchine""
put (found_mug, "coffeemachine""
think: To solve task, need to take mug, then cool it with fridge, then pur coffeemachine.
Obs: OK.
Act: think: First need to lind a mag. A mug more likely to appear in drawer (1-5), cabinet (1-16), countertop (1-3), sinkbasin (1), stoveburner
(1-4), fridge toaster (I), shelf (1-3). I can check one one, starting with drawer
Obs: OK.
go
Obs: The drawer is closed.
Act: go countertop 1
Obs: On the counteriop 1, you see upple I, creditcard 2, mug and spatula 1.
Act: think: Now find mug 2, Next, need take
Obs: OK.
mug from countertop
Obs: You pick up the mug from countertop
Act: think: Now take a mug Next, need to go fridge and cool
Obs: OK.
Act: go to fridge
Obs: The fridge closed.
Act: cool mug? with fridge
mug using fridge
Act: think: Now I cool a mug (I), Next, I need to put in/on coffeemachine 1.
Obs: OK.
Act: go colfeemachine
Obs: On the cuffzemachine I, you see nothing.
put mug in/on coffeemachine
Obs: You pat the mug in/on the coffeemachine 1.
Response Erom Vanilla ila:
Act: think: To solve task, need find and take cool mug, then coffeemachine.
Obs: OK,
think: First nood find cool mug. cool mug likely appear fridge countertop (I-3), cabinet (I-4), drawer (1-3). Cean check one by one, starting with fridge 1. OK.
Act: take mug 2 from countertop Obs: You pick up the mug 2 from counteriup
Act: think: Now find cool muy Next, I need go coffeemachine
Obs: OK.
go coffeemachine
Obs: Nothing hagpens.
put mug coffeemachine
Obs: Nothing happens.
Act: think: Now put a cool mug (2). Nexi, need turn on coffeenacchine
Obs: OK.
Act: turn on coffeemachine
Nothing happens.
Act: think: Now I turn on coffeemachine (1). Nexi, I need to wait until it is done.
Act: think: Now wait until coffeemachine (1) done. Next, need to take a mug (2).
Obs: OK.
put mug (2) infon dishwasher (1). Next, need turn dishwasher (1). OK.
Through these diverse examples, we observe that CODEPLAN's code-based plans consistently provide clear, structured representations of the reasoning process across various task types. The formal semantics of code enable the explicit encoding of intricate logic, control flow, and hierarchical task decompositions, This explicit planning stage allows the model to methodically navigate complex problems, leading to more accurate, reliable, and interpretable solutions compared to direct response generation without structured planning.
20

---

Published as a conference paper at ICLR 2025
Table 15: Two examples from the training data of CODEPLAN, where different questions be solved through similar code-form plans.
Prompt 1:
Prompt 2:
Which numbers the following set are prime numbers: (4,7.31, 117, 324)?
is the 10th peime number greater than they are listed in order?
Plan and Response for Prompt 1:
Plan and Resperse For Frompe 2:
is-prime (n):
find 10th.prime.e. greateggreetaaaaee 50 primes []
if <= 1:
return False
num 53
for i in (2, int(n 0.5)
while (
10:
n mun False
O:
"s.prime (num):
primes append (num)
return True find prime (umbers(numbers)) prime_numbers
DOM += return primesl-11
def (s.pprme(()::
for num in numbers:
if <= 1:
is-prime (num):
False
prime_ anp.ee.semeeseeeeddddd
for in range (2, int(n * 0.5) 1):
primernumbers
numbers = [4. 7. 31. 117. 324]
return Falsc
prime
prime
(numbers)
print (prime numberrs)
The prime numbers greater tham 50 are: 53, 59, 61, 67, 71. 73, 89, 97, 10th prime number greater
prime number divisible by and itself. From the given set, and 324 are not prime numbers because they
number greater
only
117 is not a prime number because can be written as 117 3239 3rdrr13 s1,. which than and itself. Therefore, only prime numbers the set 7
other
Cases for Reusable Functions. One of the key advantages of CODEPLAN 's code-form planning is the ability to reuse functions across different instances. By abstracting certain common operations into reusable code blocks, CODEPLAN can achieve higher data efficiency. For instance, when tackling tasks involving mathematical reasoning, frequently used operations can be encapsulated in distinct functions that reusable across multiple examples, illustrated in Table 15
Cases for "Value Comparison" and "Letter Counting." Recent studies showed that value comparison and letter counting are difficult even for GPT-4o. The former requires determining the larger one of two digits (e.g.. comparing 9.8 and 9.11), while the latter requires counting the number of occurrences of a specific letter within word, such as counting the number of "r"s in "strawberry." LLMs often falter when asked to handle these seemingly simple tasks because they lack a structured process for dealing with multi-part numerical representations or iterating through the word and applying conditions.
In contrast, CODEPLAN 's code-form plan effectively disambiguates the tasks by explicitly defining how to compare values and count letters, shown in Table [6] For example, when comparing 9.8 and 9.11, CODEPLAN generates a code-form plan that systematically parses and compares each part of the numbers, ensuring that the comparison logic is clear and correct. If asked count the number of "Y"s in "strawberry." CODEPLAN produces a pseudocode plan that explicitly tracks each letter and increments the count when the specified letter is found.
B.5 ERROR ANALYSIS
While CODEPLAN demonstrates significant improvements in reasoning capabilities, it is crucial conduct a thorough eиror analysis to identify areas for future research and refinement. We categorize and examine three primary types of errors observed in the code plan generation and exccution process:
Subtask Execution Error (Table 17): This error occurs when the model correctly generates appropriate high-level plan but fails to execute individual subtasks accurately. For example, in a multi-ster math problem, the plan might correctly outline the sequence of operations, but arithmetic error in step leads to incorrect final answer. Table 17 presents another error case from the Last Letter Concatenation benchmark. Here, the plan is correct, but error in the reasoning path leads incorrect final result. This type of
21

---

Published as a conference paper at ICLR 2025
Table 16: Examples from OpenAT's ol ol-preview with and without code-form planning for the "Value Comparison" task and "Letter Counting" task.
Prompt for " VValue Compar Comparison"". ssn"' 9.8 and which is larger?
Prr Prompt for "Tetter Counting"rs
how many is "strawbeny"
Besponse
Sode
Plenning:
Response Code- form Planning:
def compare():
def get_frequene( for in "strawberry": c="r": +
delta if delta 0. answer '9.8 larger'
9.11
else:
answer 19.1 11 is larger
return ent
della 0.69. Since 0.69 0,9.8 larger.
There are "Y"s in the word "stawherry"
Response thous Planning: 9.11 is larger.
Responsa without Planning:
The word "strawbery" contains two letter letter
Here's the breakdown:
STR A B F.R.R The "Y"s located in the third and sighth positions.
Table 17: One example from the Last Letter Concatenation benchmark, which exhibits the subtask execution error highlighted in red.
PrompL:
Take the last letters of the words in "Nguyec5u Danish Aminta Javon" and concatenate them.
Plan and Response Irom CoDsPLAN:
def extract.and.coneattos
letters stoput..t)t:
words input.str. (plit()
concatenated_resuul
for word in words:
last.letter word[-1]
concatenated result += last letter
return concatenated resull
input.str "Ofe Aliza Betzy Rohan"
result
(input.xtr)
Current word "Nguyec5n",, last letter "s", comcatenated result "n". Current word is "Danish", last letter concatenated resull "ni". Current word "Aminta", last letter is "a", concatenated result is "nia". Current word "Javon", last letter is "n", concatenated result is "nian". The
Table 18: One example from the AlpacaEval benchmark, which exhibits the plan following error highlighted in red.
Prempt:
Please read the parugraph below and count many times the wards "Amazzon" "river", and "you" appear. Please present the results in format of "word, number of appearances" with each word seporate line. Sort the lines order of the number
of appcarances.
The Amazon, mesmerizing
of naturc's wonders, is home to the logendary Amazon River. Flowing through awe-inspiring landscapes like the Amazon rainforest, river weaves its way through Brazil, Colombia, and Peru, giving life to countless creatures. From the mighty prowling Amazon jungle vibrant macaws above the canopy, this remarkable
with biodiversity...
Plan and Response from CoD2BLAB:
count_words (ext):
words [" ["Amazon", "river", "yau"]
results ()
for word in words:
for results word =
text.s)lit(".");
for word in words:
sentence:
results [word] +=
sorted_words sorted (results.. items (), key=lambda x: x[I], reverse=True)
return "Aa". join (I1" word [count]" for word, count in sorted words
Amazon,
13
you,
eтror suggests that while CODEPLAN enhances high-level reasoning, there still room for improvement in the performance of low-level reasoning steps.
22

---

Published as a conference paper at ICLR 2025
Table 19: One Example from the MATH benchmark, which exhibits the plan error.
Prompt:
February 1. 2008 is Friday. The last Friday of February 2008 will be February 2008. If 2008 leap year, what the value of x=?
Plan and Response from CAD3PIAR;
def fatd.last. of..uuuuuarrff..oooeererrra.a... (year):
if is.leap-year (year))
days.in_ else:
days.in.february
frida days-in.february feeyuuan.february
return last friday
def is.leap-year(year))
return year & == 0 and (year % 100 t= or year %6 400 == 0)
year
hwddllat.friday.ffffebuuuu
(year)
(x)nt(
Since 2008 leap year. February has 29 days. Since is Friday, 29%7 28 Friday, Thus, 28. The answer is 28
This error type reveals potential disconnects between the planning and realization stages. In Table the model generates correct plan but deviates from it during the execution phase, omitting crucial steps. Addressing this error type could involve strengthening the coupling between planning and execution phases during training.
Plan Following Error (Table 18):
Plan Error (Table 19): The most fundamental type of error occurs when the generated plan itself is flawed or incomplete. This indicates limitations in the model's ability to formulate comprehensive strategies for complex problems. Consider the example in Ta- ble 19 from a mathematical reasoning task. The generated plan incorrectly calculates "days february 7", which should be "(days in february -1 % 7". This type of error suggests that further refinement of the planning mechanism is necessary, particularly for tasks requiring nuanced multi-step reasoning.
Our analysis reveals that while CODEPLAN significantly enhances reasoning capabilities, there remain opportunities for improvement across various aspects of the planning and execution process. The subtask execution errors highlight the need for enhanced numerical precision and robustness in low-level computations. Plan following errors suggest potential benefits from stronger integration between the planning and realization stages during training. Finally, plan errors underscore the importance of further refining the model's ability generate comprehensive and nuanced strategies for complex reasoning tasks.
23


Reviews:
[{"review_id": "FFInPNkj4N", "reviewer": "Reviewer_6LBD", "summary": "This paper proposed to train LLMs to write code-form plans before performing reasoning on downstream tasks. The code-form plans are pseudocode written by Llama3-8B, and such augmentation is conducted on the 2M WebInstruct data (from Mammoth2 paper). They trained multiple models (mistral/llama2-7b/llama2-13b) on these data and observed large performance improvements over vanilla fine-tuning and training with language-form plans. Such performance gain is observed on 13 benchmarks including math reasoning, symbolic reasoning, instruction following, and multi-hop QA.", "questions": "Most tables and figures in the paper are hard to recognize. I strongly suggest increasing the font size of these tables and figures, including Figure 2,3,4, and Table 3. For Table 3, do not separate the table into two parts. You don't need to report multiple metrics for multi-hop QA. Use abbreviations for dataset names.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors proposed a novel idea (CodePlan): training LLMs to write code-based plans before conducting natural language reasoning. This idea utilizes the conciseness and structure of code to form high-level plans, and can benefit from LLMs' strong coding knowledge learned from pre-training.\n2. The authors built a large-scale (2M) fine-tuning dataset with code-form plans based on WebInstruct. This resource could be valuable for future research.\n3. The authors conducted solid experiments to prove CodePlan's supriority over vanilla fine-tuning and training with natural language plans. The benchmarks are tested out-of-distribution (no task-specific fine-tuning), which proves the generalization of the model.", "weaknesses": "I didn't find any critical weakness that would lead to rejecting the paper. However, the paper can be improved from the following perspectives:\n\n1. The training data can be further improved. The current data is composed of two components: a pseudocode plan and the original natural language answer. I have the following concerns:\n\n(1) The original answer is not further modified to better align and be more coherent with the pseudocode plan. This could limit the effectiveness of the training process.\n\n(2) Two components seem separated and are not sufficiently integrated together as a coherent response, which does not look like a real human-style thinking procedure. As a result, models are still adapted to this specific response format instead of a universal reasoning strategy like CoT or the think-and-reason style of GPT-o1.\n\n(3) The authors mentioned in the analyses that code-based reasoning are more effective than CodePlan on some tasks. So it may be worth exploring a combination of these two strategies. For example, there are papers that combined reasoning and code writing [1,2], which can be used to integrate with CodePlan for a more universal solution for model reasoning.\n\n2. The authors did not conduct experiments on recent open-source models. The tested models (Mistral/llama-2) are released more than a year ago and are known for limited reasoning abilities. For example, fine-tuned llama-2 is still <50% on GSM8k while the more recent llama-3 is ~80%.\n3. For presentation: the language-form planning baseline should be moved to the main table for a more comprehensive comparison, as I think this is an important baseline to show the effectiveness of CodePlan. Vanilla fine-tuning is a weak baseline because additional data are augmented into CodePlan's fine-tuning.\n\n[1] MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning\\\n[2] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eG2wTcr5IN", "reviewer": "Reviewer_Jy2Q", "summary": "This paper describes a way to improve LLM reasoning ability by training it to plan in the form of code. Specifically, it first prompts LLM to generate code form plan given existing prompts and responses. Then, it finetunes LLM to generate plan given prompt, and generate response given plan and prompt. Experimental results show performance improvements across over the baselines on multiple benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written\n- The experiments are extensive, over multiple types of models and benchmarks", "weaknesses": "- The main novelty of this paper is that it uses code form rationale/planning instead of free text form. Besides code form, the specific way of training LLM to predict a prompted rationale is already proposed by the paper STAR https://arxiv.org/pdf/2203.14465 (see the Rationalization section). This is not clear from the paper.\n- Given that the main novelty is code form plan, its effectiveness over a free text form plan is not demonstrated from the experiments. The authors should include a comparison against STAR, and an ablation study on replacing code form plan with text form plan with the exact same pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HdYfbwneW", "reviewer": "Reviewer_5Vzm", "summary": "This paper proposes an approach to improve LLM performance on varied reasoning tasks including mathematical and symbolic reasoning and multi-hop QA. The approach fine-tunes an LLM to predict Python pseudo-code \"plans\" from a natural language query, and to then condition on this pseudo-code to generate the answer to the query (the code is not executed). To obtain the pseudo-code plans for training, an LLM is prompted using queries and ground-truth solutions on 2M query/response pairs from WebInstruct. The resulting query, pseudo-code, solution tuples are then used to fine-tune the LLM. The paper evaluates on three different base LMs, and a variety of datasets including GSM8K, MATH, SVAMP, and HotpotQA, finding substantial improvements over direct prediction of answers and an variant of the method that uses natural language plans rather than pseudo-code.", "questions": "Q1) Did the authors train Quiet-STaR to get the numbers in Table 3, or the results are from the model from the original paper? I found it strange that Quiet-STaR is worse on nearly every metric and dataset compared to the Mistral-7B version, is there any explanation for this?\n\nQ2) It wasn't clear to me how CodeReason works: is the code actually executed, or does the prompt just encourage the model to produce executable code (but otherwise it works the same as CodePlan)?\n\nQ3) \"models are instructed to generate CoT-style responses\". Are these CoT also included in the training data (if not, seems there is a mismatch between training and inference)?\n\nQ4) do the few-shot examples also show Code Plan examples?\n\nQ5) For the case studies in appendix B.3, are these code plans generated by the model itself (from only the query), or are the annotated code plans used in training (from the query and answer)?\n\n*Minor Questions / Suggestions (no need to respond in the author response)*\n\n- \"delicate balance\" part in 2.1 is a bit vague\n\n- Figure 1: the digit example makes sense, but the renewable energy didn't to me. Where do the digits come from? There is some fact-based reasoning in the \"response\" text for *why* solar energy has limited accessibility, and it's not clear to me why it would be easier for the model to predict the numeric scores (needed to produce the code plan) rather than the facts. Is the main advantage of code here just to make sure all energy sources are mentioned, without repetition, and contrasted with each other?\n\n- Many of the attributes in Table 1 are subjective, e.g. what makes this work efficient but QuietSTaR not? Why is CoT interpretable but QuietSTaR is not?\n\n- Is \"NLL\" the average token-level NLL?\n\n- \"code is far more prevalent in pre-training corpora than natural language plans\". This is an interesting claim, but needs evidence. \"natural language plan\" is probably harder to define and so also probably hard to quantify?\n\n- Why is a 13B model necessary (the reason given in footnote 1 for why Llama-3 was not used)? Mistral-7B was used, and there is a ~7B Llama-3 model.\n\n- \"we synthesize 5K examples based on official implementation\" implementation of what?\n\n- The Kingma 2013 citation should probably be Kingma and Welling.\n\n- There are a number of misspellings throughout the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1) The paper's experimentation was overall thorough:\n- It applied its training procedure to three different open source models, and evaluated them across a range of datasets covering various types of reasoning where code might potentially help.\n- The analysis experiments, in particular Finding 3 that used natural language plans, were a helpful step toward understanding the approach (but see below for a few questions/points on this).\n\nS2) The results seemed strong, finding that the approach obtains large improvements over vanilla (non-planning) training and using natural language plans. \n\nS3) I found the paper well-motivated, given the findings of past papers that training on code may improve reasoning abilities [1,2,3], and that models can be trained to reason (in natural language) using approaches like STaR.\n\n[1] The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code. Liu et al. 2023\n[2] To Code, or Not To Code? Exploring Impact of Code in Pre-training. Aryabumi et al. 2024. \n[3] If LLM Is the Wizard, Then Code Is the Wand. Yang et al. 2024.", "weaknesses": "W1) While the paper took steps toward explaining it (via the NLL experiments in Table 4), I still found it a bit underanalyzed *why* using pseudocode plans works better than natural language plans. It would be useful to also investigate whether the pseudocode plans are also more accurate or otherwise higher in quality semantically than natural language plans, perhaps doing some human evaluation on a small number of samples.\n\nW2) The experimental setup left a few stones unturned which would be natural to investigate:\n- The model used to produce the code plans is Llama3-8B-Instruct, which is different (and potentially stronger) than any of the models fine-tuned on the plans (Mistral-7B, Llama-2-7B and 13B), so there's an element of model distillation in the paper. It would be helpful to evaluate whether a model could bootstrap itself as in the STaR paper, or at least to add a note on this.\n- The evaluation could be more rigorous in places, see questions and suggestions below.\n\nW3) While I think this could definitely be fixed for a camera-ready version of the paper, the writing of the paper could be a bit improved:\n- Make it clearer that in no cases (if I understand right) are the code plans being executed -- they are just conditioned on by the LLM. It would help to call them \"pseudo-code\" consistently throughout the paper (rather than just in a few places as it is currently).\n- Make the comparison clearer to STaR (as the method is similar) and Quiet-STaR (as their results are reported). The \"planning in natural language\" baseline in Finding 3 has some similarities to STaR (training the model to produce latent natural language plans) but also some key differences (all annotations are produced using query and answer; a different model is used to annotate than is being trained).\n- I didn't feel that Table 1, with the comparison between CodePlan and other methods, added much to the paper as many of the attributes are unexplained.\n- Clarify some details -- see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NxT4zvJ9Ti", "reviewer": "Reviewer_guz9", "summary": "This paper proposes incorporating pseudocode-based reasoning into the model's response process, which can effectively improve model performance. To obtain these pseudocode reasoning processes, the authors suggest using a large model to expand the responses directly on the WebInstruct dataset. Through extensive experiments, the authors demonstrate that training the model on the expanded dataset significantly enhances its performance.", "questions": "1. The paper used LLaMA3 8B for generation—is this a type of distillation? It would be necessary to report LLaMA3 8B's performance on the evaluation datasets directly. Additionally, given that the size of LLaMA3 8B is quite similar to the model discussed in the paper, why not simply use LLaMA3 8B directly?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The experiments are extensive.  \n2. A mathematical understanding of the underlying principles was developed, making the entire framework more solid.", "weaknesses": "1.  If the goal is to prove the value of the code-form plan, I think there are two aspects to consider:  \n1.1 If the goal is to demonstrate the benefit of this reasoning format to the model, then it would be appropriate to compare it with Chain-of-Thought (CoT) reasoning. Showing that a model performs better when using its code-based planning compared to traditional CoT results would more directly establish the superiority of the CodePlan format.    \n1.2 If the aim is to showcase the advantages of the code-form plan format and its ability to be automatically generated at scale with strong results, then the baseline of natural language enhancement should be more rigorously established. However, in the paper, the adaptation of natural language reasoning prompt (in Table 7) seems only slightly modified from the code-plan one, making the comparison less convincing.  \n[More comments on this prompt issue, natural language tends to be more redundant than code. Imposing a requirement like “The logic should be high-level, instead of replicating low-level details in the response” inherently makes it more challenging for the model. Additionally, as shown in the results of Table 4, modeling \\( \\log P(Z|X) \\) is challenging, which partly suggests that the difficulty lies in generating this logic itself. This doesn't necessarily indicate that CodePlan is more effective; rather, it suggests that generating code-based plans may be easier because the task is easier. This easiness could be due to the prompts being optimized for generating code rather than for producing natural language explanations.]\n\n\n2. The baseline results are different from Results in MAmmoTH2 (https://arxiv.org/pdf/2405.03548) . In Table 2 from MAmmoTH2, Mistral with WebInstruct, achieves 68.4/36.7 in GSM8k\\MATH. While in this paper, Table 3 shows 54.1/31.5, which are much lower than reported.\n\n\n3. Since the data generation process used LLaMA3 8B, the Plan-and-Solve (PS) Prompting baseline should also use LLaMA3 8B to generate the natural language plan prompt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PespMfGyil", "reviewer": "Reviewer_k1j8", "summary": "This paper introduces CODEPLAN, which formulates planning for complex problems in the form of code. To enable the model to acquire such ability, the authors have existing LLMs to generate this code-style planning for existing samples as training data. The authors have conducted experiments on various reasoning tasks and have achieved improvements.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper demonstrates performance improvements over baselines across different datasets.\n- The authors have validated the effectiveness of using code as a form of planning.\n- The authors have validated the effectiveness of using existing LLMs to generate code-style planning.", "weaknesses": "1. The font size in Table 1 is too small.\n2. Overall, I believe this paper would be more suitable as a technical report or a short article. The idea of using code form for planning is a useful trick, but it lacks enough innovation:\n - It only addresses the issue of the **form** of planning, without fundamentally enhancing the model's planning and reasoning capabilities\n - A widely held assumption in the community is that code is important for LLMs to learn complex reasoning and long-text abilities. There is already an amount of work on this topic. Therefore, the techniques and conclusions of this paper are not surprising. \n - Under code-style planning, the core issue is how to generate effective code-style planning for a query from a specific domain or a general domain. However, this paper does not actually address this core issue. The method of generating through prompting shown in Table 2 is overly simplistic.\n3. This method is constrained by the effectiveness of the LLM which generates code plannings for training. A common approach is to improve the quality of the training data through code execution results or log information. I suggest the author to consider this optimization method.\n4. Since the paper posits that code-form plans are superior to other forms (such as text, first-order logic, graph, etc.), the authors should provide direct comparative experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)