Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
Authors: Jin Zhang
Flood Sung
Zhilin Yang
Yang Gao
Chongjie Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
self-training
high-level abstraction
self-reflection
meta learning
anticipatory plans
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel self-training algorithm that self-teaches LLMs to generate high-level abstract plans before solving problems.

Abstract: In the field of large language model (LLM) post-training, the effectiveness of utilizing synthetic data generated by the LLM itself has been well-presented. However, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve as abstract meta-knowledge for problem-solving, before engaging with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction of irrelevant details. During data generation, LEPA first crafts an anticipatory plan based on the problem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming to acquire plans that are instrumental in yielding correct solutions. During model optimization, the LLM is trained to predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates remarkable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.

Published as a conference paper at ICLR 2025
LEARNING TO PLAN BEFORE ANSWERING: SELF- TEACHING LLMs TO LEARN ABSTRACT PLANS FOR PROBLEM SOLVING
Jin Zhang Flood Sung", Zhilin Yang", Yang Gao', Chongjie Zhang?
Institute for Interdisciplinary Information Sciences, Tsinghua University, China Moonnhot Al
Washington University in St. Louis
jin-z sin-zhan20@maile tsinghua. edu. on
ABSTRACT
In the field of large language model (LLM) post-training, the effectiveness of uti- lizing synthetic data generated by the LLM itself has been well-presented. How- ever, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve abstract meta- knwwledge for problem-solving. before engag- ing with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction irrelevant details. During data generation, LEPA first crafts anticipatory plan based the prob- lem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming acquire plans that in- strumental in yielding correct solutions. During model optimization, the LLM is trained predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates re- markable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.
INTRODUCTION
Large Language Models (L.LMs) have revolutionized the field of natural language processing, demonstrating remarkable capabilities in handling complex language tasks (Achiam al., 2023
Zhao et al. 2023] Yang et al. 20244] Shahriar et al., 2024). While post-training optimization of
LLMs demands substantial volume of data (Xiao et al. 2023 Waan et al.) 2024b), recent works reveal that LLMs obtain the potential of generating high-quality synthetic data themselves (Zelik man et al. 2022, Gulcehre al. 2023 Singh al., 2023) Bansal et al. (2024). These works, known as self-training methods, improve the LLM by iterating between generating data with LLMSs and op- timizing L.LMs with the generated data. Self-training methods alleviate the requirement of expensive human annotations and make post-training much more scalable.
A central challenge in self-training is, what essential information should such self-generater syn- thetic data encapsulate? Despite remarkable progress, this problem has not been well studied. Pre- vious works only generate step-by-ster problem solutions, and train the LLM to maximize the log- likelihood of generating these solutions (Zelikman et al. 2022 Singh et al., 2023). This approach only trains the LLM to memorize knowledge about task specific solutions, and fails to capture the high-level abstract meta- ketallnnoweegge necessary for generalization across similar problems. As

---

Published as a conference paper at ICLR 2025
sequence, previous self- toff-training methods obtain only limited generalization abilities, and struggle on difficult natural language tasks such as Hendrycks MATH (Hendrycks et al. 202I
(a) Hendrycks Math
[c) Proposed Method (LEPA)
Question:
Plan:
There are ten meenkats colony, Every night, two meerkats stand guard while the others sleep, During a certain period of nights, meerkat stands guard with other meeriat exactly once. During that period, many nights sleap does each meerkat ger?
When soleing combinatorial mathematiss problems, consider following steps:
1. Identiff the number of elements and the number of elements that are chosen at time.
Determine the total number combinations.
each element. 4. Calculate the total number of nights of sleep for each meerkat by subtracting the
Calculate number of combinations
number combinations for each element from the total number combinations
Answer:
(b) Previous Method (BeST)
Provide solution the required format, including the cornect answer.
solve problam step by step.
There are 10 meerkats, and every night, 2 meerkats stand guard, This
Answer:
that total number of nights of guard duty is 10C2, which is number ways to choose meerkats out 30.
Step Identily the number elements and number elements that chosen at time.
The problem states that there are meerkats, and every nighti meerkats stand guard. means that meerkats are chosen at time.
10C2 101 (2) [10-21]]- /01/ stj=
BI )
"8lj=44 45
Step Determine the total number of combinations.
So, the total number nights guard duty 45,
since meerkats are chosen at time, and there are meerkats, total number combinations is given by the combination formula. C(10, 2)-
Since each meeriat stands guard with every other meerkat esectly each meerkat stands guard for 45/2 22.5 nights
Calculate number of combinations
since every meerkat stands guard with every other meerkat exactly once, each meerkat
However, this not a whole number of nights, so we need to find the number of rights each meerkat gets. Since each meerket stands quard with every other meerkat, each meerkat
of combinations (since they stand guard with 9 other meerkats].
Step Calculate the total number of nights of sleep for each meerkat.
stands guard with other meerkats. This means that each meerkat stands guard for nights.
The total number of nights is the total number of combinations (45) minus the number of combinations for each meerkat (9), gives
Answer: g
Answer: 36
Figure I: A didactic example demonstrating how LEPA outperforms baseline methods by learning to generate anticipatory plans before answering. (a) An example problem in the Hendrycks MATH test set. (b) An incorrect solution given by the LLM trained with a baseline method, ReST. The model fails to generate correct reasoning steps. (c) A correct solution given by the LLM trained with our proposed method, LEPA. The model generates high-quality plans, and then follows the plan to solve the problem correctly.
To tackle this challenge, we draw insights from cognitive science (Wang & Chiew 2010 Radüntz 2020]: humans simplify complex problems through high-level abstraction before engaging with details (Ross] 2009). Such abstraction not only lightens the cognitive load but also distills high- level meta-knowledge that is transferable to analogous problems. This idea is also evidenced by recent advances in meta-learning (Finn et al.] 20077] Rakelly et al., 2019), which learn generalizable mntll-nnowedde that enables fast adaptation to similar problems. We propose novel self-training algorithm, LEarning to Plan before Answering (LEPA), that Icarns to generate anticipatory plans be- fore generating detailed step-by-step problem solutions. The anticipatory plans serve as high-level abstract meta-Knowledde that outlines the solution generation path and shields the LLM from the distraction of irrelevant details. During data generation, LEPA prompts the LLM to first devise an anticipatory plan that encapsulates the high-level problem-solving steps, and then generate a solu- tion that aligns with both the problem and the plan. If the solution is correct, the plan-solution pair is stored into the training dataset. Otherwise, the LLM is asked to reflect on the plan and the incorrect solution, and refine the plan until it successfully prompts the LLM to generate correct solutions. With this self-reflection mechanism, LEPA acquires plans that are instrumental in yielding correct solutions. During model optimization, we utilize supervised fine-tuning (SFT) to train the LLM to predict both the plans after self-reflection and the corresponding solutions. As shown in Figure after self-training with LEPA, the LLM generates helpful abstract anticipatory plans that out- Tine the solution steps and are generalizable to similar problems, thus achieving better performance than baseline algorithms. LEPA is extensively evaluated on various challenging language reasoning benchmarks including Hendrycks MATH, and significantly outperforms baseline methods.
To summarize, our main contributions are listed as follows:
1. We present the fundamental problem of what information should self-generated data en- capsulate in the field of LLM self-training
2. We propose novel self-training algorithm, LEPA, that learns to generate anticipatory plans, which serves as high-level abstract knowlege guiding solution generation, before generating detailed problem solutions.
2

---

Published as a conference paper at ICLR 2025
Solution is incorrect
Ask LLM refine Solution
Solution is correct
Prablem
Solution
Add (Problem, Solution) training
Training Dataset
LLM
dataset
(a) Baseline algorithms' data generation procedure.
Solution is incorrect
Ask LLM to refine Plan with self-refiection
Solution correct
Problem
Plan
Solution
Add (Problem, Plan, Solution) training dataset
Training Datasel
LLM
LLM
(h) LEPA's data generation procedure.
Comparison between baseline algorithms' and LEPA's data generation procedure. (a) Baseline algorithms only generate step-by-step solutions to each problem, lacking high-level ab- stract meta weeea-knowwedge that guides solution generation. (b) LEPA generates anticipatory plans before generating detailed problem solutions. These plans are optimized with self-reflection, and encapsu- late the high-level abstract problem-solving steps. The plans efficiently guide the LLM to generate correct solutions.
Figure 2:
3. We evaluate LEPA on several challenging language reasoning benchmarks and demonstrate LEPA's superior performance compared to based algorithms.
LEARNING TO PLAN BEFORE ANSWERING (LEPA)
This section introduces LEPA, a novel self-training algorithm that self-trains the LLM to devise high-level anticipatory plans, which serve abstract solution-generraiion blueprints, before gen- crating detailed problem solutions. LEPA iterates between a data generation phase and model optimization phase. In the data generation phase, LEPA generates high-quality plan-solution pairs with self-reflection. In the model optimization phase, LEPA fine-tunes the LLM with the gener- ated data using SFT. Finally, discuss multiple advantages that the anticipatory plans offer for enhancing the self-training process.
2. DATA GENERATION PHASE
LEPA operates within the common self-training framework, which involves an initial LLM denoted as dg set of prompts containing N problems Pprompt [= and binary scoring function foor (x,, y:) that evaluates the correctness of solution y44 with a score either 1.
In each iteration directly prompt the LLM to generate step-by-step solutions to problems. Instead, LEPA instructs the LLM to first generate anticipatory plan of that serves generation, and then generate the actual solutions y based on the plan and the problem. To avoid the degenerate case of generating plans containing detailed step-by-step problem solutions, L.PA
depicted in Figure 2 LEPA differs from previous methods in that it does not
abstract blueprint for solution

---

Published as a conference paper at ICLR 2025
stresses in the prompt that the plan should be general high-level meta-knowledge that is applica- ble to similar problems, and should not contain any gobleem-speciti information such as detailed calculations. If the solution is correct, i.e., Foor (Tz, Vr) (1z, Pl, v/) is added to the training dataset Di train. Otherwise, LEPA refines the plan with self- reflection. The LLM is prompted with the problem, the previous plan, the corresponding incorrect solution, and the correct answer (if accessible). Then LEPA instructs the LLM to reflect on why the previous plan fails to guide itself generate correct solutions, and then generate new plan based on its reflection results. To avoid information bypassing, LEPA also stresses in the reflection prompt that the reflected plan should not contain problem sppecffcc information, including detailed calculation and the correct answer. LEPA evaluates the refined plan by instructing the LLM to solve the problem with the refined plan. If the generated solution is correct, the vroblem-plan-solutin wu- ple (ara, Pi, vf)) is added to the training dataset. Otherwise, I.PAA repeats the self-reflection process, unless either a correct solution is generated or the number of trials reaches a certain limit (. The self- reflection process empowers LLMs enhance anticipatory plans based on correctness feedback and analysis of unsuccessful attempts, thus efficiently seeking out superior plans.
then the problem wroblem-plan-solune plan-solution tuple
2.2 MODEL OPTIMIZATION PHASE
In each iteration, after acquiring the training dataset Di Droiin LPEA optimizes the model with SFT. LEPA formats data into a two-round conversation. In the first round, The user inputs the problem a; and requires the LLM to generate an anticipatory plan, and the assistant output is the plan pf. In the second round, the user instructs the LLM to solve the problem based the plan it proposed, and the assistant output is the solution y/- The training objective is to minimize the following negative log-likeldood loss:
(OF Dr Divain)= = -E (D) train [lg po, (p), vx)))
While we employ SFT for algorithm simplicity, LEPA is also compatible with more sophisticated reinforcement learning (RL) algorithms such Direct Policy Optimization (DPO) (Rafailov al.) 2024) and Proximal Policy Optimization (PPO) (Schulman et al., 2017). We believe RL algorithms can further boost LEPA's performance, and are important future directions. The pseudo-code for LEPA is presented in [lgorithm[[] Detailed prompts and -aper--araeeters used by LEPA is deferred to Appendix A
2.3 WHY IS THE ANTICIPATORY PLAN BENEFICIAL?
Central to LEPA's efficacy the anticipatory plan, offering multiple advantages for self-training. This subsection discusses these benefits in detail.
Reducing cognitive workload. As demonstrated in Figure I without the anticipatory plans, the L.M may get lost in the problem-solving process, leading to erroneous solution steps. In contrast, the anticipatory plans serve as blueprints that outline the necessary problem-solvin steps, and shield the LLM from the distraction of irrelevant details. Consequently, when generating detailed problem solutions, the LLM is conscious of what to do at the current step, and successfully solves the prob- lem. Research in cognitive science (Wang & Chiew 2010 Radüntz 2020) supports the notion that such structured approach significantly eases cognitive load and improves learning efficiency.
Learning generalizable high-level meta-knowledge. The anticipatory plans abstract high- level meta-knowledge that does not involve problem specifics, and is thus generalizable across sim- ilar problems. For example, the plan demonstrated in Figure [ be readily adapted to variety of combinatorial mathematical problems with similar underlying structures but different parameters. From the meta-learning perspective, LEPA can be interpreted as a meta-learning algorithm that ex- tracts the meta-knowledge in the form of anticipatory plans. The learned meta-knowledge empowers the L.MM to solve similar problems more effectively.
Learning generalizable high-level meta-knowledge. When the correct answer is accessible, the anticipatory plans enable self-reflection that avoids the pitfall of information bypassing. Previous methods like STaR (Zelikman et al. 2022) directly modify incorrect solutions by referring to the
4

---

Published as a conference paper at ICLR 2025
Algorithm LEPA: LEarning to Plan before Answering
1: Require: An imitial LLM dg. a set of problems D Prrompt = {a; |==0 NW a binary scoring function
feor (3i, V:), number of iterations T, maximum self-reflection trials I, learning
2: for t 0 to T do
// In each iteration do
3: Initialize empty training D brain
4: for i - to N - do
// For each problem do
5:
Ask O2 to generate anticipatory plan P; to problem Ask Pg to generate solution 4.0 based 2; and p.:0 if Feor( V4
6:
7:
)= then 4.0 Add f Pi Vi to D Prrain
// Solution is correct, add to training set
8:
9:
else
10:
for j - to do
// Self-reflection iterations
Ask O, to self-reflect on p p.--1 and v. f and generate p
11:
Ask 0, to generate solution y/ fu based 2; and p; 1J if f foor(1i (2 (2 == then
12:
14:
Add {.s, yt } to Dirain Break
15:
//Solution is correct, stop self-reflection
16:
end if
17:
end for
18:
end if
end for
20: B+t1 t- G2 a CsFr Dirain)
// Model Optimization with SFT
end for
correct answer, and are very likely to cheat by only modifying the final answer and ignoring the
consistency between intermediate steps and the final answer (Singh et al. 2023). In contrast, as LEPA requires the anticipatory plans to not include any problem-specine information including the final correct answer, it isolates the correct answer from solution generation. The model must generate correct solutions without seeing the correct answer, preventing the model from cheating during solution generation.
EXPERIMENTS
To demonstrate the effectiveness of LEPA, we evaluate on several challenging reasoning bench- marks, including Hendrycks MATH (challenging math problems) (Hendrycks et al., [2021), Hel- laswag (sentence completion reasoning) (Zellers et al. 2019), BoolQ (paragraph understanding and reasoning) (Clark et al. 2019), and PIQA (physics reasoning) (Bisk et al. (020). For Hendrycks MATH, we evaluate solution correctness with the function provided by the dataset creators (https:
//github. com/ http://hnnrrcckss /match) h We utilize Llama 3 8B Instruct (Dubey et al.) 2024) the
initial LLM. LEPA is compared against several representative self-training algorithms: ReST (Gul- cehre et al. 2023), R. (Singh et al.) 2023). and STaR (Zelikman et al. 2022). All these baseline methods only generate step-by-step solutions to problems. Both ReST and ReSTEM gen- erate solutions with rejection sampling. In each iteration, ReST fine-tunes the model trained after the previous iteration, while EM instead fine-tunes from the initial LLM. STaR generates so- lutions by prompting the LLM to modify incorrect solutions with the aid of correct answers, and also fine-tunes from the initial LLM in each iteration. We demonstrate algorithms'' test accuracy at convergence_] For fair comparison, all methods do not utilize few-shot examples in their prompts. We also demonstrate the initial LLM's efficacy, with either zero-shot CoT prompt (Kojima et al.) 2022 or a LEPA prompt that instructs to first generate an anticipatory plan before answering.
As STaR's accuracy drops significantly on MATH, instead demonstrate its highest accuracy.
5

---

Published as a conference paper at ICLR 2025
Table 1: Test accuracy of LEPA and various baselines on four challenging reasoning benchmarks. "CooI" and "Plan+CoT" refer to the initial LLM's performance with a zero-shot CoT prompt and
the LEPA prompt, respectively. LEPA demonstrates superior accuracy comparison to all other algorithms on each of the benchmarks. Numbers in the parentheses are LEPA's performance im- provement over the best-performing baseline algorithm on each benchmark.
CoT
Plan+CoT
ReST
RMSTEM
STaR
LEPA
Hellaswag
60.8%
56.1% 1%
86.3%
86.4%
85.7%
91.2%
(+4.8% %
Hendrycks MATH
19.5%
22.1%
28.2%
27.2%
25.9%
30.2%
(+2.0%)
BoolQ
77.3%
80.8%
84.5%
86.3%
85.8%
(+2.1 %) 88.4%
PIQA
67.0%
75.7%
81.4%
83.5%
84.2%
85.9%
++177%
Average
56.1%
58.7%
70.1%
70.8%
70.4%
73.9% % (+3.1% ))
3.1
MAIN RESULTS
Table ! presents a comparative analysis of algorithm performance across the four reasoning bench- marks. Notably, the absence of self-training, the LEPA prompt (Plan+CoT) enhances the ini- tial LLM's performance three benchmarks when compared to the traditional zero-shot CoT prompt (CoT). This suggests that the practice of formulating anticipatory plans before generating detailed solutions can significantly improve model efficacy. However, on the Hellaswag benchmark, Plan+CoT falls short of CoT, implying that such enhancement not uniformly achievable across different tasks, potentially due to the initial LLM's lack of calibration for producing high-quality anticipatory plans. As for self- training performance, baseline self- trafniii algorithms only train the LLM to predict step-by-stey solutions, lacking abstract high-level kewa-knowledge about problem- solving. As a consequence, these algorithms perform poorly on these benchmarks. In contrast, LEPA efficiently extracts high-level abstract meta-knowledge with the anticipatory plans, thereby surpassing all baseline algorithms consistently across all benchmarks.
Figure illustrates algorithms' learning curve across learning iterations. LEPA's superior perfor- mance is evident across all benchmarks. Specifically, on Hellaswag, LEPA lags initially during the early iterations 10), where the LEPA prompt slightly less effective than the zero-shot CoT prompt. However, as training progresses, LEPA's performance incrementally surpasses that of the baseline algorithms, suggesting that self-training instrumental in awakening the LLM's capacity to conceive and leverage anticipatory plans effectively. On the remaining three benchmarks, LEPA acquires better initial performance and converges at higher test accuracies, demonstrating the effec- tiveness of introducing the anticipatory plans. We also observe great performance drop of STaR on Hendrycks MATH. This is because STaR is very likely to generate false-positive solutions, i.e., solutions with wrong rationales but correct final answers (Singh et al. 2023), and greatly hinders learning on complex reasoning benchmarks like Hendrycks MATH.
3.2 ABLATION STUDIES
LEPA consists of three key components: the anticipatory plan, plan optimization with self-reflection, and utilizing more inference compute to achieve better performance. This subsection discusses the necessity each component with ablation studies.
Anticipatory plans. We test variant of LEPA that does not introduce anticipatory plans in the data generation phase, and only trains the LLM to predict the step-by-step solutions optimized with self-reflection. As shown in Table 2 this variant ("Without Plan") under-performs LEPA. There are two reasons for this degrade in performance. Firstly, without the anticipatory plans, the LLM does not learn abstract high-level meta-knowledge about problem-sslving. Secondly, as discussed in Section 2.3 directly performing self-reflection on the solutions is very likely to generate false- positive solutions, which greatly hiders learning.
Self-reflection.
To demonstrate the necessity of self-reflection in LEPA's plan optimization, test a variant that instead utilizes rejection sampling (Singh et al., 2023) to sample plan-answer pairs, As shown in Table this variant ("Without self-Reetletton")) also performs worse than LEPA..

---

Published as a conference paper at ICLR 2025
Hellaswag
Hendrycks MATH
0.28
Test Accuracy
Test Accuracy
0.60
0.28
ReST ReSTEN
0.75
0.70
Re57
0.22
LEPA
0.65
ReSTCM
0.20
STaR
LEPA
0.16
Iteration
Iteration
BoolQ
PIQA
0.850
Test Test Accuracy
0.00
Test Accuracy
0.800
0.04
0.775
ReST ReSTCM
0.725
ReST
ReSTEN
0.700
STaR
STaR
0.78
0.850
14
16
10
12
Iteration
Iteration
Figure 3: Algorithms' learning curves the four benchmarks. LEPA achieves better performance than baseline algorithms.
Table 2:
Ablation study on the anticipatory plan and self-reflection. We also demonstrate the perfor- mance of EM the baseline with the highest average test accuracy. "Without Plan" LEPA without anticipatory plans, and "Without Self ReflectRtnnn is LEPA without seff-reflection.
RMSTEM
LEPA
Without Plan
Without Self-Reflection
Hendryeks MATH BoolQ
27.2%
30.2%
24.3%
28.8%
86.3%
88.4%
84.8%
86.9%
PIQA
84.2%
85.9%
84.5%
84.8%
This result implies that self-reflection is effective than rejection sampling in optimizing the anticipatory plans, as it gives linguistic feedback for LLMs to improve the previous plans.
Different ways of utilizing inference compute. LEPA generates both anticipatory plans and prob- lem solutions, utilizing more compute inference time. it is worth discussing how much contri- bution the extra compute makes, and whether the anticipatory plan an effective way to utilize inference compute. For the first question, discussed in Section 3.2 without self-training utilizing inference compute with anticipatory plans can improve performance on three of the four bench- marks, and degrade performance on benchmark. In contrast, after self-training. the anticipatory plans can consistently help LEPA outperform baseline methods. This result demonstrates that extra inference compute contributes part LEPA's performance, and self- soff-training also vital for un- locking the LLM's ability to efficiently utilize these extra compute. For the second question, we test three other variants that train the L.M to utilize inference compute different ways. The first vari- ant adds silence tokens in the solution to give the LLM more compute to generate answers (Goyal jet al., 2023). The second variant trains the LLM to first output a solution, and then outputs a new solution if it finds the original solution incorrect. For data generation of this variant, solutions are generated with rejection sampling, analogous to ReST. We synthesize training data by appending correct solutions to the end of incorrect solutions. The third variant simply asks the LLM generate long solutions. All variants fine-tune the LLM with ReST. As shown in Table 3 LEPA is the only method that successfully utilizes additional inference compute outperform baseline methods. In

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on ways of utilizing inference compute. We on the Hendrycks MATH dataset."Silencce "Silence token" is the variant that adds silence tokens in the solution. "Correction" is the variant that trains the LLM to output new solutions it finds its initial solution incorrect. "Long So- lution" is the variant that instructs the LLM to generate long solutions. "# of Tokens" is the average token length of the LLM's responses to test problems, and "Accuracy" is the LLM's test aceuracy. LEPA is the only method that efficiently utilizes additional inference compute to outperform base- line methods. We put the results in two rows due to the page width limit.
STaR
REST
LEPA
# of Tokens
Accuracy
# of Tokens
Accuracy
of Tokens
Accuracy
175.1
25.9%
477.8
28.2%
826.4
30.2%
Silence Tokens
Correction
Long Solution
# of Tokens
Accuracy
# of Tokens
Accuracy
of Tokens
Accuracy
869.3
28.3%
979.4
27.8%
1409.7
25.4%
contrast, the first variant performs similarly to the ReST baseline, suggesting that silence tokens offer limited benefits for the challenging Hendrycks MATH benchmark. Both the second and the third variant underperform ReST, as the LLM is trained to predict tokens with scant correlation to correct solution formulation. The results above implies that introducing the anticipatory plans is a more efficient way to generate long self training data compared to the aforementioned alternatives. Detailed implementation these variants are deferred to Appendix B
Incorporation with RL algorithms. To demonstrate that LEPA is also applicable to more ad- vanced RL optimization algorithms, we test variant of LEPA that utilizes REINFORCE (Zhang) (et al., 2021b) as the underlying optimization algorithm, which called LEPA+REINFORCE The only difference between LEPA and LEPA+REINFORCE is that LEPA+REINFORCE labels data with rewards of either or -1 (based on the final answer correcmess), and optimizes the LLM with the labelled data using the REINFORCE algorithm. On Hendrycks MATH, LEPA+REINFORCE achieves a test accuracy of 30.6%, while the original L.PAA achieves 30.2%. This performance improvement demonstrates the potential of incorporating LEPA with more advanced optimization algorithms like RL, and is promising future direction.
LPEA with RL. To investigate the potential of incorporating LPEA with RL algorithms, we demonstrate variant of LEPA that utilizes REINFORCE as the optimization algorithm in the model optimization phase. This variant is named REPA-RRIEEO The only difference from LEPA is that LEPA+
does not discard failure data. Instead, it utilizes the solution correctness as the reward (1 for correct solutions, 11 for incorrect solutions). This implementation makes no modi- fication to the data generation process. On Hendrycks MATH, LEPA-REINFORCE achieves test accuracy of 30.6%, while LEPA achieves 30.2%. This performance improvement demonstrates the feasibility and effectiveness of incorporating LEPA with RL algorithms. Additional ablation studies including algorithm performance on OOD benchmarks, other LLMs, additional benchmarks, and evaluation with Simple- are deferred to Appendix C
3.3
CASE STUDY
We present case demonstrate how LEPA's self-reflection mechanism optimizes the anticipatory plans and the solutions. As shown in Figure 4 the initial plan generated by the model is too vague, and cannot provide enough guidance for the model to solve the problem correctly. Consequently, during solution generation, the model generates irrelevant steps, makes mistake in the symbol of the expression, and fails to answer correctly. In the self-reflection process, the model finds out that the previous answer failed to calculate the correct symbols. So it modifies the plan to contain more detailed instructions on how to solve this problem. Note that the plan after self-reflection is still gen- eral meta- meta-knowledd knowledge that is applicable to wide range of similar problems. With this modified plan, the model pays more attention to signs, generates only necessary steps, and successfully generates a correct solution.
8

---

Published as a conference paper at ICLR 2025
Question:
What is the value of - 6v6 + +15+ GV6?
Initial Plan:
Analyge the problem carefully, Identify the mathematical object or expression involved, and determine the relevant mathematical field or discipline belongs to. in this case, identily the type expression, Its structure, and the mathematical operations Involved. Consider the properties and relations between the given numbers, expressions, and variables, look any opportuntties apply mathematical concapts, thearems, formulas simplity solve problem.
Answer guided by the initial plan:
Analyyting the problem carsfully, identify the mathematical object spec fically retionalizing the denominstors and simplifying expressions.
expression involving square roots. The relevant hematiatiaal field algebra,
Step 1: consider the structure the expression and identify the perfect squares within the radicals, Recognizing that 6v6 can be rewritten v38 = 6.6. rewrite expression as v5 - 6y6+ v115 6v6 v'9 + -v66+ 19 6v6..
apply the difference of squares formula to the first radical, obtaining 6V8 (3 va) =3 Siillary, apply the difference of squares
formula to the second radical, obtaining 15 + svē = (3+ vōj" =3+ vo.
Step 3:
combine expressions, adding terma, obtain v5+ 3++ vB 6 648.
Answer: + 2y5
Self-reflection:
The model falls to solve the problem because incorrectly applies the difference of squares formula, resulting the same expression for both radicals. This is likally due the model's lack of attention the rigns and the correct application the formula. The new instruction should focus on the correct application of the difference of squares formula and the importance considering the signs.
Plan after self- reflection:
When simplifying radical expressions, consider the structure of the radicand and the
Identify the racical expression
components.
2. Simpliif each radical separatelyy considering the signs.
3. Combine like terms to obtain the simplifled expression.
Verify
by checking original expression simplified expression for equivolence.
Answer guided by the plan after self reflecteftt::::
Stop Identify radical expression and its components.
radical expression v/15- 6x6 + +-v15++ 6/6, which consists of radical Step Simplify each redical seporately considering the signs.
We can start by fying the first radical: v15 - 6v6
(3 v6)
=3 vv.
Similarly simplify second radical: 1
(3 vo]*
v6.
Step Combine
obtain simplified expression.
The simplified expression is 3- VB + 3 <B = 6.
Verify the result chocking the original expression and the simplitied expression for equivalence. or gigaall expression and the simplified expression are equivalent, so our answer is correct.
Answer: 6
Figure 4: A case study demonstrating how LEPA optimizes the anticipatory plans and the solutions with self-reflection. The initial plan is too broad and lacks detail, and fails to provide enough guid- ance generate correct answers. The self-reflection process successfully analyses what is wrong, and generates a new, high-quality plan that provides more guidance while maintaining generaliz- ability. With the new plan after self-reflection, the model successfully generates correct solutions.
RELATED WORKS
Self-training. With the fast development of LLMs, the thirst for data continues to grow. A promis- ing way is to generate high-quality data with the LLM itself. A branch of works mainly focus on designing the data generation progress. STaR (Zelikman et al.) (022) operates by initially prompt- ing the LLM to produce step-by-step solutions, followed by an adjustment phase where the LLM corrects its errors with the aid of the correct answers. One severe limitation of STaR is that the modification process makes it very possible to generate false-positive solutions, i.e., solutions with wrong rationales but correct final answers. RFT (Yuan et al. 2023), REST (Gulcehre et al., 2023), and RM (Singh et al. 2023) instead adopt rejection sampling for data generation, and suffer less from the false-posiin positive issue. TRICE (Hoffman et al. (024) improves over STaR by utilizing a Markov-chain Monte Carlo pnrectationmaxiiizzae ducing a control-variate method to control gradient variance. Re-ReST (Dou et al. 2024) utilizes self-reflection to correct the generated wrong answers. LMSI (Huang et al.) 2022 considers the scenario where the correctness of model-ggnerated data cannot be verified during training, and filers data with majority voting. Apart from these methods, SPAG (Cheng et al. 2024) generates data by asking LLMs to self-play in adversarial games. These previous methods above only generate step- by-step solutions to problems, and lack high-level meta-knowledge that are generalizable across
algorithm to sample solutions, and intro-

---

Published as a conference paper at ICLR 2025
similar problems. In contrast, LEPA learns abstract meta-knowleege in the form of anticipatory plans, and achieves better performance on complex benchmarks.
Scaling inference compute. As proposed by Snell et al. (2024) and confirmed by the recent inspir- ing GPT OI model (Hu et al. 2024), scaling inference compute can further boost LLM performance. Similar to LEPA, PS Prompting (Wang et al., 2023b) also scales inference compute by asking the LLM to first generate plan before answering. but does not consider how to generate data and fine- tune the LLM. Moreover, it does not consider how automatically optimize the anticipatory plans. HSP (uu et al. 2024) is the most relevant work to ours, which trains the LLM to output hints before solving the problem. However, HSP's hints are pre-collected rather than seff-genrrated, and induce additional data collection costs. PHP (Zheng et al. 2023) utilizes previously generated answers as hints, and encourages the LLM to answer with reference to its previous answers. LEPA efficiently utilizes inference compute by training the LLM to generate helpful anticipatory plans, which contain high-level meta-knowledge on problem-solvingg before generating actual problem solutions. These plans are automatically optimized by the LLM itself, and do not require additional human design.
Meta-learning. Meta-learning aims at "learning to learn", i.e., designing meta-algorithms that op- timize learning algorithms automatically (Finn et al. 2017; [Sung et al., 2017, Rakelly et al.) 2019)] Zhang et al. 2021a Wang et al., 2023a). LEPA can be interpreted as a meta-learning algorithm that learns the meta-knowledd designing plans with human effort. The most relevant work is Quiet-STaR (Zelikman et al.) 2024). which meta-learns meta-tokens that help the LLM to predict the next token. LEPA considers the setting of problem-solving rather than general next-token prediction, and meta-learns the generation of anticipatory problem-solving plans.
of designing the anticipatory plans for each problem. rather than
Planning in LLMs. Recently, several works have demonstrated the effectiveness of integrating planning in LLMs. ReAct ( (Yao al., 2022) and DEPS (Wang et al., 2024c) generate plans before dealing with recision-making problems, and LUMOS Yin et al. 2023) fine-tunes the LLM on pre-collected datasets containing planning data. To our best ledge LEPA is the first work to integrate planning in the process of self-training, and improves the LLM's planning ability by training on self-generated data.
Self-reflection. Self-reflection enables LLMs to reflect on their mistakes and generate better re- sponses. It can be viewed as a process of in-context optimization to produce better responses. Previ- ous works demonstrate that self-reflection can significantly improve LLM response quality (Renze & Guven 2024; Shinn et al. 2024, Madaan et al. (2024). LEPA utilizes self relf-reflection to optimize plans and solutions in the data generation phase, and acquires data of higher quality.
5 CONCLUSION
This paper presents the fundamental problem of what data should be generated in self-training al- gorithms. Inspired by cognitive science research and recent meta-learning advances, we propose novel idea of learning abstract meta-knowledge in the form of anticipatory problem-solving plans. Based this idea, we propose a novel self-training algorithm, LEPA, which automatically generates and learns the anticipatory plans. Experiment results on several challenging reasoning benchmarks demonstrate the effectiveness of LEPA. An interesting future direction is incorporate LEPA with more advanced model optimization methods such as RL. It is also worth exploring how well can LEPA perform on larger and more advanced LLMs, and how to scale LEPA utilize more infer- ence compute. Furthermore, as LLMs may solve simple problems without planning, important future direction is to automatically identify complex problems that require planning from simple problems that can be easily solved without planning. This identification can avoid wasting compute resources and help the LLM solve problems more efficiently.
ACKNOWLEDGEMENT
This work is supported by National Natural Science Foundation of China (62176135), the National Key R&D Program of China (2022ZD0161700), Shanghai Qi Zhi Institute Innovation Program SOZ202306 and the Tsinghua University Dushi Program.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Concerns about safety and reliability are key points of discussion in the LLM community. The use of anticipatory plans LLMs is a step towards making the models' actions more understandable and transparent to people. Yet, LEPA cannot guarantee that every solution will strictly match the plans it creates, which means further work is needed to solidify the trustworthiness of LLMs.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint MtXXiv:2333.88770 2023.
Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training Ilm reasoners via compute-optimal sampling. arXiv preprint arXiv:2408.16733 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Pengyu Cheng. Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du Self-playing adversarial language game enhances IIm reasoning. arXiv preprint arXiv:2404 10642, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 2019.
Zi-Yi Dou, Cheng-Fu Fu Yang, Xueqing Wu, Kai-We Chang, and Nanyun Peng. Reflection-reinforced self-training for language agents. arXiv preprint arXiv:2406.014 01445. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1135. PMLR, 2017.
Jinlan Fu, Shenzhen Huangfu, Hang Yan, See-Kiong Ng. and Xipeng Qiu. solving prompting: Guiding Ilms to effectively utilize encoded knowledge. arXiv preprint arXiv:2402.14314 2024.
Hint-before-
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.0888 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.033000 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Matthew Douglas Hoffman, Du Phan. David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Haichuan Hu, Ye Shang, Guolin Xu, Congqing He, and Quanjun Zhang. Can gpt-ol kill all bugs? arXiv preprint arXiv:2409.10033 2024.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.11610 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Thea Radüntz. The effect of planning, strategy learning, and working memory capacity on mental workload. Scientific reports, 7009), 2020.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off poo-cy meta-reinforrcement learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Matthew Renze and Erhan Guven. Self-reflection in Ilm agents: Effects on problem-solving perfor- mance. arXiv preprint arXiv:2405.0663: 2024.
Brian H Ross. The psychology of learning and motivation: Advances in research and theory. 2009.
Amrita Saha, Vardaan Pahuja, Mitesh Khapra. Karthik Sankaranarayanan, and Sarath Chandar. Complex sequential question answering: Towards learning to converse over linked question answer pairs with knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint #tXiv:1707.063477 2017.
Sakib Shahriar, Brady D Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, and Laiba Batool. Putting gpt-4o to the sword: A comprehensive evaluation of language, vision, speech, and multimodal proficiency. Applied Sciences, 14(17):7782, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.0658 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311 2024.
Flood Sung, Li Zhang. Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: 1 706.09529, 2017.
Jianhao Wang, Jin Zhang, Haozhe Jiang, Junyu Zhang, Liwei Wang, and Chongjie Zhang. Offline meta reinforcement learning with in-distribution online adaptation. In International Conference on Machine Learning, pp. 18626-36669 PMLR, 2023a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Lim. Plan-and-solve prompting: Improving zero-shot ofain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04M5 04091, 2023b.
12

---

Published as a conference paper at ICLR 2025
Yingxu Wang and Vincent Chiew. On the cognitive process of human problem solving. Cognitive systems research, 11(1):81-22, 2010.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024a.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment tech- niques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXivv2407.16211 2024b.
Zihao Wang. Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Shawn Ma, and Yitao Liang. De- scribe, explain, plan and select: interactive planning with iims enables open-world multi-task agents. Advances in Neural Information Processing Systems, 36, 2024c.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 2022.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-We Lei-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source llms. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2023.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint a300::300000
2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 35:15476-1544 2022.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arviv preprint arxiv 1905:::8905008830 2019.
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjic Zhang. Metacure: Meta reinforcement learning with empowereent-drivee exploration. In International Conference on Machine Learning, pp. 12600- PMLR, 2021a.
Junzi Zhang, Jongho Kim, Brendan O'Donoghue, and Stephen Boyd. Sample efficient reinforce- ment learning with reinforce. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10887-10895, 2021b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. A survey of large language models. arXiv preprint a8Xiv:2303.18223, 2023.
Chuanyang Zheng. Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive- prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023.
13

---

Published as a conference paper at ICLR 2025
A DETAILED PROMPTS AND
This section demonstrates the detailed prompts and the presents the prompts used by LEPA and baseline algorithms.
used by LEPA and baseline
algorithms. Figure S
As for hyper-parameters, for fair comparison, we ensure that all algorithms have the same number of trials (5) in the data generation phase. LEPA is allowed to have maximally self-reflection processes for each problem. For ReST and ReSTTM, 5 solutions are sampled for each question. For STaR, it has maximally opportunities to modify the previous incorrect answer. All algorithms fine-tunes the LLM for epoch in each model optimization phase. For the data generation phase of all algorithms, we use a temperature of 0.5 for sampling. We use temperature of 0.0005 for all test results. We use 3 as the learning rate for all learning algorithms.
Prompt for anticipatory plan generation:
You are expert at destaning plans for large language models to solve problems. The problem be solved [Question)
Output the plan you design. Note that plan should be general knowledge that help solve similar problems, so do not contain any question-specifi information. Also, the content will be directly added to the prompt, so pay attention to it format. The plan should be concise, no longer than 1024 tokens. ns.
Output only plan. oupput any other
Prompt for solution generation:
Based the plan your response on its own line after "Anower!", and you do not need to use Whoved command. Your response should be conciso, no longer than 1024 tokens. The problem (Question)
solve the problem form SANSWER (without quotes) SANSWER
In each step of your solution, explain how the plan affect problem. Remember put your answer
The last
Prompt reflectlio:
You are an export in designing plans for large language models to solve problems, You have found that the original plan fails to solve a problem. You need to analyre the failure case, and design new plan. The new plan should help large language model solve the failure case. You encouraged design plans
original plan to better explore high-quality plans.
The problem Is: (Question)
original ilan
(Original Plan]
The incorrect solution given the large language model under the original plan
[Original solution]
The destred correct final answer is:
(Correct Ansvver]
fail problem? wrong the answer? design plan model correctly solve the problem? How distinct the new plan from the original plan? What contents should the new plan obtain? Pay special atzention to the formatting requirements. Does the models output strictly tollow the required output format? Answer concislly, na longer than 2580 takens.
Anelyze information
Why
Prompt for new plan generation after self-reflection:
the Based on the analysis above, output the new plan. Note that the new plan should be general knowledge that help solve similar problems, so do not contain ancouraged
distinct from original contain the directly added to prompt, so pay attention to formet. The content should be short and concise, no longer than 1024 tokenss onlp the plan. Do not output any other words.
specifffiific information. You better explore high-quaiity plans. Als, the content
(a) LEPA prompt.
Prompt solution generation:
Solve the following problem stop by stop. The answer to the problem. Remember to put your answer on its own line after "Answer:" and you do not need to use Whosed command. Your response should be concise, no longer than 1024 tokens. The problem is: [Question]
your response should be the Answer: SANSWER (without quotes] where SANSWER is
Prompt for solution modification lonly used STaR): solution wrong. correct answer [Correct Answver] Modify previous solution the correct Output the modified solution only Do not any other words.
(b) Prompt used by haseline methods.
Figure 5: Detailed prompts used by (a) LEPA and (b) baseline algorithms.
B ABLATION DETAILS
This section presents the details of the variants discussed in the "Different ways of utilizing inference compute" part of Section 3.2
For the second variant, we first sample correct and incorrect solutions for each problem with re- jection sampling. Then we synthesize training data by first adding sentence of "Oops, made mistake. The correct solution is: to the end incorrect solutions. Then we append a correct solution to the end of this sentence.
14

---

Published as a conference paper at ICLR 2025
For the third variant, explicitly instruct the LLM output solutions that are approximately 2,000 words long. We observe that the LLM generates verbose responses that obscure the important steps in solving the problem.
C ADDITIONAL ABLATION STUDIES
OOD performance. We evaluate OOD performance by training Hendrycks MATH and testing on the Math category of MMLU-Pro (Wang et al. (0224).. As shown in Table LEPA consistently outperforms baseline algorithms in this OOD setting.
CoT
Plan+CoT
REST
RCSTEM
STaR
LEPA 38.9%
Performance
30.4%
33.9%
35.1%
35.3%
35.8%
Table 4: Performance of different algorithms training Hendrycks MATH and testing the Math category of MMLU-Pro. "CoT"" and "Plan+CoT" refer to the initial LLM's performance with a zero-shot CoT prompt and the LEPA prompt, respectively. LEPA achieves better generalization than baseline algorithms.
Other LLMs. We additionally evaluate algorithm performance on Llama 3.1 8B Instruct. As shown in Table on the Hendrycks MATH dataset, the LEPA prompt can slightly improve over the zero-shot CoT prompt on the initial LLM. As for self-training, LEPA significantly outperforms the baseline algorithm. These empirical results are consistent with our main results presented in Section
3.1
Additional Benchmarks. We additionally evaluate on CSQA Saha et al. 2018) and MMLU Hendrycks et al., 2020), and results are shown in Table 6 line algorithms on these benchmarks.
LEPA consistently outperforms base-
Algorithm CoT
Plan+CoT
REST
ReSTEM
STaR
L.PA 49.6%
Performance 37.2%
38.4%
45.3%
46.9%
45.0%
Table 5: Algorithm performance on Hendrycks MATH, with Llama 3.1 8B Instruct as the initial LLM. "CoT" and "Plan+CoT" refer to the initial LLM's performance with zero-shot CoT prompt and the LEPA prompt, respectively
CoT
Plan+CoT
ReST
ReSTEM
STaR
LEPA
CSQA
67.1%
69.3%
73.2%
74.0%
74.1%
75.2%
MMLU
61.9%
60.1%
64.3%
65.6%
65.8%
66. %
Table 6: Performance comparison of different methods on CSQA and MMLU benchmarks. LEPA achieves higher performance than baseline algorithms.
Evaluation with Simple-Eval. We re-evaluate Hendrycks MATH performance with Simple-Eval, and the results are demonstrated in Table 7 With the new evaluation, LEPA still outperforms base- line algorithms.
15

---

Published as a conference paper at ICLR 2025
CoT
Plan+CoT
ReST
RM
STaR 29.2%
LEPA
26.1 1 1%
28.5%
31.2%
31.4%
33.7%
Table 7: Hendrycks MATH performance evaluated with Simple-Eval, With the new evaluation, LEPA still outperforms baseline algorithms.
16


Reviews:
[{"review_id": "A0gn3fO8L9", "reviewer": "Reviewer_b2S7", "summary": "This paper introduces a self-training approach aimed at enhancing large language model (LLM) reasoning.\nLEPA addresses the need for LLM self-generated data to contain meta-knowledge, rather than merely task-specific steps, enabling better generalization. By integrating anticipatory plans, LEPA provides a structured approach to problem-solving, reducing cognitive load, enhancing solution quality, and preventing the generation of incorrect rationales for correct answers.", "questions": "1. Will it work on other LLM?\n2. How to deal with questions which have no correct answer after data creation pipeline?\n3. How to better choose data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Novel self-improvement method to improve the LLM's reasoning capability.", "weaknesses": "1. Only improve in-domain performance. \n2. No OOD experiments. \n3. Only one LLM (Llama 3 8B Instruct) is evaluated here.\n4. Data generation pipeline is not clear illustrated. \n5. Experiment section is not convincing with so few benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQQ6gunypD", "reviewer": "Reviewer_VGrZ", "summary": "The paper addresses the problem of what information should self-generated data encapsulate in the LLM self-training, by prompting an LLM (specifically Llama 3.1-8B-Instruct in this paper) to first generate an anticipatory plan before generating solutions to reasoning problems. Then optimize the anticipatory plan via self-reflection using the LLM itself until the plan leads to a correct solution. Once the corrected solution is generated, the reasoning problem and the plan as well as the solution are added to the training dataset to fine-tune the LLM.\n\nExperimental results across four reasoning benchmarks show better performance compared to three baselines, especially after more training iterations.", "questions": "1. Presentation of Table 3 is confusing and cannot figure it out in the title. Could you please explain the metrics in the Table and the relation of three variants of tokens to the three methods?\n2. Have you ever tested the effectiveness of LEPA on other LLMs?\n3. How do you think we can optimize the model when the solution to a reasoning problem is open-ended such that the evaluation of solution correctness is hard?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses an important problem that what essential information of self-generated data should an LLM learn during self-training, and considers an anticipatory plan as the essential information.\n2. Experiment results show the the anticipatory plan can guide an LLM to generate correct solutions after the plan is refined iteratively through self-reflection and added to the training data for further fine-tuning.\n3. The ablation studies validate the usefulness of key components of the proposed method.", "weaknesses": "1. Although the paper claims that the proposed method is also compatible with more sophisticated RL algorithms, the current version does not support the claim. If an RL algorithm is incorporated, the method will be quite different from the proposed one as the key components of current methods (e.g., self-reflection and fine-tuning method) should be significanly altered. So the performance of current method is limited to the LLM performance via prompting and self-reflection.\n2. The justification of the anticipatory plan in Section 2.3 is not solid theorectically by simply referring to some work in cognitive science and the example shown in Figure 1.\n3. The idea of generating a plan before solutions is similar to recent work on generating a plan using an LLM before actions for emobodied agents (e.g, [1][2]), however, the paper does not address the related work in Section 4. Also Section 4 lacks related work on self-reflection of LLMs (e.g, [3][4]), which is a key component of the proposeed method.\n\nReferences:\n\n[1] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.\n\n[2] Yao, Shunyu, et al. \"React: Synergizing reasoning and acting in language models.\" arXiv preprint arXiv:2210.03629, 2022.\n\n[3] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.\n\n[4] Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R1jBORPnnG", "reviewer": "Reviewer_5cwy", "summary": "This paper introduces a novel training method LEPA for reasoning tasks. Unlike previous approaches that only focus on optimizing answer generation, LEPA also fine-tunes the LLM for plan generation. This is because planning represents a high-level abstract concept (meta-knowledge) that presents generalizable patterns across similar problems. LEPA prompts the LLM to generate an anticipatory plan and refine it until the plan can guide the LLM to a correct answer. After gathering the training data, LEPA uses SFT to optimize both the planning and answering generation. LEPA significantly enhances performance across various reasoning tasks.", "questions": "1. In Table 1, the zero-shot CoT performance for llama3-8b-instruct is only 19%, which is a little bit too low. I acknowledge that the result would be different due to the prompts or something, but 19% is not very convincing to me. I would recommend you use other evaluation methods, e.g., simple-eval (https://github.com/openai/simple-evals) to re-evaluate the answer of the MATH dataset.\n\nAlso, see the comments on weaknesses. If issues are solved, I will improve my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The highlighted advantage is that the idea is simple and straightforward. It’s also quite clear why this would help performance, and the authors present it well with clear figures and examples. The method is easy to implement and can be easily generalized to other tasks, which makes this paper impactful.\n\n2. The experiments are reasonable and effectively demonstrate the superiority of the method. There is sufficient analysis, including an analysis of inference computation.", "weaknesses": "1. My biggest concern is that this method may negatively affect the experience of using LLMs for simple problems. The current approach encourages the LLM to have careful planning before answering, which is unnecessary for simple problems and incurs unnecessary costs. In contrast, ReSF does not have this issue.\n\n2. Secondly, I am concerned that this merely replaces the solution with an easily memorable method. I am curious whether it will provide sufficient improvement for few-shot scenarios, e.g., trained within a few MATH instances and tested with the others, or out-of-distribution reasoning problems, e.g., trained within the MATH dataset and tested with math questions in MMLU-pro.\n\n3. More LLMs in experiments would be better. But this is not a big issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qckGGKkCvN", "reviewer": "Reviewer_FT6B", "summary": "The author present LEPA, a data generation pipeline. The data will not only contain the COT answer but also the planning process. The experiment shows that the LEPA beat other baselines in Hellaswag, Hendrycks MATH, BoolQ, and PIQA. The ablation study shows the importance of the self-peflection process and planning data in the data generation pipeline.", "questions": "1. In line 224, you claim that LEPA can \"preventing the creation of false-positive data\". However, even STaR can't prevent the creation of (step-level) false-positive data. It's even harder to judge whether the planning is correct/incorrect/meanless. So maybe you should give a evaluation standard to judge the quality of the planning data (and talking what is the defination of false-positive in planning).\n\n2. Why not compare with other baselines mentioned in related work, especially those used MCTS?\n\n3. Add some OOD benchmark (like MATH).\n\n4. Discuss the reason why the planning data can improve the reasoning ability of the LLMs.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow.\n\n2. The results show that planning data improves the performance of the LLMs.\n\n3. The results in Figure 3 are impressive, the LEPA seems to continuously improve the performance even after 15 iterations.", "weaknesses": "1. Training planning data is not a new idea, paper like LUMOS[1] (specifically LUMOS-O) has already shown that planning data can improve the performance of the LLMs, although I admit that the LEPA is more simple than LUMOS-O.\n\n2. There is only outcome judgment in data selection. Actually, LEPA doesn't judge the quality of the planning as well as the quality of the COT before the answer.\n\n3. The comparison in Table 2 is a bit unfair. The LEPA's training data is (likely) bigger than the \"Without Plan/Without Self-Reflection\" settings if the settings are aligned with lines 349-360.\n\n\n[1] LUMOS: LEARNING AGENTS WITH UNIFIED DATA, MODULAR DESIGN, AND OPEN-SOURCE LLMS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)