Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving

Authors: Jin Zhang, Flood Sung, Zhilin Yang, Yang Gao, Chongjie Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: llm, self-training, high-level abstraction, self-reflection, meta learning, anticipatory plans

Abstract:
In the field of large language model (LLM) post-training, the effectiveness of utilizing synthetic data generated by the LLM itself has been well-presented. However, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve as abstract meta-knowledge for problem-solving, before engaging with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction of irrelevant details. During data generation, LEPA first crafts an anticipatory plan based on the problem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming to acquire plans that are instrumental in yielding correct solutions. During model optimization, the LLM is trained to predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates remarkable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729363763426

thread_type:
official_review

<thread_object>
creator: reviewer_ft6b

<content>
summary:
The author present LEPA, a data generation pipeline. The data will not only contain the COT answer but also the planning process. The experiment shows that the LEPA beat other baselines in Hellaswag, Hendrycks MATH, BoolQ, and PIQA. The ablation study shows the importance of the self-peflection process and planning data in the data generation pipeline.

strengths:
1. The paper is easy to follow.

2. The results show that planning data improves the performance of the LLMs.

3. The results in Figure 3 are impressive, the LEPA seems to continuously improve the performance even after 15 iterations.

weaknesses:
1. Training planning data is not a new idea, paper like LUMOS[1] (specifically LUMOS-O) has already shown that planning data can improve the performance of the LLMs, although I admit that the LEPA is more simple than LUMOS-O.

2. There is only outcome judgment in data selection. Actually, LEPA doesn't judge the quality of the planning as well as the quality of the COT before the answer.

3. The comparison in Table 2 is a bit unfair. The LEPA's training data is (likely) bigger than the "Without Plan/Without Self-Reflection" settings if the settings are aligned with lines 349-360.


[1] LUMOS: LEARNING AGENTS WITH UNIFIED DATA, MODULAR DESIGN, AND OPEN-SOURCE LLMS

questions:
1. In line 224, you claim that LEPA can "preventing the creation of false-positive data". However, even STaR can't prevent the creation of (step-level) false-positive data. It's even harder to judge whether the planning is correct/incorrect/meanless. So maybe you should give a evaluation standard to judge the quality of the planning data (and talking what is the defination of false-positive in planning).

2. Why not compare with other baselines mentioned in related work, especially those used MCTS?

3. Add some OOD benchmark (like MATH).

4. Discuss the reason why the planning data can improve the reasoning ability of the LLMs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729721527464

thread_type:
official_review

<thread_object>
creator: reviewer_5cwy

<content>
summary:
This paper introduces a novel training method LEPA for reasoning tasks. Unlike previous approaches that only focus on optimizing answer generation, LEPA also fine-tunes the LLM for plan generation. This is because planning represents a high-level abstract concept (meta-knowledge) that presents generalizable patterns across similar problems. LEPA prompts the LLM to generate an anticipatory plan and refine it until the plan can guide the LLM to a correct answer. After gathering the training data, LEPA uses SFT to optimize both the planning and answering generation. LEPA significantly enhances performance across various reasoning tasks.

strengths:
1. The highlighted advantage is that the idea is simple and straightforward. It’s also quite clear why this would help performance, and the authors present it well with clear figures and examples. The method is easy to implement and can be easily generalized to other tasks, which makes this paper impactful.

2. The experiments are reasonable and effectively demonstrate the superiority of the method. There is sufficient analysis, including an analysis of inference computation.

weaknesses:
1. My biggest concern is that this method may negatively affect the experience of using LLMs for simple problems. The current approach encourages the LLM to have careful planning before answering, which is unnecessary for simple problems and incurs unnecessary costs. In contrast, ReSF does not have this issue.

2. Secondly, I am concerned that this merely replaces the solution with an easily memorable method. I am curious whether it will provide sufficient improvement for few-shot scenarios, e.g., trained within a few MATH instances and tested with the others, or out-of-distribution reasoning problems, e.g., trained within the MATH dataset and tested with math questions in MMLU-pro.

3. More LLMs in experiments would be better. But this is not a big issue.

questions:
1. In Table 1, the zero-shot CoT performance for llama3-8b-instruct is only 19%, which is a little bit too low. I acknowledge that the result would be different due to the prompts or something, but 19% is not very convincing to me. I would recommend you use other evaluation methods, e.g., simple-eval (https://github.com/openai/simple-evals) to re-evaluate the answer of the MATH dataset.

Also, see the comments on weaknesses. If issues are solved, I will improve my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730302906891

thread_type:
official_review

<thread_object>
creator: reviewer_vgrz

<content>
summary:
The paper addresses the problem of what information should self-generated data encapsulate in the LLM self-training, by prompting an LLM (specifically Llama 3.1-8B-Instruct in this paper) to first generate an anticipatory plan before generating solutions to reasoning problems. Then optimize the anticipatory plan via self-reflection using the LLM itself until the plan leads to a correct solution. Once the corrected solution is generated, the reasoning problem and the plan as well as the solution are added to the training dataset to fine-tune the LLM.

Experimental results across four reasoning benchmarks show better performance compared to three baselines, especially after more training iterations.

strengths:
1. The paper addresses an important problem that what essential information of self-generated data should an LLM learn during self-training, and considers an anticipatory plan as the essential information.
2. Experiment results show the the anticipatory plan can guide an LLM to generate correct solutions after the plan is refined iteratively through self-reflection and added to the training data for further fine-tuning.
3. The ablation studies validate the usefulness of key components of the proposed method.

weaknesses:
1. Although the paper claims that the proposed method is also compatible with more sophisticated RL algorithms, the current version does not support the claim. If an RL algorithm is incorporated, the method will be quite different from the proposed one as the key components of current methods (e.g., self-reflection and fine-tuning method) should be significanly altered. So the performance of current method is limited to the LLM performance via prompting and self-reflection.
2. The justification of the anticipatory plan in Section 2.3 is not solid theorectically by simply referring to some work in cognitive science and the example shown in Figure 1.
3. The idea of generating a plan before solutions is similar to recent work on generating a plan using an LLM before actions for emobodied agents (e.g, [1][2]), however, the paper does not address the related work in Section 4. Also Section 4 lacks related work on self-reflection of LLMs (e.g, [3][4]), which is a key component of the proposeed method.

References:

[1] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629, 2022.

[3] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.

[4] Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.

questions:
1. Presentation of Table 3 is confusing and cannot figure it out in the title. Could you please explain the metrics in the Table and the relation of three variants of tokens to the three methods?
2. Have you ever tested the effectiveness of LEPA on other LLMs?
3. How do you think we can optimize the model when the solution to a reasoning problem is open-ended such that the evaluation of solution correctness is hard?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681380291

thread_type:
official_review

<thread_object>
creator: reviewer_b2s7

<content>
summary:
This paper introduces a self-training approach aimed at enhancing large language model (LLM) reasoning.
LEPA addresses the need for LLM self-generated data to contain meta-knowledge, rather than merely task-specific steps, enabling better generalization. By integrating anticipatory plans, LEPA provides a structured approach to problem-solving, reducing cognitive load, enhancing solution quality, and preventing the generation of incorrect rationales for correct answers.

strengths:
1. Novel self-improvement method to improve the LLM's reasoning capability.

weaknesses:
1. Only improve in-domain performance. 
2. No OOD experiments. 
3. Only one LLM (Llama 3 8B Instruct) is evaluated here.
4. Data generation pipeline is not clear illustrated. 
5. Experiment section is not convincing with so few benchmarks.

questions:
1. Will it work on other LLM?
2. How to deal with questions which have no correct answer after data creation pipeline?
3. How to better choose data?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes LEPA for reasoning tasks. The main pipeline is that before really outputting the answer, it generates the plan of how to do the answer generation. The intuition behind why this works is that the plan has some high-level abstract concept that could host some general pattern across different problems with similar nature. In order to give LLM the ability to generate plan, it uses SFT to optimize the planning. All reviewers agree that this paper has good structure and easy to follow. Experiments show that the improvements are significant. Of course a more theoretical formulation to further justify this paper’s improvement in theory would be another huge plus for this paper. This paper is more empirical-driven. Because of this, reviewers have concern regarding this proposed idea’s relationship with other similar ideas. Furthermore, its performance benefit needs to be further justified with its computation cost. Yep, post-training scaling is more accepted now. I would encourage the paper to do more comprehensive analysis about cost/performance tradeoff, more analysis on whether it would fail, and more solid efforts to show its compatibility with RL (the improvements with REINFORCE seems week) to make this paper more solid.