Let's Be Self-generated via Step by Step: A Curriculum Learning Approach to Automated Reasoning with Large Language Models

Authors: Kangyang Luo, Zichen Ding, Zhenmin Weng, Lingfeng Qiao, Meng Zhao, Xiang Li, di yin, Jinlong Shu
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: large language models, chain of thought, automated reasoning, curriculum learning

Abstract:
While Chain of Thought (CoT) prompting approaches have significantly consolidated the reasoning capabilities of large language models (LLMs), they still face limitations that require extensive human effort or have performance needs to be improved. Existing endeavors have focused on bridging these gaps; however, these approaches either hinge on external data and cannot completely eliminate manual effort, or they fall short in effectively directing LLMs to generate high-quality exemplary prompts. To address the said pitfalls, we propose a novel prompt approach for automatic reasoning named LBS3, inspired by curriculum learning which better reflects human learning habits. Specifically, LBS3 initially steers LLMs to recall easy-to-hard proxy queries that are pertinent to the target query. Following this, it invokes a progressive strategy that utilizes exemplary prompts stemmed from easy-proxy queries to direct LLMs in solving hard-proxy queries, enabling the high-quality of the proxy solutions. Finally, our extensive experiments in various reasoning-intensive tasks with varying open- and closed-source LLMs show that LBS3 achieves strongly competitive performance compared to the SOTA baselines.

Decision: withdrawn
Reviewer scores: 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730580401810

thread_type:
official_review

<thread_object>
creator: reviewer_8bhm

<content>
summary:
This paper presents a new LLM prompting technique: given a problem (e.g., math), generate easy examples (phase 1), generate hard examples (phase 2), and finally solve the problem conditioned on examples. The key insight is that previous approaches to automatically generating examples might generate examples that are too hard, which isn't as helpful, so one can think of this work as bringing curriculum learning ideas into the prompting regime.

strengths:
- The idea of easy-to-hard prompting is natural, the solution is simple and elegant, which is well appreciated.
 - For the experiments, the choice of models, datasets, and baselines are all reasonably thorough and solid.

weaknesses:
- I feel like the paper could have been a bit stronger on the analysis. The paper does ablations on the number of examples used in each of the two stages (Figure 3), examines using only easy or only hard examples (Figure 5), and compares against the baselines. It demonstrates the effectiveness of the method but it doesn't really tell me why the method is effective. For example, I would be interested in the type of errors that are made as result of the suboptimal ablations; can the reason for failure be attributed to bad reasoning? How valid are the examples generated? Are the easy examples actually easy and the harder examples actually harder? What if more exemplars were used?  How sensitive is is the method to the wording of the prompt (how was the prompt in Table 3 derived?)?
 - Moreover, the fact that Llama3-70B-Instruct is better than GPT-4.0 turbo on most tasks is very counterintuitive. Might it be some artifact of the prompting?  It just seems that GPT-4.0 turbo has to be a stronger model (according to every benchmark I've seen). This definitely warrants some explanation, but there wasn't anything satisfying provided.

questions:
- Line 310: for closed-source models, why can't you set the temperature = 0 to get determinism?
 - Section 4.2: why does Llama3-70B-Instruct outperform GPT-4.0 turbo?  "We considered only a limited set of reasoning tasks" is given as an explanation, but I find this rather unsatisfactory, because GPT-4.0 turbo should really dominate Llama3-70B across the board. This makes me think that something is wrong.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688816989

thread_type:
official_review

<thread_object>
creator: reviewer_fex4

<content>
summary:
This paper proposed a way for LLM prompting. It first uses a two-stage generation of proxy queries and a way to progressively solve the proxy queries. The method is shown to outperform many baselines in different complex question reasonly tasks.

strengths:
* The prompting method outperforms many baselines for solving complex tasks for LLM, verified under different LLMs
* The method makes sense intuitively.

weaknesses:
* More ablation studies are needed to show the components proposed in this paper are necessary

questions:
The reviewer believes the biggest question related to this paper is the lack of ablation. The reviewers should show the model performances for 
* What happens if we don't include APG questions? (aka, n2=0)
* What happens if we don't generate questions using LLM, but use some gold examples in the dataset?
* Does the order of examples in the final RAG-F stages matter, if we separates easy examples and hard examples?
* For the final RAG-F, can we only retrieve hard examples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721349387

thread_type:
official_review

<thread_object>
creator: reviewer_fhem

<content>
summary:
This paper introduces a novel automatic reasoning prompt approach called LBS3, inspired by the principles of curriculum learning. LBS3 aids large language models (LLMs) in recalling both easy and hard proxy queries related to a target query. Subsequently, it employs a progressive strategy that leverages exemplary prompts derived from easy proxy queries to guide LLMs in addressing hard proxy queries, thereby enhancing the quality of the proxy solutions. Experiments conducted across various reasoning-intensive tasks using both open-source and closed-source LLMs demonstrate that LBS3 achieves competitive performance.

strengths:
1. The paper is well-written and easy to follow.

2. The performance of LBS3 is strong.

3. The idea of LBS3 is simple and effective.

weaknesses:
1. Lack of theoretical contribution. Although the performance of LBS3 is quite promising, its technical contribution compared to existing methods (such as Ana-Pro) seems minor.  

2. To enhance the paper's contribution, it would be advantageous to provide insights into how simpler exemplars can improve LLM's accuracy on more challenging exemplars.

questions:
The analysis lacks a quantitative evaluation of the model's accuracy in responding to hard proxy exemplars, which would demonstrate whether the easy proxy exemplars generated by SPG actually help improve the LLM's performance on harder ones.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732334985619

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We thank all reviewers for their thoughtful and constructive review of our manuscript, and acknowledgment of our contributions. 

**We greatly appreciate your recognition of the strengths of our work as follows:**
- **Introduction of  LBS3**
  - We present the LBS3, as a novel approach, recognized by  **all reviewers as well-motivated.**
  -  Our approach offers a **simple, natural , reasonable and elegant** (fHem, FEX4, 8bhm) framework for solving complex reasoning tasks for LLMs, which is **well appreciated**（8bhm）.
- **Methodological Effectiveness**
  - Our approach has been acknowledged by the **reviewers** fHem and FEX4 for its **effectiveness and strong performance**, verified under different LLMs .
- **Comprehensive Experiments**
  - Our Experiments is acknowledged for its **comprehensiveness and solid** by the reviewer 8bhm, including the **choice of models, datasets, and baselines**. Meanwhile,  the reviewer fHem praises our paper for being well written.

**We've revised our manuscript for the reviewers' suggestions** (**highlighted in red** in the uploaded revision pdf). Detailed responses to each reviewer's concerns are carefully addressed point-by-point.

Below summarize the **major updates** we've made:
- **Experiment**: We further conduct the following experiments to make our paper more sound and try to address reviewers' concerns. Specifically, we investigated the quality of easy-  and hard-proxy queries generated in LBS3 and added **section 4.3.2** in the main paper, and accordingly added **Appendix D** (fHem, 8bhm).
- **Explanation**: Due to space constraints in the main paper, we moved the **Limitations** and **Broader Impacts** within **Conclusion** to Appendix **E** and **F**, respectively.

We believe our work could make a novel contribution to the community and offer a novel perspective on addressing the challenges of complex reasoning tasks of LLM.

We would like to be involved in further discussions if any question is raised.

Best,

Authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732463145290

thread_type:
official_comment

<thread_object>
creator: area_chair_ekdi

<content>
title:
Reminder: Author-Reviewer Discussion Period Closing Soon

comment:
This is a reminder that the author-reviewer discussion period will end on Nov 26 AoE.

Your engagement during this phase is critical for providing valuable feedback and clarifications. If you have any remaining questions or comments, please take a moment to participate before the deadline.

Thank you for your contributions to this important process.

AC
</content>
</thread_object>

</thread>


Metareview: