Automatic Curriculum Expert Iteration for Reliable LLM Reasoning

Authors: Zirui Zhao, Hanze Dong, Amrita Saha, Caiming Xiong, Doyen Sahoo
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: large language models, reasoning, hallucinations, laziness, alignment

Abstract:
Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges in LLM reasoning. Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities. To mitigate hallucination and laziness in reasoning tasks, we propose Automatic Curriculum Expert Iteration (Auto-CEI) to enhance LLM reasoning and align responses to the model’s capabilities--assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits. We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730629539529

thread_type:
official_review

<thread_object>
creator: reviewer_d7jy

<content>
summary:
This paper proposes Automatic Curriculum Expert Iteration (AUTOCEI) to enhance LLM reasoning and align responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them, so as to mitigate hallucination and laziness in reasoning tasks. Through experiments on  BoardgameQA, MATH and Blocksworld with Llama-3.1-8B-instruct, The authors demonstrate the effectiveness of AUTO-CEI, achieving superior alignment by effectively balancing assertiveness and conservativeness.

strengths:
* This paper is well-written and presents clear ideas.

* The idea of aligning responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them is novel.

* The  motivation behind AUTO-CEI is straightforward and strong.

weaknesses:
* AUTO-CEI introduces additianal training overheads. Considering that the process of AUTO-CEI includes EXPERT ITERATION (each iteration needs large amount of resampling), the additional training overhead can not be ignored. I suggest the authors align the sampling cost between different baselines.

* Limited validation across models. The effectiveness of AUTO-CEI is validated only on Llama-3.1-8B-instruct. Further exploration is needed to assess the generalizability to other models.

* The performance is significantly affected by the hyperparameter λ. Table 2 shows that the performance of AUTO-CEI fluctuates greatly under different λ, which can lead to a considerable amount of additional cost during actual usage.

questions:
See weaknesses.

Additionaly,

(1) Have you tested the correlation between the number of reasoning steps and difficulty, as this is a key assumption of AUTO-CEI? If not, I suggest conducting a test experiment on the MATH dataset, considering it has manually labeled difficulty tags.

typos:
line 286 N -> K
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679069216

thread_type:
official_review

<thread_object>
creator: reviewer_hec4

<content>
summary:
This paper enhances LLM calibration ability in reasoning tasks by mitigating hallucinations and reducing lazy behaviors. The method uses a curriculum of expert iteration, balancing answering the question and saying 'I don't know' by adapting the reward system according to the difficulty of reasoning tasks. It shows great balancing in assertiveness and conservativeness.

strengths:
1. The paper presents a reasonable motivation that previous approaches to addressing hallucination problems often suffer from overcorrection, leading to overly conservative responses from LLMs on many questions.

2. The paper introduces an innovative reward mechanism that dynamically balances the choice between responding or refusing to answer based on question difficulty. Subsequent experiments demonstrate that this method effectively handles the trade-off between answering and refusing. Figure 3 also illustrates that as the generated text length increases, more questions are declined.

3. The writing is clear, and the figures are well-designed, enhancing the overall readability of the paper.

weaknesses:
1. The paper’s reward uses the length of the generated text to measure the level of the exploration. This limits the method’s generalizability and scalability, as it’s challenging to find a universal text processing metric for different types of text (e.g., code, tables). Furthermore, reward parameters such as c1 and c2​ also vary across datasets, which makes me concerned about the method’s effectiveness in situations where training and testing distributions are not similar. 

2. The experimental results show that even the best method in this paper achieves overall accuracy significantly lower than Vanilla EI. While I understand that incorporating refusal may lead to a drop in accuracy, achieving performance closer to Vanilla EI would be more compelling, as the current method shows that adding a refusal option will significantly reduce the accuracy.

questions:
1. The MATH dataset has manually labeled question difficulty; it would be helpful to include a figure illustrating the correlation between generated text length and question difficulty as a validation of this signal.

2. Is the method effective across datasets with different distributions?

3. There are many established metrics for measuring uncertainty with different confidence levels, such as AP used in R-tuning. The current paper only reports results for the binary case, i.e., answering or refusing to answer. I am curious about the performance of this method at different confidence levels.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694327141

thread_type:
official_review

<thread_object>
creator: reviewer_4yyx

<content>
summary:
This paper proposes an automatic curriculum expert iteration (AUTO-CEI) to enhance LLM reasoning and align responses to the model's capabilities. Enable the model to answer within its limits and decline when tasks exceed them.
Expert iteration can explore the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness.
AUTO-CEI uses the length of reasoning steps to measure difficulty and designs an automatic curriculum for expert iteration that rewards correct reasoning.
AUTO-CEI can automatically estimates the boundary of LLMs' reasoning capacity to achieve a reasonable alignment to maximize capacity and control behaviors.

strengths:
1. Using expert iteration is a good idea.
2. The authors consider the trade off between avoiding hallucinations and avoiding laziness, which is important.
3. This paper aims to avoiding reasoning hallucinations and teaching the model to say IDK to reasoning tasks which beyond its ability.

weaknesses:
1. Although considering the trade off between helpfulness and laziness, the paper control this trade off by hyper-parameters, instead of proposing some principles or methods to chose the optimal hyper-parameters.
2. The evaluation metrics in the paper are incomplete; for example, IDK only measures the proportion of times the model outputs "IDK" without assessing the accuracy of those IDK responses.
3. The experiments in the paper may involve unfair comparisons, as AUTO-CEI conducts multiple searches for EI, resulting in significantly more training steps and data compared to other methods.

questions:
Why the process of update R is called curriculum learning? IMO, I think it is more like a search process and curriculum learning is about first learning elementary stuffs and then complex stuffs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694929402

thread_type:
official_review

<thread_object>
creator: reviewer_bwmo

<content>
summary:
The paper addresses the problem of hallucinations in reasoning for large language models. Specifically, in methods that use expert iteration for improving this reasoning in LLMs. The authors propose adding a Refusal option to the EI pipeline and rewarding the refusal when the problem has a certain level of difficulty (length of reasoning is used as a proxy for difficulty). Based on the reward that a response gets, the data for the next iteration of EI is selected accordingly. To balance refusal and improvements in reasoning, the authors propose using a curriculum to balance the two objectives. Based on measures of accuracy and refusal rate, the authors show the effectiveness of their method compared to baselines and ablations.

strengths:
- The paper is clearly motivated, the problem of hallucinations in LLM reasoning has not been explored as much.
- The paper presents the problem, the methods and the baselines clearly.
- The results on the three datasets, MATH, blocksworld and boardgameQA are comprehensive and provide sufficient evidence of the utility of the method.
- The authors also present sufficient ablations of their method, with varying versions of the curriculum used.

weaknesses:
- Missing citations for central EI works: STaR (https://arxiv.org/abs/2203.14465), Training Chain-of-Thought via Latent-Variable Inference (https://arxiv.org/abs/2403.04642)
- What other measures of difficulty are possible (very recent paper that the authors can choose to ignore since it came out after the paper deadline: https://arxiv.org/abs/2410.04707)? Can a linear probe decode difficulty? A discussion is needed.
- Similarly, are there other mechanisms of knowing when to say I dont know possible? Is a linear probe enough for refusal? Like in Language models mostly know what they know (https://arxiv.org/abs/2207.05221).
- Where could the current framework of length based difficulty fail? When sampling multiple times, what is the variation in the number of steps needed to reach the answer for a problem?
- Are the number of iterations for Auto-CEI and EI etc matched? More generally, how does the amount of training data / compute required vary across the method and baselines? I think some of the implementation details could be moved to main in the revision. This would help readability.
- How could this method be extended beyond expert iteration, to more online methods of learning like PPO or REINFORCE?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732095770840

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
We thank all the reviewers' efforts in reading our paper and providing valuable feedback! In this message, we summarised the key concerns and suggestions from the reviewers and our responses.

---
**1. Hyperparameters**

Reviewer 4yYx and d7Jy are concerned about the hyper-parameter tuning. 

The mitigation of hallucination and laziness contradict each other. Optimizing precision makes the LLM respond "I don't know" most of the time; minimising the refusal rate makes the LLM try to make an assertive response all the time. Because of this, our objective is not to maximise accuracy but to achieve the sweet spot of balance between assertiveness and conservativeness. 

The hyper-parameter does not affect the training process but determines when to stop the Auto-CEI. Empirically, the user can start an initial hyper-parameter. After it terminates, if the user is unsatisfied with the result, they can further update the hyper-parameter and continue the training process. The previous models can be reused so that no extra computation is involved. 

See our reply to Reviewer 4yYx and d7Jy for more details. 

---
**2. Fair comparison in computation budget**

Reviewers 4yYx and d7Jy point out the computational overhead entailed by our method and are concerned about the unfair comparison. 

The baselines are trained until they converge to a stable result, and the best ones wrt validation are selected for comparison with our method. The baseline (such as EI + R-Tuning) does not require the same computation budget as Auto-CEI to reach its optimal. We observed that vanilla EI converges after 2-3 iterations, as reported in [1]. Thus, extra computation will not help after convergence. In addition, we also tested the case where the R-tuning method was run in multiple iterations. We take the best result among the iterations of baselines to make sure the comparison is fair.  See our reply to 4yYx and d7Jy for more details. 

[1] Teaching large language models to reason with reinforcement learning. 

---
**3. Correlation between difficulty and reasoning steps**

Reviewer BWMo, HEc4 and d7Jy suggest further verifying the correlation between difficulty and reasoning length. 

We use the MATH dataset, which has manually labelled difficulties, to verify the hypothesis. We measure the steps of reasoning with respect to the labelled difficulty. The result suggests that the reasoning length and manually labelled difficulty are correlated with statistical significance. See our reply to BWMo, HEc4 and d7Jy for more details. 

UPDATE: 
Human-labelled difficulty often diverges from computational measures, with LLM-perceived difficulty better aligning with reasoning steps (Pearson's r = 0.349, p < 1e-142). Despite the noise in the MATH dataset's non-standardized steps, Auto-CEI performs robustly, effectively balancing assertiveness and conservativeness. See our reply to d7Jy for more details. 

---
**4. Results across LLMs**

Reviewer d7Jy wishes to verify Auto-CEI's generalisability across different LLM models. 

We conducted additional experiments on Mistral-7B-Instruct-v3 on the BoardgameQA benchmark. The result shows further evidence of our method's generalisability. Due to the time limit, we may only conduct part of the experiments during rebuttal. We will complete the experiments and report the results in future revisions. Our results (demonstrated below) using Llama2 for Proofwriter also show evidence of this generalizability. See our reply to d7Jy for more details. 

---
**5. Out-of-distribution and out-of-domain generalization**

Reviewer HEc4 wish to see the out-of-distribution performance. 

We tested our method in a new logical reasoning dataset (ProofWriter [2]) containing an out-of-domain testing set. Our result shows that it still outperforms the baselines. We used the Llama2 model for this experiment, as it is a pilot study. We eventually chose BoardGameQA, which is more difficult and the latest benchmark. See our reply to HEc4 for more details. 

[2] ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language

---
Thank you again for your valuable feedback! We hope our responses clarify your concerns and increase your impression and confidence in our work. Please let us know if you have further concerns. We look forward to hearing back from you soon!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732566506480

thread_type:
official_comment

<thread_object>
creator: area_chair_as1x

<content>
comment:
Dear Reviewers, 

The rebuttal discussion period is coming to a close and the paper currently has a mix of positive and negative reviewers. The authors have spent a lot of time responding to each concern -- can you take a look at the author responses and let them know any remaining concerns you have?

Best,  
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Automatic Curriculum Expert Iteration (AUTO-CEI) to enhance LLM reasoning and align responses to the model's capabilities—assertively answering within its limits and declining when tasks exceed them, so as to mitigate hallucination and laziness in reasoning tasks. Specifically, the work adds a Refusal option (saying “I don’t know“) to the expert iteration pipeline and rewarding the refusal when the problem has a certain level of difficulty (length of reasoning is used as a proxy for difficulty). The authors proposed to use a curriculum to balance between refusal and answering the question. The paper presents a clear motivation (reducing hallucination, improving reasoning), and describes their methods clearly. The approach is sound and the empirical results on MATH, blocks-world and boardgameQA are comprehensive and provide sufficient evidence of the utility of the method.