Automatic Curriculum Expert Iteration for Reliable LLM Reasoning

## Abstract

Abstract Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to “I don’t know”) persist as major challenges in LLM reasoning.
Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities.
To mitigate hallucination and laziness in reasoning tasks, we propose Auto matic C urriculum E xpert I teration ( Auto-CEI ) to enhance LLM reasoning and align responses to the model’s capabilities–assertively answering within its limits and declining when tasks exceed them.
In our method,
Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate “I don’t know” responses after sufficient reasoning attempts.
The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits.
We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness.

## 1 Introduction

Hallucination is a long-standing issue in large language model (LLM) research, which refers to the phenomenon where LLM-generated content appears plausible but is actually nonsensical or inaccurate, often misleading humans by seeming deceptively credible (Ji et al., ).
It is more evident when solving complex reasoning problems beyond their capability, in which LLM tends to fake evidence or logic to answer the questions assertively.
The reason for LLM’s hallucinations, overall, is a misalignment of its real capability and its behaviours: LLM should not behave overly assertively or confidently when facing unfamiliar (Kang et al., ) and difficult problems (Dziri et al., ). Instead, a preferred behaviour of LLMs is to acknowledge their limitations by responding with “I don’t know.”
However, we must also avoid the LLM from defaulting to degenerate responses like “I don’t know,” especially when the question is within its capability. A reliable LLM should strike a balance between maximizing performance and avoiding hallucination.

Various methods have been proposed to mitigate LLM hallucinations. However, most studies have focused on factual hallucinations, i.e., fabricating non-existent evidence, while often neglecting reasoning hallucinations (Creswell et al., ). Reasoning tasks require deriving conclusions from evidence using rules, often involving multiple steps. Reasoning hallucination refers to the phenomenon where LLMs apply invalid rules or misinterpret the conclusion, leading to an incorrect final result even without factual hallucination in the evidence.
Dziri et al. () found that LLMs exhibit a probability of error at each reasoning step. As these errors compound, the likelihood of incorrect final answers increases exponentially as the reasoning chain grows longer. In addition, most techniques mitigate factual hallucinations using external knowledge or conduct post-training to do “patching”.
These techniques can reasonably mitigate factual errors, but hallucinations caused by faulty reasoning are far more challenging due to the inevitable compounding error of the transformer.
As such, most of these techniques cannot improve LLM’s inherent reasoning capability, and simply using patching techniques alone can even make LLMs behave lazily.

Can LLM learn short reasoning or self-correction to reduce compounding errors and reasoning hallucinations? Kang et al. () showed that learning the optimal shortest solution for difficult problems (e.g., NP-complete problems) has very high sample complexity. Thus, directly learning short solutions for all complex problems is unrealistic.
Havrilla et al. () showed that reinforcement learning (RL) can help improve LLM’s performance in various reasoning tasks. RL methods, such as Expert Iteration (Anthony et al., ) and PPO (Schulman et al., ), explore various LLM reasoning trajectories (i.e., Chain of Thought (Wei et al., )) and use reward function to guide the incorrect trajectories back on track, therefore improving the robustness and reducing the effect of compounding errors. However, compounding errors are inherently impossible to eliminate. Even though RL reduces step-wise errors, the probability of error still increases exponentially with the length of reasoning. As a result, it is necessary for LLMs to acknowledge their limitations, particularly when faced with difficult problems beyond their capability. This raises a key question:

> *How can we assess the limits of LLMs’ reasoning abilities and adjust their response behaviour appropriately to align with these limits?*

To address this, we propose Automatic Curriculum Expert Iteration (Auto-CEI), which simultaneously enhances LLM reasoning and aligns its behaviour to ensure precise answers while acknowledging its limitations. Auto-CEI assumes that the number of reasoning steps required to reach a correct answer provides a reasonable estimate of both the problem difficulty and the limits of LLM reasoning. This assumption is grounded in computational theory, as each reasoning problem has its underlying computational complexity, and each reasoning step corresponds to an elementary computing operation. Learning the reasoning steps (precondition/effect of reasoning rules) has similarly low sample complexity (Kang et al., ). Despite the existence of concise optimal solutions, difficult problems (e.g., NP-complete problems) require extended reasoning to find those solutions. This assumption helps align LLM’s behaviours: easier problems require fewer reasoning steps and are less prone to compounding errors, justifying greater assertiveness from LLMs. Complex problems needing more steps and suffering more compounding errors require more conservativeness.

Figure: Figure 1: Demonstration of the key idea of Auto-CEI. Auto-CEI automatically designs a sequence of curricula and uses Expert Iteration to train LLM in each curriculum. Expert Iteration explores the reasoning trajectories (i.e., Chain of thought) and learns to get incorrect paths back on track to reduce compounding errors. Using the reward (denoted by blue and red colour) produced by the curriculum, Expert Iteration also samples appropriate “I don’t know” responses (IDK responses) after sufficient reasoning attempts (in the blue region). To mitigate laziness, it avoids the short IDK responses (in the red region). The curriculum automatically adjusts rewards for the reasoning length, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its assertive and conservative behaviours with these limits.
Refer to caption: x1.png

The key idea of Auto-CEI is shown in Figure 1: using the length of reasoning steps to measure difficulty, Auto-CEI designs an automatic curriculum for expert iteration that rewards correct reasoning, compensates for “I don’t know” acknowledgements after a sufficient number of attempts, and penalizes both overly conservative and assertive wrong responses. The expert iteration adjusts the LLM’s responses by resampling based on these rewards. The curriculum automatically updates the compensation reward to encourage more reasoning attempts before saying “I don’t know” over time. It gradually adjusts the reward function to optimise an objective function, balancing the overall precision and the proportion of saying “I don’t know” to control hallucination and avoid laziness. As such, it gradually pushes the limits to maximise the potential of LLM reasoning and aligns its behaviours with these limits. Auto-CEI is effective across various reasoning tasks such as logical, mathematical, and planning problems, balancing reliability and conservativeness.

Contributions.
In summary, Auto-CEI automatically estimates the boundary of its reasoning capacity, thus achieving a reasonable alignment to maximise capacity and control behaviours. It learns to reliably solve the problems within its boundary as much as possible; it also knows to acknowledge “I don’t know” when the problem is beyond its limit. We carried out comprehensive experiments in various reasoning benchmarks, and Auto-CEI significantly outperforms the concurrent baseline methods, boosting precision by 10-24% while maintaining a relatively low refusal rate of 18-36% across diverse reasoning tasks in planning, logical and math reasoning.

## 2 Related work

Hallucinations and laziness of LLM.
It is widely acknowledged that LLMs often produce hallucinated responses, including fabricated facts and misleading logic (Radhakrishnan et al., ; Ji et al., ). Most works focus on reducing hallucinations by ensuring the factual accuracy of generated content, often using retrieval-based methods that enhance LLMs with external knowledge (Shuster et al., ). This methodology has been effective in improving the reliability of LLMs in domains that require high factual accuracy, such as knowledge-based question answering (Lewis et al., ).
Particularly, Retrieval-Augmented Generation (RAG) reduces hallucinations by fetching relevant documents from external sources during inference, integrating factual information to enhance accuracy and relevance (Izacard et al., ). These methods ensure that the LLMs’ responses align more closely with verified data (Komeili et al., ; Gururangan et al., ). Moreover, several studies have explored hybrid approaches that combine retrieval with fact-checking modules or domain-specific fine-tuning to improve factual accuracy (Lee et al., ).
Retrieval-based strategies show strong potential in mitigating hallucinations, outperforming purely generative models that rely only on knowledge from training (Borgeaud et al., ; Petroni et al., ).
However, hallucinations are not limited to factual inaccuracies; they can also extend to faulty or incomplete reasoning, a significant concern for multi-step reasoning-based tasks (Creswell et al., ). Moreover, LLMs often exhibit what can be described as “laziness,” which refers to the tendency of the model to reject or avoid generating correct but complex answers in favour of simpler, superficial, or incomplete responses (Bubeck et al., ; Bang et al., ). This phenomenon has been noted in tasks requiring step-by-step logical reasoning, where LLMs tend to skip intermediate steps or provide too general answers, rather than addressing the nuanced complexity of the problem at hand (Rae et al., ).

Reinforcement learning for LLM reasoning.
Reinforcement learning has been widely used in the alignment stage of LLM training to enhance their capabilities. The research in this area can be broadly divided into two aspects: (1) RL algorithms; (2) reward construction.
Conventional advanced deep RL algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al., ), have been extensively applied in the training process of modern LLMs, helping to improve both format and reasoning abilities (Achiam et al., ; Team et al., ; Dubey et al., ).
However, due to the introduction of the value network, the computational overhead of these algorithms is extremely high, often making them unaffordable for academic settings where resources are limited.
More recently, expert iteration (also known as rejection sampling or reward-ranked fine-tuning) (Dong et al., ; Gulcehre et al., ), preference-based RL (Rafailov et al., ; Xiong et al., ), and REINFORCE-type RL (Williams, ; Shao et al., ; Li et al., ; Ahmadian et al., ) have emerged as more efficient alternatives for the alignment stage, especially when operating within a fixed budget. These methods have demonstrated significant improvements over supervised fine-tuning in reasoning tasks (Aksitov et al., ; Gou et al., ; Havrilla et al., ; Shao et al., ; Dong et al., ; Trung et al., ; Xiong et al., ).
Additionally, Lightman et al. () introduced process supervision to refine the reward signal for each reasoning step. Compared to traditional outcome-based supervision, process supervision has been shown to yield substantial gains (Lightman et al., ; Wang et al., ; Shao et al., ).
Despite the impressive progress in this area, most existing works focus on maximizing the upper bound of reasoning ability. In contrast, our algorithm aims to develop a more balanced model incorporating self-awareness into the reasoning procedure.

## 3 Preliminaries

### 3.1 Formulation: Reasoning problems

We address reasoning problems in general, such as logical reasoning, mathematical reasoning, and planning. Reasoning involves deriving unknown facts from provided evidence and reasoning rules. In logical reasoning, evidence is represented as clauses with assigned boolean values, and applying reasoning rules determines the boolean values of unknown clauses. In arithmetic reasoning, evidence consists of clauses linked to scalar values, and reasoning requires both logical deduction and arithmetic operations. In planning problems, the goal is to identify a sequence of actions that move an initial state to a target state by applying actions based on their preconditions and effects.

Unlike methods that convert problems into Boolean satisfiability (SAT) and solve them with SAT solvers, the LLM reasoning approach, such as Chain of Thought (CoT) (Wei et al., ), follows a sequential decision-making strategy. To capture reasoning problems, we rely on planning concepts that consist of four main elements: state space $S$, action space $A$, transition function $T$, and goal function $G$. The state space $S$ includes all possible configurations, such as the boolean or scalar values assigned to the given variables or clauses, as well as unknown clauses. Actions, defined in the action space $A$, represent rules or equations that can be applied to these states. The action has its preconditions and effects. A state should satisfy the corresponding precondition to execute an action, and then the effects will be applied to the state. The precondition and effect are encoded in the transition function. When an action $A$ is executed in a state $S$, the transition function $T(s,a)$ determines the resulting state $s^{\prime}$, e.g., new values would be assigned to (unknown) clauses or variables. The goal function $G$ checks whether the current state matches the desired outcome.

### 3.2 Backgrounds

Expert Iteration. Expert iteration (EI) (Anthony et al., ) is an iterative process in which experts are built upon a base policy, the base policy is refined by learning from the expert, and this cycle repeats with newly derived base policies. EI is effective in improving the quality of generated responses in LLM literature
(Dong et al., ; Gulcehre et al., ).
Havrilla et al. () discussed the usage of EI in LLM reasoning problems. They used LLM sampling to generate responses, and preferred responses were selected by a ground-truth or reward model to build the expert. They then used LLM to imitate the expert via SFT. EI explores the neighbouring area of LLM policy in the token space, which helps LLM derive back to the correct reasoning trajectories if there are errors or find suboptimal trajectories. As a result, in their experiment, EI improved the LLM reasoning over time, which performed similarly or even better than the PPO (Schulman et al., ).

Curriculum Reinforcement Learning.
Curriculum Reinforcement Learning (Narvekar et al., ) focuses on sequencing tasks or data samples into a structured curriculum to solve complex problems that are difficult to learn from scratch. A curriculum can be considered an ordering over experience samples at the most basic level. The underlying assumption is that learning from elementary samples or tasks can be transferred and help learn more complex tasks. In this paper, we design the curriculum by reshaping the reward to gradually guide the LLM in conducting more reasoning attempts to solve the problem before acknowledging its limits. Its attempts could be used to discover suboptimal trajectories that mitigate its compounding errors in easy problems and also transferred to increase the chances of solving more complex problems.

## 4 Automatic Curriculum Expert Iteration

Auto-CEI aims to estimate and push the limits of LLM reasoning concerning the number of reasoning steps, maximising its capacity while aligning its behaviours accordingly. We assume that, for a broad class of reasoning problems, the number of reasoning steps (in the chain of thoughts) required to reach the correct answer estimates the problem’s difficulty and the limits of LLM reasoning. We justify the assumption by computational theory. The reasoning problem has its underlying computational complexity, where each of the reasoning steps can be treated as an elementary computing operation. Learning to apply the preconditions and effects of those elementary reasoning actions/rules has low sample complexity (Kang et al., ). Although problems may have concise optimal solutions, given the inherent computational complexity (e.g., NP-complete), finding that solution still needs long reasoning attempts. Auto-CEI estimates the reasoning limits in terms of the number of reasoning steps until the LLM can respond reliably, solving a maximum number of problems. But beyond this point, further continuing the same assertive behaviour of the LLM policy leads to a significant increase in mistakes, whereas being overly conservative within this range limits reasoning capacities due to the high refusal rate. Thus, it estimates the LLM’s capability’s boundary when fine-tuned for multi-step reasoning tasks.

Auto-CEI uses the average number of the reasoning steps of the initial (after supervised finetuning, SFT) policy to determine the curriculum, i.e., the reward function. It starts from optimising an initialised policy that is reasonably conservative with a certain distribution to say, “I don’t know.” It then runs into the Expert Iteration process, in which the LLM policy will sample many reasoning trajectories and receive various rewards. We build up the expert by resampling the trajectories according to the reward and removing the assertively wrong responses. After convergence, Auto-CEI updates the reward function according to the performance, in a way to encourage the LLM to produce more reasoning steps before saying “I don’t know.” The curriculum eventually tries to optimise an objective function, which balances assertiveness and conservativeness and stops the curriculum if the objective function reaches the optimal value.

### 4.1 Initialisation

Figure: Algorithm 1 Initialisation($D,\pi$)

We adopt R-Tuning (Zhang et al., ), one of our baseline methods, as the initialization strategy. R-Tuning produces a good starting point with a reasonable proportion of refusal behaviours. It also makes LLM lazier and has a lot of room for optimisation. We first use SFT to train a language model given the training dataset $D$ consisting of $n$ question-answer pairs. Each answer uses the chain of thought. After SFT, we will let the language model answer the questions in the same dataset $D$ using random sampling. As we use random sampling, there will be a certain number of correct answers, and the others will be wrong. Thus, we split the questions and the new answers generated by LLM into two datasets, $D_{1}$ and $D_{2}$, where $D_{1}$ collects correct answers and $D_{2}$ consists of wrong answers. For $D_{2}$, in each of the wrong answers, we add an expression at the end to acknowledge the limitations and abstain from answering the questions, such as “Sorry, I am not sure if I answer the question correctly. There might be mistakes in my answer.” Thus, we have collected a new Refusal-Aware dataset $D_{2}$ for our finetuning.

After collecting $D_{1}$ and $D_{2}$, we concatenate both datasets to form $D_{\mathrm{init}}$ and use SFT to fine-tune the LLM again. We also examined the SFT result to ensure it had enough sample points for assertive and refusal answers. We set a threshold of 25% for the distribution of refusal behaviours in the validation set, as we need to have enough variety of acknowledge “I don’t know” for Auto-CEI. If the model has a very low distribution of acknowledge “I don’t know”, we then repeat the previous process of collecting $D_{1}$ and $D_{2}$ again, where the refusal answers will also be collected in $D_{1}$ and use the new dataset for a new SFT training. We observed that, empirically, the distribution of refusal answers would increase over the steps. Thus, after the initialisation, we will have a good initial policy with enough base knowledge to answer questions and a distribution of acknowledging “I don’t know.”

### 4.2 Expert Iteration

Figure: Figure 2: The shape of $R(x,y)$ for $y$ is a refusal answer.

Given the initial policy, we run the Expert Iteration to improve the policy over time.

We build the expert using the reward function and resampling.
The reward function $R(x,y)$ takes the question $x$ and an answer $y$ as input and outputs a scalar. $R(x,y)$ is designed to give a large positive reward for correct answers, a small compensatory reward for refusal answers after long reasoning trajectories, a small negative penalty for refusal answers after very short reasoning trajectories, and a large negative penalty for assertive wrong answers. The reward is designed as follows:

$$ $R(x,y)=\left\{\begin{array}[]{ccl}1&\mbox{for}&y~{}\mbox{correctly answers}~{} x~{};\\ \frac{1-\exp(-c_{2}(\text{len}(y)-c_{1}))}{1+\exp(-c_{2}(\text{len}(y)-c_{1})) }&\mbox{for}&y~{}\mbox{is a refusal answer}~{};\\ -1&\mbox{for}&y~{}\mbox{wrongly answers}~{}x~{}.\end{array}\right.$ (1) $$

In this function, $c_{1}$ and $c_{2}$ are two hyperparameters determined by the distribution of answers from the initial LLM. If $y$ is a refusal answer and the number of steps for the reasoning trajectory $\text{len}(y)$ is longer than $c_{1}$, then it will receive a small positive compensation reward; otherwise, it turns into a small negative penalty. The shape of this function is shown in Figure 2. $c_{1}$ is initialised by the mean value of reasoning steps produced by the initial LLM policy in the validation set. $c_{2}$ is computed by solving $\frac{1-\exp(-c_{2}\cdot 2\sigma)}{1+\exp(-c_{2}\cdot 2\sigma)}=0.9$, meaning that if the number of reasoning steps has reached $c_{1}+2\sigma$ ($\sigma$ is the standard deviation of the reasoning steps by initial policy), then the reward should be higher than $0.9$. It means the answer is longer than roughly 97% of the reasoning trajectories in the validation set, assuming the distribution of reasoning steps is a normal distribution.

Figure: Algorithm 2 ExpertIter($D,\pi_{\mathrm{ei}},R,D_{\mathrm{val}},\pi$)

The reward function helps us build the expert, enabling us to improve LLM reasoning performance over time. We resample the responses according to the rewards to build the expert. We first use LLM to generate $K$ samples $\{y_{i}\}_{i=1}^{K}$ for each of the questions in the training dataset $D$.
Then, we use the reward function to get a reward value of each sample $r_{1},r_{2},\ldots,r_{K}$. Over that reward, we employ the temperature-scaled softmax function to construct a new distribution: $p^{\prime}(y_{i})=\exp(r_{i}/\tau)/\sum_{i}\exp(r_{i}/\tau)$ and subsequently use $p^{\prime}(y_{i})$ to conduct resampling and generate $N$ new samples $\{y^{\prime}_{i}\}_{i=1}^{K}$. $\tau$ is the temperature of resampling, the same as the overall accuracy of the initial SFT model finetuned by $D$. If the SFT model has high overall accuracy, we will encourage LLM to explore more randomly; otherwise, the responses with higher rewards will be sampled more densely. $\tau$ will be capped in a range $[0.4,0.7]$ to avoid extreme cases. Thus, the distribution is reshaped to encourage more instances that produce correct answers and produce more reasoning steps before saying “I don’t know.” Since resampling also kills instances that waste some of the exploration data, we concatenate the $\{y_{i}\}_{i=1}^{K}$ and $\{y^{\prime}_{i}\}_{i=1}^{K}$ together to form the new training dataset. Lastly, we removed the answers that had $-1$ rewards.

### 4.3 Curriculum update

Figure: Algorithm 3 Auto-CEI($D,\pi,R,f,D_{\mathrm{val}}$)

After ExpertIter converges, the curriculum updates the reward function to optimise (i.e. maximise) an objective function:

$$ $f=(1-\lambda)P_{\text{Pre}}+\lambda(1-P_{\text{IDK}}),$ (2) $$

where $P_{\text{Pre}}$ denotes the precision, i.e., the correctness of answers that are not refusal answers, and $P_{\text{IDK}}$ denotes the proportion of refusal answers. This function $f$ tries to find a good balance between avoiding both hallucination (low $P_{\text{Pre}}$ and low $P_{\text{IDK}}$) and laziness (high $P_{\text{Pre}}$ but high $P_{\text{IDK}}$) when finetuning the LLM.
$\lambda\in[0,1]$ is a hyperparameter to control the tradeoff between hallucination and laziness; a higher value of $\lambda$ will lead to more assertive behaviour (i.e. lower IDK rate) while smaller $\lambda$ would make the LLM policy behave more conservatively. We empirically show that setting $\lambda$ is fairly intuitive; in most task scenarios, setting $\lambda$ to a reasonably small value ($\approx$ 0.2) achieves the desired effect for the LLM policy. Over the EI iterations, the curriculum will update the value of $c_{1}$ in Equation 1, i.e., try to push $c_{1}$ to encourage more exploration before saying “I don’t know”. Thus, with the new reward function, we repeat the Expert Iterations and make it converge again.

Assuming that the function of $f$ does not have local optimal point wrt $c_{1}$, the curriculum uses local search (hill climbing) (Russell & Norvig, ) to search $c_{1}$ and conduct Expert Iterations until it finds the highest $f$. The hill-climbing algorithm searches its neighbouring value of $c_{1}\pm d$ via a step size $d$, and if $f$ at the new step is higher, then it updates $c_{1}$ to the new value. It will stop the search if its neighbouring $c_{1}$ has a smaller value than its current $f$. The domain of $c_{1}$ is $[\mu-2\sigma,\mu+2\sigma]$ where $\mu$ denotes the average length of the reasoning steps produced by the initial LLM policy. $d$ is defined by $\mathrm{min}\{0.5,4\sigma/10\}$, where $\sigma$ denotes the standard deviation of the reasoning length produced by the initial LLM policy. We also do not want to make the step too large, thus capping it by a threshold value of $0.5$, an empirically selected value.

Overall, the curriculum update, together with initialisation and Expert Iteration of Auto-CEI, are in the pseudo-code listed in Algorithm 3.

## 5 Experiment

### 5.1 Experimental Setup

Benchmarks.
To demonstrate the effectiveness of Auto-CEI in reasoning tasks, we select BoardgameQA (Kazemi et al., ), MATH (Hendrycks et al., ), and Blocksworld (Valmeekam et al., ) as benchmarks, spanning from logical and mathematical reasoning to planning. They have various domains and complexities. We briefly introduce the benchmarks and report our detailed experimental settings in the Appendix B.

- •
BoardgameQA is one of the latest benchmarks for logical reasoning. The data is synthesised from formal logical reasoning rules and clauses. The problem provides a self-contained context, pieces of evidence, reasoning rules, and questions. Given the evidence and reasoning rules, LLM must decide if the queried clause is true, false, or unknown. Moreover, BoardgameQA has cases where some evidence might contradict others. To deal with this case, BoardgameQA also provides preferences among the reasoning rules, meaning that the conclusions drawn from the preferred rules have higher priority. In addition, since BoardgameQA does not provide CoT data for unknown questions, we use GPT-4 to generate CoT reasoning trajectories for unknown problems for the training dataset. Thus, the training data is consistent. The prompt for data generation is in the Appendix B.1.
- •
MATH is a challenging mathematical reasoning benchmark, including advanced-level algebra and geometry. In each of the problems, the necessary evidence is provided in the context, and LLM is required to learn correct mathematical theorems and rules to connect the pieces of evidence in the given problem and draw conclusions. It typically requires long reasoning steps in which each mathematical theorem and rule is correctly applied to get the correct answers.
- •
Blocksworld is a symbolic planning problem. It defines a domain with a few blocks in different colours. The problem requires the LLM to rearrange the blocks step-by-step and make the configuration of the blocks satisfy some preferred constraints. Concretely, it requires the LLM to output a sequence of pick and place behaviours, which transform the current configuration of blocks to the goal configuration. Blocksworld has been proved to be NP-hard in finding optimal (i.e., shortest) planning trajectories (Gupta & Nau, ), while suboptimal policies might be easier to learn.

Baselines.
To validate the effectiveness of Auto-CEI, we conduct extensive experiments comparing it against multiple baselines.

- •
SFT. We perform supervised fine-tuning on the original training dataset $D$ and use the overall accuracy as a reference metric.
- •
Vanilla EI. As noted by Havrilla et al. (), Expert Iteration achieves strong results in enhancing reasoning capabilities. We apply Expert Iteration to improve LLM reasoning, evaluating its performance by retaining only correct and assertive answers during the iteration process.
- •
SFT + R-Tuning (Zhang et al., ). This is an SFT-based post-training method for hallucination mitigation. Please see our initialisation part in Section 4.1 for details.
- •
EI + R-Tuning. This baseline uses Vanilla Expert Iteration to boost the performance of LLM reasoning first and then uses R-Tuning for post-training.
- •
RLKF (DPO). Reinforcement Learning from Knowledge Feedback (RLKF) (Xu et al., ) is an RL-based post-training method for hallucination mitigation. It aligns the LLM’s behaviours according to the consistency and correctness of LLM’s sampled responses: it teaches LLM to respond assertively if its responses are correct and consistent and acknowledges “I don’t know” if its responses are mainly wrong or inconsistent. We implemented a DPO version of this method. Please see the Appendix for details.

Metrics.
We evaluate LLM’s responses’ overall accuracy, precision, and refusal rate (refusal rate). The overall accuracy measures the overall performance of LLM in a specific reasoning task. A higher accuracy means the LLM can solve more problems. We compute the accuracy by $P_{\mathrm{Acc}}=\frac{\text{Number of correct responses}}{\text{Number of all
responses}}$. The precision measures the accuracy of LLM when it is willing to answer the questions assertively. It reflects the overall reliability of LLM reasoning. We compute the precision by $P_{\mathrm{Pre}}=\frac{\text{Number of correct responses}}{\text{Num of all
responses}-\text{Num of IDK}}$. In addition, the hallucination (error) rate refers to
$\frac{\text{Number of non-refusal but incorrect responses}}{\text{Num of all
responses}-\text{Num of IDK}}$.
The refusal rate measures how conservative the LLM policy is. It is computed by $P_{\mathrm{IDK}}=\frac{\text{Num of refusal responses}}{\text{Num of all
responses}}$. A good LLM policy should have reasonable precision and refusal rate balance, i.e., a high precision and a reasonable refusal rate according to the difficulty of the task.

Implementation details.
We use Llama-3.1-8B-instruct (Dubey et al., ) as the backbone model and use Lora ($r=128,\alpha=64$) to fine-tune. The experiments are conducted in a server with 8$\times$Nvidia A100 (40GB) GPUs. We use DeepSpeed Stage 2 to conduct the training. See Section C in Appendix for more details about the Auto-CEI setup.

### 5.2 Results

The main result is reported in Table 1. In this result, Auto-CEI produces high precision (Pre) and keeps a reasonably lower refusal rate (IDK) across all tasks. It also has the highest objective function value $f$ defined by Equation 2 ($\lambda=0.2$). We summarize our findings in the following paragraphs.

**Table 1: Main results. Acc: accuracy (%); Pre: precision (%); IDK: refusal rate (%); $f$: objective function.**
|  | BoardgameQA | MATH | Blocksworld |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc | Pre | IDK | $f$ | Acc | Pre | IDK | $f$ | Acc | Pre | IDK | $f$ |
| SFT | 60.23 | 60.23 | – | – | 38.65 | 38.65 | – | – | 51.99 | 51.99 | – | – |
| Vanila EI | 67.77 | 67.77 | – | – | 42.04 | 42.04 | – | – | 71.04 | 71.04 | – | – |
| SFT + R-Tuning | 55.23 | 80.36 | 31.27 | 0.780 | 19.49 | 60.67 | 67.88 | 0.550 | 38.54 | 90.69 | 57.51 | 0.811 |
| EI + R-Tuning | 57.76 | 80.77 | 28.49 | 0.789 | 29.57 | 55.80 | 47.02 | 0.552 | 67.92 | 93.95 | 27.71 | 0.896 |
| RLKF (DPO) | 49.32 | 54.17 | 8.95 | 0.615 | 40.02 | 42.19 | 5.14 | 0.527 | 35.24 | 38.08 | 8.21 | 0.486 |
| Auto-CEI (Ours) | 59.70 | 84.52 | 29.37 | 0.817 | 35.56 | 55.63 | 36.08 | 0.575 | 74.78 | 91.53 | 18.30 | 0.896 |

EI improve LLM reasoning. In our experiments, the Vanila EI produced higher overall accuracy than SFT in all tasks, which indicates the overall capacity of LLM in reasoning is improved. This improvement is because EI helps LLM to sample various trajectories and collect those who draw a correct solution and learn those solutions; for the next iteration of EI, it samples more trajectories near those correct trajectories and keeps collecting and learning. Over time, LLM learns to start from various trajectories and still get to the correct solution, even though some might be suboptimal. Thus it becomes more robust to the randomness in token sampling when generating responses and the resulting compounding errors.

R-Tuning makes LLM over-conservative. We found that SFT + R-Tuning and EI + R-Tuning all show the trend of over-conservativeness (i.e., laziness). The results of both baselines have a relatively high refusal rate (IDK) and lower overall accuracy. The low overall accuracy indicates that LLM’s capacity has been limited, and LLM produces refusal responses to many problems that lie within the LLM’s capability.

RLKF degrades reasoning performance. Reported by Xu et al. (), RLKF performs well for short, knowledge-grounded arithmetic problems. However, for reasoning problems, the responses generally become much longer for more complex questions. This causes a few additional issues for RLKF techniques. First, the LLM finds it hard to distinguish the correct chosen/rejected responses given the long response length if the dataset is not very large; thus, the reward accuracy is relatively low (it only has 30% accuracy for Blocksworld and 70% for BoardgameQA and MATH). Second, the original strategy of reward shaping greedily maximizes the reward for correct answers and minimises incorrect or refusal responses. As such, the LLM will be over-assertive. As it greedily imitates the optimal responses guided by the inaccurate reward, it will suffer more from the compounding error when facing the unseen context of the reasoning problems, thus degrading its performance.

Auto-CEI improves the LLM reasoning. The result shows that the overall accuracy of Auto-CEI is higher than SFT + R-Tuning. Since our method initialises its optimisation using SFT + R-Tuning, this result indicates that Auto-CEI improves LLM’s reasoning capacities. It is because Auto-CEI gradually adjusts the curriculum to encourage LLM to solve more problems within its capacities and only accept refusal responses after sufficient reasoning attempts. It thus avoids the LLM’s laziness in easy problems and improves the robustness of the LLM’s problem-solving when it samples suboptimal trajectories. The improvement of precision and refusal rate within a few expert iterations is shown in Figure 4 of the Appendix.

Figure: (a) BoardgameQA
Refer to caption: x2.png

Auto-CEI estimates LLM’s limit in reasoning and produces the best alignment between its behaviours and its limit. We compare the error (hallucination) rate and refusal rate (IDK) according to the response length. Figure 3 shows that the error rate of SFT and EI grows exponentially with the increase in response length. On the other hand, we can also see that Auto-CEI has a relatively uniform low rate of hallucination for different response lengths, and its refusal rate (refusal rate) grows according to the error rate of SFT/EI, as is ideally expected. This result indicates that Auto-CEI is indeed able to estimate LLM’s current capability limit and is further capable of reaching a reasonable alignment between its assertive and conservative behaviours according to its limit. It thus behaves assertively for problems within its limit (low refusal rate for short reasoning length) and conservatively for problems beyond its limit (high refusal rate for long reasoning length). Overall, it produces reliable reasoning while simultaneously maintaining maximum reasoning ability.

### 5.3 Ablation study & Discussion

We conduct an ablation study to verify the effectiveness of our solution design choices and further discuss our underlying assumptions. As the effect of Expert Iterations has been demonstrated in the main result, in this section, we mainly focus on the effect of curriculums and the practical usage of hyper-parameter $\lambda$. The result of the ablation study is shown in Table 2.

**Table 2: Ablation study. Acc: accuracy (%); Pre: precision (%); IDK: refusal rate (%).**
|  | BoardgameQA | MATH | Blocksworld |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Acc | Pre | IDK | Acc | Pre | IDK | Acc | Pre | IDK |
| No Curriculum | 56.10 | 85.56 | 34.43 | 27.14 | 52.06 | 47.86 | 68.43 | 93.42 | 26.75 |
| $\lambda=0.5$ | 59.57 | 81.12 | 26.56 | 37.62 | 49.90 | 24.61 | 79.31 | 84.25 | 5.86 |
| $\lambda=0.2$ | 59.70 | 84.52 | 29.37 | 35.56 | 55.63 | 36.08 | 74.78 | 91.53 | 18.30 |

Effect of $\lambda$. The hyperparameter $\lambda$ decides the optimal objective function and when to stop the optimisation. A higher $\lambda$ will try to minimise the refusal rate (refusal rate) and thus make the LLM more assertive. The lower $\lambda$ will try to focus on maximising precision and tends to keep a higher refusal rate, making the LLM policy more conservative. The $\lambda$ does not affect the overall training process in Auto-CEI, and the decision of the hyperparameter is mainly determined by the user’s demand. Empirically, $\lambda=0.2$ has a reasonable balance between the precision and refusal rate. In practice, we suggest the user start by $\lambda=0.2$ and further adjust the parameter according to the demands. For example, a lower $\lambda$ would be preferred in highly risky tasks, whereas a higher $\lambda$ is better for training search heuristics that require more assertive behaviours, a higher $\lambda$ is better.

Effect of curriculum. We aim to use the curriculum to push the limit of LLM reasoning, thus achieving a good alignment of its behaviours and maximizing its reasoning ability simultaneously. We compare the ablation that does not gradually update the reward function defined in Equation 1. The result in Table 2 (No Curriculum) shows that LLM converge to a suboptimal point where its overall accuracy is lower and its refusal rate is higher. In addition, there might be a chance that LLM directly reaches the optimal $f$ after one curriculum with some other choices of $\lambda$. However, as discussed previously, the choice of $\lambda$ is determined by the user’s demand, and the optimisation conducted by the curriculum should be able to find the optimal point specified by different $\lambda$.

## 6 Conclusion

This paper focused on mitigating LLM’s hallucinations and laziness in reasoning tasks. We propose Automatic Curriculum Expert Iteration (Auto-CEI) to push the limits of LLM’s reasoning capacities and align its assertive and conservative behaviours according to these limits for reliable reasoning. We conduct experiments on various challenging reasoning benchmarks, including logical reasoning, mathematics, and planning. Our result suggests that Auto-CEI maximizes its reasoning ability while achieving a superior alignment between response behaviours and this ability. The fundamental assumption under Auto-CEI is that solving difficult reasoning problems requires more reasoning steps, thus making the number of reasoning steps a good estimation measure of difficulties. Recent research works by OpenAI () and Snell et al. () suggest that a longer reasoning length in the inference time scales up the LLM reasoning ability and enables it to achieve expert-level capabilities. These researches provide evidence for our assumption. Meanwhile, they also show the chance that our method can also be applied to improve the reliability of those language models with super-long reasoning lengths.

#### Limitations.

This work focuses on challenging reasoning problems, where complexity can be estimated by the number of reasoning steps.
For simpler problems with shorter reasoning paths or well-studied datasets (e.g., GSM8K), R-Tuning reliably performs well, leaving limited scope for further optimization. This is partly due to the reduced impact of compounding errors in these easier tasks. We also found that the model might decline to answer even if its response is correct. Thus, further adjustment and fine-tuning should be applied to solve this issue, and our method should be able to be extended accordingly as well. In addition, our method might require a long time for optimisation as it requires running Expert Iteration multiple times. Nonetheless, compared to the cost of pre-training, the overhead of the iterative post-training algorithm remains worthwhile for achieving better alignments.

## Reproducibility Statement

In Section 4, we provide pseudocode for our algorithm and explain the details of each part. We provide sufficient details in Appendix Section C for our implementation of Auto-CEI and training settings. For our experiment benchmarks, we introduce our benchmark selection and metric computation in Section 5.1 and provide the link to download the dataset or the source code from the published papers to generate the dataset, the format of the dataset, and the dataset size we choose in Appendix Section B. We also provide the baseline methods’ details, implementation and training in Appendix Section D.

Codes are available upon request and can be shared privately before the public release.

## Appendix A Improvement in Expert Iteration

Figure: Figure 4: Improvement within EI. The precision and refusal rate converge after 3 iterations.
Refer to caption: x5.png

## Appendix B Experimental Settings

### B.1 BoardgameQA

For boardgameQA, we use the main dataset to do fine-tuning and evaluation (^1^11The dataset is downloaded from [https://storage.googleapis.com/gresearch/BoardgameQA/ BoardgameQA.zip](https://storage.googleapis.com/gresearch/BoardgameQA/BoardgameQA.zip)). We take the dataset under the folder BoardgameQA-Main-depth1, BoardgameQA-Main-depth2 and BoardgameQA-Main-depth3.

We show one example of the problem in BoardgameQA in Listing 1. For more details about the dataset itself, please refer to the paper by Kazemi et al. ().

Figure: Listing 1: BoardgameQA Problem Example

In our experiment, we put the example as the input prompt and verify if the final answer is the same as the label. The proof provides the chain of thought inference data, which is used in our fine-tuning.

#### BoardgameQA data argumentation

For the problems whose label is unknown, the proof only provides one simple statement: The provided information is not enough to prove or disprove the statement “{the query statement}”. This make the responses inconsistent with other responses whose label is proved or disproved. Thus, we use GPT-4 to generate the chain of thought data for unknown cases. The prompt of our data generation is provided in the Listing 2.

Figure: Listing 2: Prompt for data generation

When using GPT-4 to generate the data, we choose temperature=0.6 and top_p=0.95. The model we selected is gpt-4-turbo.

### B.2 MATH

We download the MATH dataset via Hugging Face at [https://huggingface.co/datasets/
hendrycks/competition_math](https://huggingface.co/datasets/hendrycks/competition_math). It automatically divide the dataset into training, validation, and test set. We used all the training data in our experiment and evaluated it using all validation and testing sets to check if the final answer was the same as the labelled data. For more details about the dataset itself, please refer to the paper by Hendrycks et al. (). One example format of the dataset is shown in the Listing 3.

Figure: Listing 3: Example data point of MATH dataset

### B.3 Blocksworld

The Blocksworld dataset can be generated using the code in the GitHub repository by Valmeekam et al. (): [https://github.com/karthikv792/LLMs-Planning](https://github.com/karthikv792/LLMs-Planning). For our case, we generate domains from 4 blocks to 6 blocks, and randomly samples 500 data points for training. We randomly sample 500 data points for validation and testing sets whose optimal solution length (i.e., ground truth plan) is no longer than ten steps. We uniformly sample the tasks according to the ground truth lengths to form the testing set (i.e., 100 two-step tasks, 100 four-step tasks, …, and 100 ten-step tasks).

The original format of the chain of thought for Blocksworld planning is unnecessarily long, therefore making it difficult to learn a sufficient policy. We modified the format of the chain of thought to make it more concise and informative. The elementary format is shown below:“{problem_description}. Since {precondition_text}. Thus, we take action {action} …Since {precondition_text}. Thus, we take action {action}. Since the goal is GOAL. The goal conditions are satisfied. ###”. One example of the dataset format is shown in the Listing 4.

Figure: Listing 4: Example of Blocksworld dataset

For more details about the dataset, please refer to the paper by Valmeekam et al. ().

## Appendix C Implementation details

### C.1 Auto-CEI Setting

#### Neural Network Model

We use Llama-3.1-8B-instruct (Dubey et al., ) as the backbone model and use Lora ($r=128,\alpha=64$) to fine-tune. The Llama-3.1-8B-instruct model is downloaded at [https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). We use the Fastchat template designed for Llama-3.1-8B-Instruct. The template is demonstrated in the Listing 5. We fill in the Prompt by our query and train the LLM to respond by Completion.

Figure: Listing 5: Chat Template

#### Initialisation

As suggested in the paper, we used R-Tuning (Zhang et al., ) as our initialisation strategy. Overall, we use LLM to generate various responses via random sampling. It will produce correct and incorrect responses with intermediate reasoning steps. We collect those incorrect responses and attach an expression of acknowledging limitations. In practice, we use temperature=1.0 and top_p = 0.95 to sample responses. We sample 16 responses in our experiments for each query. The expression of acknowledging limitation is “Sorry, I am not sure if I answer the question correctly. There might be mistakes in my answer”.

#### Expert Iteration

In this process, the language model will keep sampling responses, and we select the correct responses or have sufficient reasoning attempts before acknowledging incapability. We use temperature=1.0 and top_p = 0.95 to sample responses and sample 16 responses for each query.

Expert Iteration resamples the response according to the reward function defined in Equation 1. In this function, $c_{1}$ is initialised by the mean value of reasoning steps produced by the initial LLM policy in the validation set, and $c_{2}$ is computed to make sure that if the number of reasoning steps has reached $c_{1}+2\sigma$ ($\sigma$ is the standard deviation of the reasoning steps by initial policy), then the reward should be higher than $0.9$. It means the answer is longer than roughly 95% of the reasoning trajectories in the validation set, assuming the distribution of reasoning steps is a normal distribution. We use the keywords to identify the length of the reasoning steps. We set a fixed format for different reasoning tasks, and each task will have a marker for length identification. In general, we use the keyword “Since” to identify the number of reasoning steps for the Blocksworld task, and “$.\backslash$n” for the BoardgameQA and MATH tasks.

#### Curriculum Update

The curriculum updates the reward function gradually. Specifically, it gradually increases or reduces the value of $c_{1}$ defined in Equation 1. We use the hill-climbing algorithm to search for the optimal value of the objective function $f$ under the assumption that there is no suboptimal value of $f$ wrt $c_{1}$. We define the steps of updating $c_{1}$ by $\mathrm{min}\{0.5,4\sigma/10\}$, where $\sigma$ denotes the standard deviation of the reasoning length produced by the initial LLM policy. We assume that the domain is $[\mu-2\sigma,\mu+2\sigma]$ where $\mu$ denotes the mean length of the initial LLM policy, and the domain determines the step size of updating $c_{1}$. We also do not want to make the step too large, thus capping it by a threshold value of $0.5$, an empirically selected value. As such, the hill climbing algorithm will search the neighbouring value of $c_{1}$ according to the step size $d$ and identify if the neighbouring point has a larger $f$ value; otherwise, we believe $c_{1}$ has reached its optimal point to achieve the highest $f$.

### C.2 Training details

The experiments are conducted in a server with 8$\times$Nvidia A100 GPUs. We use DeepSpeed Stage 2 to conduct the training (^2^22The configuration of DeepSpeed Stage 2 can be found at [https://github.com/huggingface/transformers/blob/ main/tests/deepspeed/ds_config_zero2.json](https://github.com/huggingface/transformers/blob/main/tests/deepspeed/ds_config_zero2.json)). During the SFT stage, we choose the batch size = 256, and epoch number is 5 for initial SFT process, and 2 for SFT in R-Tuning and EI process. The learning rate of SFT is $1\times 10^{-4}$. We use the HuggingFace SFT trainer(^3^33[https://huggingface.co/docs/trl/v0.11.1/en/sft_trainer#trl.SFTTrainer](https://huggingface.co/docs/trl/v0.11.1/en/sft_trainer)) to conduct the SFT training automatically.

## Appendix D Details of baseline methods

### D.1 R-Tuning

The setting is the same as Auto-CEI in the initialisation part of Section C.1.

### D.2 RLKF

Reinforcement Learning from Knowledge Feedback (RLKF) (Xu et al., ) is an RL-based post-training method for hallucination mitigation. It aligns the LLM’s behaviours according to the consistency and correctness of LLM’s sampled responses: it teaches LLM to respond assertively if its responses are correct and consistent and acknowledges “I don’t know” if its responses are mainly wrong or inconsistent. We implemented a DPO version of this method as one of our baseline.

#### Data collection

Xu et al. () define the pairwise preference to define the reward function:

$$ $\mathrm{pair}=\left\{\begin{array}[]{lcl}(x,y_{c})>(x,y_{r}),&\mbox{for}&\mbox {All samples are correct};\\ (x,y_{c})>(x,y_{w}),&\mbox{for}&\mbox{There exists correct response in samples };\\ (x,y_{r})>(x,y_{w}),&\mbox{for}&\mbox{All samples are wrong}.\end{array}\right.$ (3) $$

In this equation, $x$ refers to the query, and $y$ means the response sampled from LLM. $y_{c}$ denotes the correct response, $y_{w}$ denotes the wrong responses, and $y_{r}$ denotes the refusal response (acknowledging “I don’t know”). We use this strategy to build up the dataset. Since we do not have a special out-of-domain dataset in our setting, we only build the reliable in-domain preference dataset. We sample the LLM using temperature=1.0 and top_p=0.95, and sample 16 instances for each query. The template refusal responses in RLKF is the same as the R-Tuning. If all responses are correct, we will annotate all 16 responses using the refusal template to form 16 pairs of preference data. If there are correct and incorrect responses, we sample 16 preference pairs randomly. If all samples are wrong, we use all of the wrong responses to generate the refusal responses and form 16 pairs of preference data.

#### Training

We use the TRL DPOTrainer(^4^44[https://huggingface.co/docs/trl/main/en/dpo_trainer](https://huggingface.co/docs/trl/main/en/dpo_trainer)) to conduct the training. Similar to the previous setting, we use Lora ($r=128,\alpha=64)$ for fine-tuning. The experiment is implemented in 8$\times$Nvidia A100 GPUs. The batch size of our training is 256, and the learning rate is $1.0\times 10^{-5}$, and the epoch number is 5.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: See weaknesses.
soundness: 3
strengths: - The paper is clearly motivated, the problem of hallucinations in LLM reasoning has not been explored as much.
- The paper presents the problem, the methods and the baselines clearly.
- The results on the three datasets, MATH, blocksworld and boardgameQA are comprehensive and provide sufficient evidence of the utility of the method.
- The authors also present sufficient ablations of their method, with varying versions of the curriculum used.
summary: The paper addresses the problem of hallucinations in reasoning for large language models. Specifically, in methods that use expert iteration for improving this reasoning in LLMs. The authors propose adding a Refusal option to the EI pipeline and rewarding the refusal when the problem has a certain level of difficulty (length of reasoning is used as a proxy for difficulty). Based on the reward that a response gets, the data for the next iteration of EI is selected accordingly. To balance refusal and improvements in reasoning, the authors propose using a curriculum to balance the two objectives. Based on measures of accuracy and refusal rate, the authors show the effectiveness of their method compared to baselines and ablations.
weaknesses: - Missing citations for central EI works: STaR (https://arxiv.org/abs/2203.14465), Training Chain-of-Thought via Latent-Variable Inference (https://arxiv.org/abs/2403.04642)
- What other measures of difficulty are possible (very recent paper that the authors can choose to ignore since it came out after the paper deadline: https://arxiv.org/abs/2410.04707)? Can a linear probe decode difficulty? A discussion is needed.
- Similarly, are there other mechanisms of knowing when to say I dont know possible? Is a linear probe enough for refusal? Like in Language models mostly know what they know (https://arxiv.org/abs/2207.05221).
- Where could the current framework of length based difficulty fail? When sampling multiple times, what is the variation in the number of steps needed to reach the answer for a problem?
- Are the number of iterations for Auto-CEI and EI etc matched? More generally, how does the amount of training data / compute required vary across the method and baselines? I think some of the implementation details could be moved to main in the revision. This would help readability.
- How could this method be extended beyond expert iteration, to more online methods of learning like PPO or REINFORCE?

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Why the process of update R is called curriculum learning? IMO, I think it is more like a search process and curriculum learning is about first learning elementary stuffs and then complex stuffs.
soundness: 3
strengths: 1. Using expert iteration is a good idea.
2. The authors consider the trade off between avoiding hallucinations and avoiding laziness, which is important.
3. This paper aims to avoiding reasoning hallucinations and teaching the model to say IDK to reasoning tasks which beyond its ability.
summary: This paper proposes an automatic curriculum expert iteration (AUTO-CEI) to enhance LLM reasoning and align responses to the model's capabilities. Enable the model to answer within its limits and decline when tasks exceed them.
Expert iteration can explore the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness.
AUTO-CEI uses the length of reasoning steps to measure difficulty and designs an automatic curriculum for expert iteration that rewards correct reasoning.
AUTO-CEI can automatically estimates the boundary of LLMs' reasoning capacity to achieve a reasonable alignment to maximize capacity and control behaviors.
weaknesses: 1. Although considering the trade off between helpfulness and laziness, the paper control this trade off by hyper-parameters, instead of proposing some principles or methods to chose the optimal hyper-parameters.
2. The evaluation metrics in the paper are incomplete; for example, IDK only measures the proportion of times the model outputs "IDK" without assessing the accuracy of those IDK responses.
3. The experiments in the paper may involve unfair comparisons, as AUTO-CEI conducts multiple searches for EI, resulting in significantly more training steps and data compared to other methods.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The MATH dataset has manually labeled question difficulty; it would be helpful to include a figure illustrating the correlation between generated text length and question difficulty as a validation of this signal.

2. Is the method effective across datasets with different distributions?

3. There are many established metrics for measuring uncertainty with different confidence levels, such as AP used in R-tuning. The current paper only reports results for the binary case, i.e., answering or refusing to answer. I am curious about the performance of this method at different confidence levels.
soundness: 3
strengths: 1. The paper presents a reasonable motivation that previous approaches to addressing hallucination problems often suffer from overcorrection, leading to overly conservative responses from LLMs on many questions.

2. The paper introduces an innovative reward mechanism that dynamically balances the choice between responding or refusing to answer based on question difficulty. Subsequent experiments demonstrate that this method effectively handles the trade-off between answering and refusing. Figure 3 also illustrates that as the generated text length increases, more questions are declined.

3. The writing is clear, and the figures are well-designed, enhancing the overall readability of the paper.
summary: This paper enhances LLM calibration ability in reasoning tasks by mitigating hallucinations and reducing lazy behaviors. The method uses a curriculum of expert iteration, balancing answering the question and saying 'I don't know' by adapting the reward system according to the difficulty of reasoning tasks. It shows great balancing in assertiveness and conservativeness.
weaknesses: 1. The paper’s reward uses the length of the generated text to measure the level of the exploration. This limits the method’s generalizability and scalability, as it’s challenging to find a universal text processing metric for different types of text (e.g., code, tables). Furthermore, reward parameters such as c1 and c2​ also vary across datasets, which makes me concerned about the method’s effectiveness in situations where training and testing distributions are not similar. 

2. The experimental results show that even the best method in this paper achieves overall accuracy significantly lower than Vanilla EI. While I understand that incorporating refusal may lead to a drop in accuracy, achieving performance closer to Vanilla EI would be more compelling, as the current method shows that adding a refusal option will significantly reduce the accuracy.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: See weaknesses.

Additionaly,

(1) Have you tested the correlation between the number of reasoning steps and difficulty, as this is a key assumption of AUTO-CEI? If not, I suggest conducting a test experiment on the MATH dataset, considering it has manually labeled difficulty tags.

typos:
line 286 N -> K
soundness: 3
strengths: * This paper is well-written and presents clear ideas.

* The idea of aligning responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them is novel.

* The  motivation behind AUTO-CEI is straightforward and strong.
summary: This paper proposes Automatic Curriculum Expert Iteration (AUTOCEI) to enhance LLM reasoning and align responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them, so as to mitigate hallucination and laziness in reasoning tasks. Through experiments on  BoardgameQA, MATH and Blocksworld with Llama-3.1-8B-instruct, The authors demonstrate the effectiveness of AUTO-CEI, achieving superior alignment by effectively balancing assertiveness and conservativeness.
weaknesses: * AUTO-CEI introduces additianal training overheads. Considering that the process of AUTO-CEI includes EXPERT ITERATION (each iteration needs large amount of resampling), the additional training overhead can not be ignored. I suggest the authors align the sampling cost between different baselines.

* Limited validation across models. The effectiveness of AUTO-CEI is validated only on Llama-3.1-8B-instruct. Further exploration is needed to assess the generalizability to other models.

* The performance is significantly affected by the hyperparameter λ. Table 2 shows that the performance of AUTO-CEI fluctuates greatly under different λ, which can lead to a considerable amount of additional cost during actual usage.

Meta-review: The paper proposes Automatic Curriculum Expert Iteration (AUTO-CEI) to enhance LLM reasoning and align responses to the model's capabilities—assertively answering within its limits and declining when tasks exceed them, so as to mitigate hallucination and laziness in reasoning tasks. Specifically, the work adds a Refusal option (saying “I don’t know“) to the expert iteration pipeline and rewarding the refusal when the problem has a certain level of difficulty (length of reasoning is used as a proxy for difficulty). The authors proposed to use a curriculum to balance between refusal and answering the question. The paper presents a clear motivation (reducing hallucination, improving reasoning), and describes their methods clearly. The approach is sound and the empirical results on MATH, blocks-world and boardgameQA are comprehensive and provide sufficient evidence of the utility of the method.

Decision: Accept (Poster)