Automatic Curriculum Expert Iteration for Reliable LLM Reasoning
Authors: Zirui Zhao
Hanze Dong
Amrita Saha
Caiming Xiong
Doyen Sahoo
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Reasoning
Hallucinations
Laziness
Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Auto-CEI pushes and estimates the limits of LLM reasoning capacities and aligns LLM's assertive and conservative behaviours according to these limits for more reliable reasoning.

Abstract: Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges in LLM reasoning. Current efforts to reduce hallucinations primarily focus on factual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem-solving capabilities. To mitigate hallucination and laziness in reasoning tasks, we propose Automatic Curriculum Expert Iteration (Auto-CEI) to enhance LLM reasoning and align responses to the model’s capabilities--assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning its behaviour with these limits. We compare Auto-CEI with various SOTA baselines across logical reasoning, mathematics, and planning tasks, where Auto-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness.

Published as a conference paper at ICLR 2025
AUTOMATIC CURRICULUM EXPERT ITERATION FOR RELIABLE LLM REASONING
Zirui Zhao Hanze Dong2t
Amrita Saha²
Caiming Xiong?
Doyen Sahoo2
National University of Singapore
"Salesforce Al Research
ABSTRACT
Hallucinations (i.e., generating plausible but inaccurate content) and laziness (i.e. excessive refusals or defaulting to "I don't know") persist as major challenges LLM reasoning. Current efforts to reduce hallucinations primarily focus fac- tual errors in knowledge-grounded tasks, often neglecting hallucinations related to faulty reasoning. Meanwhile, some approaches render LLMs overly conservative, limiting their problem- capabilities. To mitigate hallucination and laziness in reasoning tasks, propose Automatic Curriculum Expert Iteration (AUTO- CEI) to enhance LLM reasoning and align responses to the model's capabilities- assertively answering within its limits and declining when tasks exceed them. In our method, Expert Iteration explores the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness; it also promotes appropriate "I don't know" responses after sufficient reasoning attempts. The curriculum automatically adjusts rewards, in- centivizing extended reasoning before acknowledging incapability, thereby push- ing the limits of LLM reasoning and aligning its behaviour with these limits. We compare AUTO-CEI with various SOTA baselines across logical reasoning, math- ematics, and planning tasks, where AUTO-CEI achieves superior alignment by effectively balancing assertiveness and conservativeness,
INTRODUCTION
Hallucination is long-standing issue in large language model (LLM) research, which refers to the phenomenon where L.L.-ggeerrted content appears plausible but is actually nonsensical or inaceu- rate, often misleading humans by seeming deceptively credible (Ji et al. 2023). It is more evident when solving complex reasoning problems beyond their capability, in which LLM tends to fake evi- dence or logic to answer the questions assertively. The reason for L.M''s hallucinations, overall, is misalignment of its real capability and its behaviours: LLM should not behave overly assertively or confidently when facing unfamiliar (Kang et al., (024a) and difficult problems (Dziri et al.) (2024). Instead, a preferred behaviour of LLMs to acknowledge their limitations by responding with "I don't know." However, we must also prevent the LLM from defaulting to degenerate responses like "I don't 't know," especially when the question is within its capability. A reliable LLM should strike a balance between maximizing performance and avoiding hallucination.
Various methods have been proposed to mitigate LLM hallucinations. However, most studies have focused factual hallucinations, i.e., fabricating non-existent evidence, while often neglecting reasoning hallucinations (Creswell et al.) (022). Reasoning tasks require deriving conclusions from evidence using rules, often involving multiple steps. Reasoning hallucination refers to the phe- nomenon where LLMS apply invalid rules or misinterpret the conclusion, leading to an incorrect final result even without factual hallucination in the evidence. Dziri al. (2024) found that LLMs exhibit a probability of error at each reasoning step. As these errors compound, the likelihood of incorrect final answers increases exponentially as the reasoning chain grows longer. In addition, most techniques mitigate factual hallucinations using external knowledge or conduct post-training to do "patching". These techniques can reasonably mitigate factual errors, but hallucinations caused
"Work was done when Zirui Zhao interned at Salesforce Al Research, Singapore.
1 'Corresoonencc to Hanze Dong waw.eredddo@@sllsoeorrcooomomm
'The code is available at httptttttggggiiti//
es forceA Resea rch CAl

---

Published as a conference paper at ICLR 2025
Probleemn
Easy
Diffcult Problems
Curriculum
Curriculum 2
Curriculum N
@ Starting State
Goal Answer
Assert ve Responses
>> IDK Responses
IK Compensation Reward
IDK Pensity
Rewarr Soore
Demonstration of the key idea of AUTO-CEI, AUTO-CEI CEI automatically designs sequence of cur- ricula and uses Expert Iteration to train LLM in each curriculum. Expert Iteration explores the reasoning tra- jectories (i.c., Chain of thought) and learns to get incorrect paths back on track to reduce compounding errors. Using the reward (denoted by blue and red colour) produced by curriculum, Expert Iteration also samples appropriate "I don't know" responses (IDK responses) after sufficient reasoning attempts (in the blue region). To mitigate laziness, it avoids the short IDK responses (in the red region). The curriculum automatically adjusts rewards for the reasoning length, incentivizing extended reasoning before acknowledging incapability, thereby pushing the limits of LLM reasoning and aligning assertive and conservative behaviours with these limits.
Figure I:
by faulty reasoning are far more challenging due to the inevitable compounding error of the trans- former. As such, most of these techniques cannot improve LLM's inherent reasoning capability, and simply using patching techniques alone can even make LLMs behave lazily.
Can LLM learn short reasoning or self-correction reduce compounding errors and reasoning hal- lucinations? Kang et al. (2024b) showed that learning the optimal shortest solution for difficult problems (e.g., NP-complete problems) has very high sample complexity. Thus, directly learning short solutions for all complex problems is unrealistic. (Havrllla et al. (2024) showed that reinforce- ment learning (RL) can help improve LLM's performance various reasoning tasks. RL methods, such as Expert Iteration (Anthony et al.) 2017) and PPO (Schulman et al., 2017), explore various LLM reasoning trajectories (i.e., Chain of Thought (Wei al. 2022)) ) and use reward function to guide the incorrect trajectories back on track, therefore improving the robustness and reducing the effect of compounding errors. However, compounding errors are inherently impossible to climinate. Even though RL reduces step-wise errors, the probability of error still increases exponentially with the length of reasoning. As a result, it is necessary for LLMs to acknowledge their limitations, particularly when faced with difficult problems beyond their capability. This raises key question:
How can we assess the limits of LLMs' reasoning abilities and adjust their re- sponse behaviour appropriately to align with these limits?
To address this, we propose Automatic Curriculum Expert Iteration (AUTO-CEI), which simultane- ously enhances LLM reasoning and aligns its behaviour to ensure precise answers while acknowl- edging its limitations. AUTO-CES -EI assumes that the number of reasoning steps required to reach correct answer provides a reasonable estimate of both the problem difficulty and the limits of LLM reasoning. This assumption is grounded in computational theory, each reasoning problem has its underlying computational complexity, and each reasoning step corresponds to an elementary com- puting operation. Learning the reasoning steps (precondition/effict of reasoning rules) has similarly low sample complexity (Kang et al. 20224b. Despite the existence of concise optimal solutions, difficult problems (e.g., NP-complete problems) require extended reasoning find those solutions. This assumption helps align LLM's behaviours: easier problems require fewer reasoning steps and are less prone compounding errors, justifying greater assertiveness from LLMs. Complex prob- lems needing more steps and suffering more compounding errors require more conservativeness.
The key idea of AUTO-CEI is shown in Figure T using the length of reasoning steps to measure difficulty. AUTO-CEI designs an automatic curriculum for expert iteration that rewards correct rea- soning, compensates for don't know" acknow ledgemenns after a sufficient number of attempts, and penalizes both overly conservative and assertive wrong responses. The expert iteration adjusts the LLM's responses by resampling based on these rewards. The curriculum automatically updates the compensation reward to encourage time. It gradually adjusts the reward function to optimise an objective function, balancing the overall precision and the proportion of saying "I don't know" control hallucination and avoid laziness.
reasoning attempts before saying "I don't 't know" over
2

---

Published as a conference paper at ICLR 2025
As such, it gradually pushes the limits to maximise the potential of LLM reasoning and aligns its behaviours with these limits. AUTO-CEI is effective across various reasoning tasks such as logical, mathematical, and planning problems, balancing reliability and conservativeness.
Contributions. In summary, AUTO-CEI automatically estimates the boundary of its reasoning ca- pacity, thus achieving reasonable alignment maximise capacity and control behaviours. It learns to reliably solve the problems within its boundary as much as possible; it also knows to acknowledge "I don't 't know" when the problem is beyond its limit. We carried out comprehensive experiments in various reasoning benchmarks, and AUTO-CEI significantly outperforms the concurrent baseline methods, boosting precision by 10-24% while maintaining a relatively low refusal rate of 18-36 18-36% across diverse reasoning tasks in planning, logical and math reasoning.
2 RELATED WORK
Hallucinations and laziness of LLM. It widely acknowledged that LLMs often produce halluci- nated responses, including fabricated facts and misleading logic (Radhakrishnan et al. 2023 Ji et al. 2023). Most works focus reducing hallucinations by ensuring the factual accuracy of generated content, often using retrieval-based methods that enhance LLMs with external knowledge (Shuster ct al. 2021). This methodology has been effective in improving the reliability of LLMs do- mains that require high factual accuracy, such as knowledge-based question answering (Lewis et al.) 2020 Particularly, Setrieval-Augmentt
Generation (RAG) reduces hallucinations by fetching rel- evant documents from external sources during inference, integrating factual information to enhance accuracy and relevance (Izacard al. 2023). These methods ensure that the L.MSs' responses align more closely with verified data (Komeili et al.) 2022) Gururangan et al., (021). Moreover, sev- cral studies have explored hybrid approaches that combine retrieval with fact-checking modules or domain-specific fine-tuning to improve factual accuracy (Lee et al. 2022). Retrieval-based strategies show strong potential in mitigating hallucinations, outperforming purely generative models that rely only knowledge from training (Borgeaud et al. [022]] Petroni et al. 2020). However, hallucina- tions are not limited to factual inaccuracies; they can also extend to faulty or incomplete reasoning, significant concern for multi-step reasoning- often exhibit what can be described as "laziness," which refers to the tendency of the model to reject or avoid generating correct but complex answers in favour of simpler, superficial, or incomplete re- sponses (Bubeck et al. 2023: Bang et al., 2023). This phenomenon has been noted in tasks requiring step-by-step logical reasoning, where LLMS tend to skip intermediate steps or provide too general answers, rather than addressing the nuanced complexity of the problem at hand (Rae et al.) 2021).
tasks (Creswell et al. (2022). Moreover, LLMs
Reinforcement learning for LLM reasoning. Reinforcement learning has been widely used in the alignment stage of LLM training to enhance their capabilities. The research in this area can be broadly divided into two aspects: (1) RL algorithms: (2) reward construction. Conventional ad- vanced deep RI. algorithms, such as Proximal Policy Optimization (PPO) (Schulman et al.) (017), have been extensively applied in the training process of modern LLMs, helping to improve both format and reasoning abilities (Achiam et al. 2023 Team et al. 2023 Dubey et al.) 2024). How- ever, due to the introduction of the value network, the computational overhead of these algorithms is extremely high, often making them unaffordable for academic settings where resources are lim- ited. More recently, expert iteration (also known as rejection sampling or reward-ranked fine-tuning)
(Dong et al. 22023] (023//Gulee Gulcehre al. 2023 Zelikman et al. 2022] Hoffman al. 2024), preference- based RL |Rafailov et al. 2024; Xiong et al. 2024a), and REINFORCEE-yp RL (Williams) 1992] Shao et al. 2024, Li et al. [2023] Ahmadian al. 2024 have emerged as more efficient alter- natives for the alignment stage, especially when operating within a fixed budget. These methods have demonstrated significant improvements over supervised fine-tuning in reasoning tasks (Aksi-
tov et al., (2033, Gou et al. [2023 Havvilla et al.] 2024; Shao et al. 2024 Dong et al. 2024 Trung ct al. 2024, Xiong et al.| 2024b). Additionally, Lightman et al. (2023) introduced process super- vision to refine the reward signal for each reasoning step. Compared to traditional outcome-based
supervision, process supervision has been shown to yield substantial gains (Lightman et al., 2023] Wang et al.) 2024a| Shao et al. 2024). Despite the impressive progress in this area, most existing works focus on maximizing the upper bound of reasoning ability. In contrast, our algorithm aims to develop a more balanced model incorporating self- awrrenesssssss into the reasoning procedure.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
3.1 FORMULATION: REASONING PROBLEMS
We address reasoning problems in general, such as logical reasoning, mathematical reasoning, and planning, Reasoning involves deriving unknown facts from provided evidence and reasoning rules. In logical reasoning, evidence is represented as clauses with assigned boolean values, and applying reasoning rules determines the boolean values of unknown clauses. In arithmetic reasoning, evi- dence consists of clauses linked to scalar values, and reasoning requires both logical deduction and arithmetic operations. In planning problems, the goal is identify a sequence of actions that move an initial state to a target state by applying actions based on their preconditions and effects.
Unlike methods that convert problems into Boolean satisfiability (SAT) and solve them with SAT solvers, the LLM reasoning approach, such as Chain of Thought (CoT) (Wei et al. 2022), follows a sequential recision-making strategy. To capture reasoning problems, we rely on planning concepts that consist of four main elements: state space S. action space A, transition function T, and goal function G. The state space S includes all possible configurations, such as the boolean or scalar values assigned to the given variables or clauses, as well as unknown clauses. Actions, defined in the action space A, represent rules or equations that can be applied to these states. The action has its preconditions and effects. A state should satisfy the corresponding precondition to execute an action, and then the effeets will be applied to the state. The precondition and effect are encoded in the transition function. When action A is executed in state S, the transition function ('ss, a) de- termines the resulting state s', e.g., new values would be assigned to (unknown) clauses or variables. The goal function G checks whether the current state matches the desired outcome.
3.2 BACKGROUNDS
Expert Iteration. Expert iteration (EI) (Anthony et al. 20177) is an iterative process in which experts are built upon base policy, the base policy is refined by learning from the expert, and this cycle repeats with newly derived base policies. EI is effective in improving the quality of generated re- sponses in LLM literature (Dong et al. 2023 Gulcehre et al.) 2003). [Havrllla et al. (2024) discussed the usage of EI in LLM reasoning problems. They used I.L. sampling generate responses, and preferred responses were selected by ground-truth or reward model to build the expert. They then used LLM imitate the expert via SFT. El explores the neighbouring area of LLM policy in the token space, which helps LLM derive back to the correct reasoning trajectories if there are errors or find suboptimal trajectories. As a result, in their experiment, EI improved the LLM reasoning over time, which performed similarly or even better than the PPO (Schulman et al. 2017).
Curriculum Reinforcement Learning. Curriculum Reinforcement Learning Narveeaar et al., 2020) focuses on sequencing tasks or data samples into a structured curriculum to solve complex problems that are difficult to learn from scratch. curriculum can considered an ordering over experience samples at the most basic level. The underlying assumption is that learning from elemen- tary samples or tasks can be transferred and help learn more complex tasks. In this paper, we design the curriculum by reshaping the reward to gradually guide the LLM in conducting more reasoning attempts to solve the problem before acknowledging its limits. Its attempts could be used to discover suboptimal trajectories that mitigate its compounding errors in easy problems and also transferred to increase the chances of solving more complex problems.
4 AUTOMATIC CURRICULUM EXPERT ITERATION
AUTO-CEI aims to estimate and push the limits of LLM reasoning concerning the number of rea- soning steps, maximising its capacity while aligning its behaviours accordingly. We assume that, for a broad class of reasoning problems, the number of reasoning steps (in the chain of thoughts) required to reach the correct answer estimates the problem's 's difficulty and the limits LLM reason- ing. We justify the assumption by computational theory. The reasoning problem has its underlying computational complexity, where each of the reasoning steps can be treated as an elementary com- puting operation. Learning to apply the preconditions and effects of those elementary reasoning actions/rules has low sample complexity (Kang et al.) (2024b). Although problems may have concise optimal solutions, given the inherent computational complexity (e.g., NP-complete), finding that
4

---

Published as a conference paper at ICLR 2025
solution still needs long reasoning attempts. AUTO-CEL estimates the reasoning limits in terms of the number of reasoning steps until the LLM can respond reliably, solving a maximum number of problems. But beyond this point, further continuing the same assertive behaviour of the LLM policy leads to a significant increase in mistakes, whereas being overly conservative within this range limits reasoning capacities due to the high refusal rate. Thus, it estimates the L.M''s capability's boundary when fine-tuned for multi-step reasoning tasks.
AUTO-CEI uses the average number of the reasoning steps of the initial (after supervised finetuning, SFT) policy to determine the curriculum, i.e., the reward function. It starts from optimising an initialised policy that reasonably conservative with a certain distribution to say, "I don't know." then runs into the Expert Iteration process, in which the LLM policy will sample many reasoning trajectories and receive various rewards. We build up the expert by resampling the trajectories according to the reward and removing the assertively wrong responses. After convergence, AUTO- CEI updates the reward function according to the performance, in a way to encourage the LLM to produce more reasoning steps before saying "I don't know." The curriculum eventually tries to optimise an objective function, which balances assertiveness and conservativeness and stops the curriculum if the objective function reaches the optimal value.
4.1 INITIALISATION
We adopt R-Tuning (Zhang al. 2023). one of our baseline methods, as the initialization strategy. R-Tuning produces good starting point with a reasonable proportion of refusal behaviours. It also makes LLM lazier and has a lot of room for optimisation. We first use SFT to train a language model given the train- ing dataset D consisting of n question-answa pairs. Each answer uses the chain of thought. After SFT, we will let the language model answer the questions in the same dataset D using random sampling. As we use random sampling, there will be a certain number of correct answers, and the others will be wrong. Thus, we split the questions and the new answers generated by LLM into two datasets, D1 and D2, where Dj collects correct answers and D2 consists of wrong answers. For D2. in each of the wrong answers, we add an ex- pression at the end to acknowledge the limitations and abstain from answering the questions, such as "Sorry, I am not sure if answer the question correctly. There might be mistakes in my answer." Thus, we have collected a new Refusal-Aware dataset D2 for our finetuning.
Algorithm 1 INITIALISATION T)
1: Jaf <- SFT(T, D)
2: D' + (2'(''1)) 1/)'// 1y mzn x' E D}
3: Add IDK to all wrong y' in D' to get Dinit
4: To f SFT((T, Dma))
5: return To
After collecting D1 and D2, we concatenate both datasets to form Dinit and use SFT to fine-tune the LLM again. We also examined the SFT result ensure it had enough sample points for assertive and refusal answers. We set a threshold of 25% for the distribution of refusal behaviours in the validation set, as we need to have enough variety of acknowledge "I don't 't know" for AUTO-CEL If the model has a very low distribution of acknowledge "I don't 't know", we then repeat the previous process of collecting Dj and D. again, where the refusal answers will also be collected in D2 and use the new dataset for a new SFT training. We observed that, empirically, the distribution of refusal answers would increase over the steps. Thus, after the initialisation, we will have good initial policy with enough base knowledge to answer questions and distribution of acknowledging "I don't 't know."
4.2 EXPERT ITERATION
M(x, y)
Given the initial policy, we run the Expert Iteration to improve the policy over time.
0.5
We build the expert using the reward function and resampling. The reward function R(x, 3) takes the question x and an answer y as input and outputs scalar. R(x, y) is designed to give large pos- itive reward for correct answers, a small compensatory reward for refusal answers after long reasoning trajectories, small negative penalty for refusal answers after very short reasoning trajectories, and large negative penalty for assertive wrong answers. The re-
0
0.5
len(v)
C1
Figure 2: The shape of R(æ, v) for y is a refusal answer.
5

---

Published as a conference paper at ICLR 2025
ward is designed as follows:
1
for correctly answers
R(x,y)=
1- 1-exp( - (len (g) m))
for y is refusal answer; for y wrongly answers x
1texp( -02 (Jen (x) -v))
In this function, ei and 02 are two hyperparameters determined by the distribution of answers from the initial LLM. If y is a refusal answer and the number of steps for the reasoning trajectory len(y) longer than C1. then it will receive small positive compensation reward; otherwise, turns into small negative penalty. The shape of this function is shown in Figure 2 C1 is initialised by the mean value of reasoning steps produced by the initial LLM policy in the validation set. 02 is computed by 1-exp(-ca-20
solving T+exp(--ee 22) = 0.9, meaning that if the number of reasoning steps has reached ci +2 (ar is the standard deviation of the reasoning steps by initial policy), then the reward should be higher than 0.9. It means the answer is longer than roughly 97% of the reasoning trajectories the validation set, assuming the distribution of reasoning steps is normal distribution.
The reward function helps us build the expert, enabling us to improve LLM reasoning perfor- mance over time. We resample the responses according to the rewards to build the expert. We first use LLM to generate K samples {y { for each of the questions in the training dataset
Algorithm 2 EXPERTITER(D, Teis R. D TT)
1:
while Tei doesn't t converge in Dval do for x E D do
2:
3:
Y, {y: yJ [<)(
4:
R₂ f- [ri = R(x, 3/ E Y2)
D. Then, we use the reward function to get reward value of each sample ri; .......,'.. Over that reward, we employ the temperature- scaled softmax function construct new dis- tribution: p' ((()) = exp(r:/T)/ E3 exp(r,/T)
5:
Resamples Y2 according to Rg to form
new responses set Y
6:
end for
7: Duew + {((r,)))))))) 11 Y v D}
8: Tei SFT(TT,T Duew)
and subsequently use y' (3:) to conduct resam- pling and generate N new samples ( т is the temperature of resampling, the same as
9: end while
10: return x²'
the overall accuracy of the initial SFT model finetuned by D. If the SFT model has high overall accuracy, we will encourage LLM to explore more randomly; otherwise, the responses with higher rewards will be sampled more densely. r will he capped in a range [0.4,0.7 to avoid extreme cases. Thus, the distribution is reshaped to encourage more instances that produce correct answers and produce more reasoning steps before saying "I don't know." Since resampling also kills instances that waste some of the exploration data, concatenate the { k and {u/) together to form the new training dataset. Lastly, we removed the answers that had -1 rewards.
4.3 CURRICULUM UPDATE
After EXPERTITER converges, the curriculum up- dates the reward function to optimise (i.e. max- imise) an objective function:
Algorithm AUTO-CEI(D, म, R, J, Dval)
Wei +- INITIALISATION(D,
2: while Optimal f is not found do
(1 A)P + AA1 PIDK);
(2)
3: Update ej in reward function R & Eq I
4: Tei E-EXPERTTTERDD, Tein R, Dvals m)
where Pree denotes the precision, i.e., the correct- ness of answers that are not refusal answers, and PIDK denotes the proportion of refusal answers. This function ff tries to find good balance between avoiding both hallucination (low Pree and low PIDK) and laziness (high Prie but high PIDK) when finetuning the L.M. À E [0, 1] is hyperparameter control the tradeoff between hallucination and laziness; a higher value of À will lead to more assertive behaviour (i.e. lower IDK rate) while smaller A would make the LLM policy behave more conservatively. We empirically show that setting À is fairly intuitive; in most task scenarios, setting À to a reasonably small value (*s 0.2) achieves the desired effect for the LLM policy. Over the EI iterations, the curriculum will update the value of ei in Equation I i.e., try to push C1 to encourage more exploration before saying "I don't know". Thus, with the reward function,
5: Measure f by Tei and Dval
6: end while
7: return Tee
repeat the Expert Iterations and make it converge again.
Assuming that the function of f does not have local optimal point wrt C1 the curriculum uses local search (hill climbing) (Russell & Norvig 2016) to search e and conduct Expert Iterations until it

---

Published as a conference paper at ICLR 2025
finds the highest f. The hill-climbing algorithm searches its neighbouring value of = d via step size d, and if f at the new step is higher, then it updates e₁ to the new value. It will stop the search its neighbouring ej has smaller value than its current f. The domain of is [u 1u-2a, a 2cr where denotes the average length of the reasoning steps produced by the initial L.M policy. is defined by min 4o /10), where denotes the standard deviation of the reasoning length produced by the initial LLM policy. We also do not want to make the step too large, thus capping it by a threshold value of 0.5, an empirically selected value.
Overall, the curriculum update, together with initialisation and Expert Iteration of AUTO-CEI, are in the pseudo code listed in Algorithm
5 EXPERIMENT
5.1 EXPERIMENTAL SETUP
Benchmarks. To demonstrate the effectiveness of AUTO-CEI in reasoning tasks, we se- lect Boardgame (Kazemi et al,, 2024). MATH (Hendrycks et al.] (02T), and Blocksworld VValmekaam et al. 2023) as benchmarks. spanning from logical and mathematical reasoning to planning. They have various domains and complexities. We briefly introduce the benchmarks and report our detailed experimental settings in the Appendix
BoardgameQA is one of the latest benchmarks for logical reasoning. The data is synthesised from formal logical reasoning rules and clauses. The problem provides self-contained con- text, pieces of evidence, reasoning rules, and questions. Given the evidence and reasoning rules, LLM must decide if the queried clause is true, false, or unknown. Moreover, BoardgameQA has cases where some evidence might contradict others. To deal with this case, BoardgameQA also provides preferences among the reasoning rules, meaning that the conclusions drawn from the preferred rules have higher priority. In addition, since BoardgameQA does not provide CoT data for unknown questions, we use GPT-4 to generate CoT reasoning trajectories for unknown prob- lems for the training dataset. Thus, the training data is consistent. The prompt for data generation is in the Appendix [F]]]
MATH is a challenging mathematical reasoning benchmark, including advanced-level algebra and geometry. In each of the problems, the necessary evidence is provided in the context, and LLM is required to learn correct mathematical theorems and rules to connect the pieces of evi- dence in the given problem and draw conclusions. It typically requires long reasoning steps in which each mathematical theorem and rule is correctly applied to get the correct answers.
Blocksworld is a symbolic planning problem. It defines a domain with a few blocks in differ- ent colours. The problem requires the LLM to rearrange the blocks step-by-step and make the configuration of the blocks satisfy some preferred constraints. Concretely, it requires the LLM to output a sequence of pick and place behaviours, which transform the current configuration of blocks to the goal configuration. Blocksworld has been proved to be NP-hard in finding optimal (i.e., shortest) planning trajectories (Gupta & Nau 1992). while suboptimal policies might be easier to learn.
Baselines. To validate the effectiveness of AUTO-CEI, we conduct extensive experiments compar- ing against multiple baselines.
SFT. We perform supervised fine-tuning on the original training dataset D and use the overall accuracy reference metric.
Vanilla EI. As noted by Havrilla et al. (2024), Expert Iteration achieves strong results in enhanc- ing reasoning capabilities. We apply Expert Iteration improve LLM reasoning, evaluating its performance by retaining only correct and assertive answers during the iteration process.
SFT + R-Tuning (Zhang et al. 2023). This is an SFT-hased post-training method for hallucina- tion mitigation. Please see our initialisation part in Section 4.1 for details.
EI R-Tunnnn. This baseline uses Vanilla Expert Iteration to boost the performance of LLM reasoning first and then uses R-Tuning for post-training.
RLKF (DPO). Reinforcement Learning from Knowledge Feedback (RLKF) (Xu et al. 2024 is RL-based post-training method for hallucination mitigation. It aligns the LLM's behaviours according to the consistency and correctness of LLM's sampled responses: teaches LLM to respond assertively if its responses are correct and consistent and acknowledges "I don't know" if
7

---

Published as a conference paper at ICLR 2025
its responses mainly wrong inconsistent. We implemented DPO version of this method. Please see the Appendix for details.
Metrics. We evaluate LLM's responses' overall accuracy, precision, and refusal rate (refusal rate). The overall accuracy measures the overall performance of LLM in a specific reasoning task. A higher accuracy means the LLM can solve more problems. We compute the accuracy by PAce Number of correct responses The precision measures the accuracy of LLM when it is willing to answer Number of all responses
the questions assertively. It reflects the overall reliability of LLM reasoning. We compute the
Number of responses precision by Pere Num of all responses Num of IDK In addition, the hallucination (error) rate refers to The refusal rate measures how conservative the LLM policy is. A good LLM policy should have reasonable precision and refusal rate balance, i.c., high precision and a reasonable refusal rate according to the difficulty of the task.
Number of reo-uel but incurrect resporses
Num all responses Num of IDK
computed by PIDK-
Num of refusal responses
Num of all responses
Implementation details. We use Llama-3.1 1--.....----instruee (Dubey et al. 2024) as the backbone model and use Lora (r 128. 64) to fine-tune. The experiments are conducted in server with 8x Nvidia A10 (40GB) GPUs. We use DeepSpeed Stage 2 to conduct the training. See Section G in Appendix for more details about the AUTO-CEI setup.
5.2
RESULTS
The main result is reported in Table In this result, AUTO-CEI produces high precision (Pre) and keeps reasonably lower refusal rate (IDK) across all tasks. It also has the highest objective function
value f defined by Equation 2 (A = 0.2). We summarize our findings in the following paragraphs.
Table I: Main results. Ace: accuracy (%); Pre: precision (%); IDK: refusal rate (%); f: objective function.
BoardgameQA
MATH
Blocksworld
Method
Acc
Pre
IDK
Acc
Pre
IDK
f
Acc
Pre
IDK
f
SFT
60.23
60.23
38.65
38.65
51.99
51.99
Vanila EI
67.77
67.77
42.04
42.04
71.04
71.04
SFT + R-Tuning
55.23
80.36
31.27
0.780
19.49
60.67
67.88
0.550
38.54
90.69
57.51
0.811
El + R-Tuning
57.76
80.77
28.49
0.789
29.57
55.80
47.02
0.552
67.92
93.95
27.71
0.896
RLKF (DPO)
49.32
54.17
8.95
0.615
40.02
42.19
5.14
0.527
35.24
38.08
8,21
0.486
AUTO-CEI (Ours)
59.70
84.52
29.37
0.817
35.56
55.63
36.08
0.575
74.78
91.53
18.30
0.896
El improve LLM reasoning. In our experiments, the Vanila El produced higher overall accuracy than SFT in all tasks, which indicates the overall capacity of LLM in reasoning is improved. This improvement because EI helps LLM to sample various trajectories and collect those who draw a correct solution and learn those solutions; for the next iteration of EI, it samples more trajectories near those correct trajectories and keeps collecting and learning. Over time, LLM learns to start from various trajectories and still get the correct solution, even though some might be suboptimal. Thus it becomes more robust to the randomness in token sampling when generating responses and the resulting compounding errors.
R- R-Tuning makes LLM over-conservative. We found that SFT + R-Tuning and EI + R-Tuning all show the trend of over-conservativennss (i.e., laziness). The results of both baselines have relatively high refusal rate (IDK) and lower overall accuracy. The low overall accuracy indicates that LLM's capacity has been limited, and LLM produces refusal responses many problems that lie within the LLM'' capability.
RLKF degrades reasoning performance. Reported by Xu et al. (2024), RLKF performs well for short, knowledge-grow -oouudde arithmetic problems. However, for reasoning problems, the responses generally become much longer for more complex questions. This causes a few additional issues for RLKF techniques. First, the LLM finds it hard to distinguish the correct chosen/rejected responses given the long response length the dataset is not very large; thus, the reward accuracy is relatively low (it only has 30% accuracy for Blocksworld and 70% for BoardgameQA and MATH). Second, the original strategy of reward shaping greedily maximizes the reward for correct answers and minimises incorrect or refusal responses. As such, the I.LM will be over-assertive. As it greedily imitates the
8

---

Published as a conference paper at ICLR 2025
optimal responses guided by the inaccurate reward, it will suffer more from the compounding error when facing the unseen context of the reasoning problems, thus degrading its performance.
AUTO-CEI improves the LLM reasoning. The result shows that the overall accuracy of AUTO- CEI is higher than SFT R-Tuning. Since our method initialises its optimisation using SFT R-Tuning, this result indicates that AUTO-CEI improves LLM's reasoning capacities. It is because AUTO-CEI gradually adjusts the curriculum to encourage LLM to solve more problems within its capacities and only accept refusal responses after sufficient reasoning attempts. It thus avoids the LLM's laziness in easy problems and improves the robustness of the LLM's problem-solvin when samples suboptimal trajectories. The improvement of precision and refusal rate within few expert iterations is shown in Figure 7 of the Appendix.
AUTO-CEI estimates LLM's limit in rea- soning and produces the best alignment between its behaviours and its limit. We compare the error (hallucination) rate and refusal rate (IDK) according to the response length. Figure 3 shows that the error rate of SFT and grows exponentially with the increase in response length. On the other hand, we can also see that AUTO-CEI has a relatively uniform low rate of hallucination for different response lengths, and its refusal rate (refusal rate) grows according to the er- ror rate of SFT/EI, as is ideally expected. This result indicates that AUTO-CEL deed able to estimate LLM's current capa- bility limit and further capable of reach- ing reasonable alignment between its as- sertive and conservative behaviours accord- ing its limit. It thus behaves assertively for problems within its limit (low refusal rate for short reasoning length) and conservatively for problems beyond its limit (high refusal rate for long reasoning length). Overall, produces reliable reasoning while simul- tancously maintaining maximum reasoning
1.-
0.8
Refusal Rate
Hallucination
6.4
0.2
2-6
6-11
10:16 15-21 21:25
26-
2-6
6.11
11-26 26-1 23-26 26.
Response Length
Response Length
(a). BoardgameQA
Hellacimetion (Eroot Rate
BrTanin
Mutin BLK (DPO)
in-
0
0.5
34
6-1
13:58
26-21
23-25
25-
0.
6-1 22-28 18-21 21-28
Response Length
Response Longth
(b). MATH
Hellucination Rale
SFT
0.8
Betusel Rate
MaTanig kTunnng
ALKS (DPO)
0.6-
0.6
AVCO-CES
ability.
6.-
5.3 ABLATION STUDY & DISCUSSION
6.
11:25
0.0
25-
Response Length
Response Length
We conduct an ablation study to verify the effectiveness of our solution design choices and further discuss our underlying assump- tions. As the effect of Expert Iterations has been demonstrated in the main result, this section, we mainly focus the effect of currieulums and the practical usage of hyper-parameter X.
(c). Blocksworld
Figure
The error refusal rate (hallucination/IDKK)
according to the response length on different datasets.
The result of the ablation study shown Table 2
Table Ablation study. Acc: accuracy (5) Pre: precision (%); IDK: refusal rate (%).
BoardgameQA
MATH
Blocksworld
Method
Acc
Pre
IDK
Acc
Pre
IDK
Acc
Pre
IDK
No Curriculum
56.10
85,56
34.43
27.14
52.06
47,86
68.43
93.42
26.75
X 0.5 59.57
81.12
26.56
37.62
49.90
24.61
79.31
84.25
5,86
0.2
59.70
84.52
29.37
35.56
55.63
36.08
74.78
91.53
18.30
Effect of A. The hyperparameter À decides the optimal objective function and when stop the optimisation. A higher À will try to minimise the refusal rate (refusal rate) and thus make the LLM more assertive. The lower À will try focus maximising precision and tends keep higher

---

Published as a conference paper at ICLR 2025
refusal rate, making the LLM policy more conservative. The À does not affect the overall training process in AUTO-CEI, and the decision of the hyperparameter is mainly determined by the user's demand. Empirically, À 0.2 has a reasonable balance between the precision and refusal rate. In practice, we suggest the user start by X = 0.2 and further adjust the parameter according to the demands. For example, lower À would be preferred in highly risky tasks, whereas higher À is better for training search heuristics that require more assertive behaviours, higher X is better.
Effect of curriculum. We aim to use the curriculum push the limit of LLM reasoning, thus achieving a good alignment of its behaviours and maximizing its reasoning ability simultaneously. We compare the ablation that does not gradually update the reward function defined in Equation [ The result in Table 2 (No Curriculum) shows that LLM converge to a suboptimal point where its overall accuracy is lower and its refusal rate higher. In addition, there might be a chance that LLM directly reaches the optimal f after one curriculum with some other choices of X. However, as discussed previously, the choice of determined by the user's demand, and the optimisation conducted by the curriculum should be able to find the optimal point specified by different A.
6 CONCLUSION
This paper focused on mitigating LLM's hallucinations and laziness in reasoning tasks. We pro- pose Automatic Curriculum Expert Iteration (AUTO-CEI) to push the limits of LLM's reasoning capacities and align its assertive and conservative behaviours according to these limits for reliable reasoning. We conduct experiments on various challenging reasoning benchmarks, including logical reasoning, mathematics, and planning. Our result suggests that AUTO-CEI maximizes its reasoning ability while achieving superior alignment between response behaviours and this ability. The fun- damental assumption under AUTO-CEI is that solving difficult reasoning problems requires more reasoning steps, thus making the number of reasoning steps good estimation measure of difficul- ties. Recent research works by OpenAl (224)) and Snell et al. (024)) suggest that longer reasoning length in the inference time scales up the LLM reasoning ability and enables it to achieve expert- level capabilities. These researches provide evidence for our assumption. Meanwhile, they also show the chance that our method can also be applied to improve the reliability of those language models with super-long reasoning lengths.
Limitations. This work focuses challenging reasoning problems, where complexity can be timated by the number of reasoning steps. For simpler problems with shorter reasoning paths or well-studied datasets (e.g., GSM8K), R-Tuning reliably performs well, leaving limited scope for further optimization. This is partly due to the reduced impact of compounding errors in these easier tasks. We also found that the model might decline answer even if its response is correct. Thus, further adjustment and fine-tuning should be applied to solve this issue, and our method should be able to be extended accordingly as well. In addition, our method might require a long time for opti- misation as it requires running Expert Iteration multiple times. Nonetheless, compared to the cost of pre-training, the overhead of the iterative post-training algorithm remains worthwhile for achieving better alignments. Our primary goal was to establish a proof-of-concept using straightforward, re- producible metrics common in curriculum learning. While we focused on the number of reasoning steps as the core metric, the framework can accommodate more sophisticated approaches. Future work could incorporate advanced metrics such as LLM -eee-d-ddetedd statistics (Wang et al.] 2024bb Damani et al.) 2024) for fine-gra fine-grained curriculum learning. The foundational nature of Auto-CHi makes it amenable to such extensions while maintaining its core benefits of automation and scala- bility.
Reproducibility Statement. In Section 4 we provide pseudocode for our algorithm and explain the details of each part. We provide sufficient details in Appendix Section f our implementation of AUTO-CEI and training settings. For our experiment benchmarks, we introduce our benchmark selection and metric computation in Section [.1 and provide the link to download the dataset or the source code from the published papers to generate the dataset, the format of the dataset, and the dataset size we choose in Appendix Section F We also provide the baseline methods' details, implementation and training in Appendix Section
Codes are available.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We thank all Program chairs, Area chairs, and Reviewers for helpful discussion.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.000 08774, 2023.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint urXiv:2202. 14740, 2024.
Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self- improvement for multi-step reasoning Ilm agent. arXiv preprint 2rXiv:2312.10000 2023.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. arXiv preprint arXiv: 2302.00033 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference machine learning, pp. 2206-2240. PMLR, 2022.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee. Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 2023.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferencee Exploiting large lan- guage models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712 2022.
Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas. Learning how hard to think: Input-adaptive allocation of Im computation. arXiv preprint arXiv:2410.04707, 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.077 07863, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Faith and fate: Limits of
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 17X/v 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308. 088998 2023.

---

Published as a conference paper at ICLR 2025
Naresh Gupta and Dana S Nau. On the complexity of blocks world planning. Artificial intelligence, 56(2-3):223-22 -224, 1992.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. arXiv preprint 108.::200.000000.6 2021.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yoi Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2403.0400 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
net/ www.moiiddeae.........
2mQsCe
Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-ther -f-thuugttt via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini. Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251): 1-43, 2023.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612 2024a.
Liwei Kang, Zirui Zhao, David Hsu, and Wee Sun Lee. On the empirical complexity of rea- soning and planning in LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2897- 2936, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. org/
doi: 10. 10018663/1/2222.indnnngg findings--mmlpp.644 URL /attps://aclanttiee
Findings-emn 164/
Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. Boardgameqa: A dataset for natural language reasoning with contradictory in- formation. Advances in Neural Information Processing Systems, 36, 2024.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 8460-8478, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. URL. httppp::/acclaatoooloo...o///2/0
98653//1/0022..ccHoongg.5.
579
Naycon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586-34555 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval aaggentele genera- tion for knowledge- -nneedggenniniveeeeeeeeeeeee nlp tasks. Advances in Neural Information P Processing Systems, 33: 9454, 2020.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quar Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
12

---

Published as a conference paper at ICLR 2025
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50, 2020.
OpenAl. Learning to reason with 1ls.. https: /ttpss:///peaatt cwm/ indx/ learning-ton toeag-------------e rewson-snnnnnoo wi ttritth-iie 11ms/ 2024. Accessed: 2024-09930.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowl- edge intensive language tasks. arXiv preprint wrXiv:2009.02252, 2020.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, at. Question decomposition improves the faithfulness of model-gencrated reasoning. arXiv preprint arXiv:2307. 1168, 2023.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann,, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Stuart Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1 1707.7674777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03333 0330000003,,,. 2024.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07556 2021.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2rXiv:2408.03314, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 7601-7614, 2024.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Informa- tion Processing Systems, 36:75993-76005. 2023.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 00669943,,00 2024a.
Xinglin Wang, Shaoxiong Feng, Yiwei Li. Peiwen Yuan, Yueqi Zhang, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. Make every penny count: Difficulty-adaptive self-consistency for cost- efficient reasoning. arXiv preprint arXiv:2408. 13457, 2024b.
13

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 55:24824-24837 2022.
Ronald J Williams. Simple statistical gradient-foloowigg algorithms for connectionist reinforcement
learning. Machine learning, 8:229-255 256, 1992.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. 2024a.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, et al. Building math agents with multi- turn iterative preference learning. arXiv preprint arXiv:2409.023330 2024b.
Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. Rejection improves reliability: Training LLMs to refuse unknown questions using RL from knowledge feedback. In First Conference Language Modeling, 2024. httthhttt:///peeeet
net. /et./foruu forum?? foimmid==ll.m oZBoR8 1=118M
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 15:15577-15488, 2022.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677, 2023.
A DIFFICULTY AND RESPONSE LENGTH
The underlying assumption of our paper that difficult problems require longer responses to get the answer. We use the MATH dataset, which has manually labelled difficulties, to verify the hypothesis. We measure the steps reasoning with respect to labelled difficulty, shown Fig 4 The result suggests that the reasoning length and manually labelled difficulty are correlated with statistical significance. The scale looks different from Fig 3 in the paper, as the reasoning length distribution long tail distribution. The distribution is visualised in 5
Human-labelled difficulty often diverges from computational measures, with LLM-perceived dif- ficulty better aligning with reasoning steps (Pearson's 's 0.349, < le-142). Despite the noise the MATH dataset's non-standardized steps, Auto-CEI performs robustly, effectively balancing assertiveness and conservativeness.
Auto-CEI
5.0
Mean Reasoning Length
700
4.5
600
500
400
3.5
300
3.0
200
100
Level
Level
Level
Level
Level 5
Difficulty Level
10
20
30
40
Figure 4: Correlation between reasoning length and labeled difficulty (MATH dataset). For each difficulty level, the mean reasoning length is plotted with emror bars representing the standard error of the estimator.
Response Length
Figure Histogram reasoning lengths generated by AUTO-CEI for the MATH dataset.
14

---

Published as a conference paper at ICLR 2025
Some difficult problems might have short optimal solutions. However, this doesn't mean the compu- tational complexity of that problem low. Take NP-complete problems as an example. Searching for the solution may require exponential time (in the worst case), but verifying the solution only takes polynomial time, as the optimal solution is much shorter than the search process. While the question of whether verification is inherently simpler than finding solution (NP ?= P) remains open, our hypothesis relies on human intuitive common sense: more difficult problems generally require higher computational complexity arrive correct solutions, which also the case for real-world problems and solutions.
B HYPERPARAMETHR TUNING
The mitigation of hallucination and laziness contradict each other. Optimizing precision makes the LLM respond "I don't 't know" most of the time; minimising the refusal rate makes the LLM try to make an assertive response all the time. Because of this, our objective is maximise accuracy but to achieve the sweet spot of balance between assertiveness and conservativeness.
The hyper-parameter does not affect the training process determines when to stop the Auto-CEI. Empirically, the user can start initial hyper-parametee After it terminates, if the user is unsatisfied with the result, they can further update the hyper-paree parameer and continue the training process. The previous models can be reused so that extra computation is involved. One example is shown in Fig
precision
lambda-0.2 lambda=0.4
lambda-0.6
ldk, rate
lambda=0.8
0.0
0.84
precision and idk. rate
0.82
0.80
ggiven A
0.70
0.2
Curriculum Index
Figure 6: The value of objective function f changes according to the curriculum and hyper-parameter The red dot denotes the highest value of f in the curve.
C REFUSAL ACCURACY
We evaluate if the language model makes refusal response even if its answer is correct. Thus, we define the refusal accuracy (IDK Acc) below.
Number of refusal response with correct answer
PIDKACe
(3)
Number of refusal response
The refusal accuracy measures the proportion of the correct answer in the refusal responses. A lower accuracy indicates better self-awareness in correctness, while higher correctness means the language model is over-conservative. The refusal accuracy, compared with the precision, is shown in 3
The result suggests that Auto-CEl's refusal responses have significantly lower accuracy. We also highlight that a model's true capacity inherently a hidden variable and can only be estimated indirectly, there is ground-truth label of which questions truly beyond LLM's capacity. 1 (refusal accuracy) a Monte-Carlo estimation of the model's ' awareness in providing an inaccurate answer.
15

---

Published as a conference paper at ICLR 2025
Table 3: Refusal accuracy Precision. Pre: precision (%) IDK Acc: refusal accuracy (%).
Boardgame QA
MATH
BlocksWort
Pre
IDK Acc
Pre
IDK Acc
Pre
IDK Acc
SFT + R-Tuning
80.36
19.62
60.67
14.27
90.69
32.44
EI R-Tuning
80.77
16.67
55.80
12.74
93.95
41.21
RLKF
54.17
21.48
42.19
13.49
38.08
24.47
Auto-CEI 84.52
12.25
55.63
9.89
91.53
27.02
D GENIRALIZABILITY CROSS LANGUAGE MODELS
We conduct additional experiments test the generalizability of method in other language models. We did experiments on Missral--7B-Instruetee on the BoardgameQA benchmark. The result shown in Table It provides further evidence of our method's generalisability.
Table 4: Additional Experiment -instal-7B-Instruutt33.. Ace: accuracy (%); Pre: precision (%); IDK:
refusal rate (%); IDK Ace: refusal accuracy (%).
Acc
Pre
IDK
IDK Acc
f(A = 0.2)
SFT EI SFT + R-TTuming
71 71.28
75.54
59.02
76.39
22.74
15.42
0.7656
EI R-Tuning
61.62
80.39
23.35
14.23
0.7964
Auto-CEI
75.01
93.47
19.75
11.50
0.9083
E IMPROVEMENT IN EXPERT ITERATION
beardgameQA Procision
MATH Precision
blocksworld Precision
0.93
Preetwion
Prestsion
0.850
0.62
0.825
0.60
0.800
0.91
Number
boardgameQA IDK Rate
MATH Rate
blockswortd IDK Rate
0.6
0.30
IDE
0.5
0.2
0
Iteration Number
Iterationn Number
Reration Number
Figure 7: Improvement within El. The precision and refusal rate converge after iterations.
F EXPERIMENTAL SETTINGS
F.I BOARDGAMEQA
For boardgameQA, we use the main dataset to do fine-tuning and evaluation We take the dataset under the folder SaarddaameQ--Mainneeettte
and BoardgameQA QA -aa-QQA--Miii Main-depthh. We show one example of the problem in Boardgame in List- For more details about the dataset itself, please refer to the paper by Kazemi et al. (2024).
ing I
In our experiment, we put the example e as the input prompt and verify the final answer is the same as the label. The proof provides the chain of thought inference data, which is used in our fine-tuning.
2The dataset is downloaded from Hoa Hoorrrgganeea///ccrrrgggggg...
https://storagge.gom compia.com/gressercc
16

---

Published as a conference paper at ICLR 2025
Listing 1: BoardgameQA Problem Example
"facts": "The learns the basics of resource management from the aardvark. The dog burns the warehcuse of the koala. The dog proceeds to the spot right after the leopard. The dog reduced ner work hours recently The hummingbird has a card that is red in color.",
"rules": "Rulel: For the halibut, if the beliet is that the dog shows her cards (a1l of them) to the hal ibut and the hummingbird rd knocks down the fortress of the halibut, then you can add that \"the halibut is not going to eat the food that belongs to the lioni" to your conclusions. Rule2: If the humningbird has card whose color appears in the flag of Japan, then the hummingoird knocks down the fortress that belongs to the halibut. Rule3: If you see that scmething proceeds to the spot right the
affer the leopard and burns the warehouse that is in possession koala, what can you cortainly conclude? You can conclude that it also shows her cards (all of them) to the halibut. Rulc4: If the cow learns elementtryy resource management from the aardvark, then the aardvark burns the warehouse of the elephant. Rule5: If at least one animal burns the wa warehouse of the elephant, then the halibut eats the food that belongs to the lion.",
"preferences"; "Ruleb is preferred over Ruiel.
"example": "A few players are playing boardgame. The current state of the game is as follows. The cew learns the basics resource management from the aardvark. The dog burns the warehouse of the koala. The dog proceeds to the spot right after the leopard. The dog reduced her work hours recently. The hummingbird has card that is red in color. And the rules the game are follows. Rulel: For the halibut, if the belief is that the dog shows her cards (all of them) to the halibut and the hummingbird knocks down the fertress of the halibut, then you can add that \"the halibut is not going to cat the food that belengs to the lion to your conclusions. Rulc2: If the hummingbird has card whose color appears in the flag of Japan, then the hummingbird knocks down the fortress that belongs to the halibut. Rule3: If you see that something proceeds to the spot right after the 1eopard and burns the warehouse that is in possession of the koala, what can you certainly conclude? You can the halibut. Rule4
concluce that it also shows her cards (all of them) If the cow learns elementary rescurce management from the gardvark, then the aardvark burns the warehcuse of the elephant. Rule5: If at least one animal burns the warehouse of the elephant, then the halibut eats the food that belongs the lien. Rule5 is preferred over Rulel. Based the on the game state and the rules and preferences, does the halibut
food of the lion?",
"proof": "We know the cow learns the basics of resource management from the aardvark, and according to Rule4 \"if the cow learns the basics of resource management from the aardvark, then the aardvark burns the warehouse the elephant so we can conclude \"the aardvark burns the warehouse of the elephant)". We knew the aardvark burns the warehouse of the elephant, and according to Rule5 \"if at least one animal burns the warehouse of the elephant, then the hal ibut eats the food of the lion'", and Rule5 has higher preference than the conflicting rules (Rulel), so we can conclude \"the halibut eats the food the lien\". So the statement \"the halibut eats the food of the lion)" is proved and the answer is \"eest".",
"goal"; ;; (hallbbt, eat, lien)",
"theory"; "Facts: Anit (cow, learn, aardvark) \n\t (deg, burn, koala) Anlt (dog, proceed, leopard) inlt (dog, reduced, her work hours recently) \n\t hummingbird, has, a card that is red in color) AnRules:\n\tRun show, halibut) (hunmingoird, kneck, halibut) => "hhalibut, eat, lion) An tRule2: (hummingbird, has, card whose color appears in the flag of Japan) => (hummingbird, knock, halibut) initRule3: (x, proceed, leopard) ( X, burn, koala) => (X, show, halibut) \n\tRule4: (cow, learn, aardvark) => (aaravark, burn, (n halibut, eat, lion) A \nPreferences:: \n\tRule5 Rulel", "label"; "proved"
(dog,
exists X (X, burn, elephant) =>
17

---

Published as a conference paper at ICLR 2025
BoardgameQA data argumentation For the problems whose label unknown, the proof only provides one simple statement: The provided information is not enough to prove or disprove the statement "[the query statement)" This make the responses inconsistent with other responses GPT-4 to generate the chain of thought data
whose label is proved or di disproved. Thus, for unknown cases. The prompt of our data generation is provided in the Listing
Listing 2: Prompt for data generation
prompt-"**
You are helping to generate the reasoning for a logical reasoning problem em
The problem has sufficient evidence to concl if the question is unknown. You will need to generate reiutation or exploration reasoning steps to show that there is no evicence to draw a conclusion. Make sure your answer is clear and concise. Always end your answer by saying So the statement "..." is not provable and the answer is "unknown".
proved disproved, thus is not provable,
Example:
Quesiton: few players are playing a boardgame. The current state of the game is as follows. The cheetah needs support from the swordfish. The crocodile winks at the moose. The dog burns the warehouse of the raven. The gecko needs support from the aardvark. The kudu learns the basics of resource management from the aardvark. The leopard cats the food of the penguin. The leopard holds the same number of points as the lion. The vi wiperfish winks at the penguin. The zander respects the doctorfish sh.. The hare does not attack the green fields whose owner is the humminghs rd. The rabbit does
knock down the fortress of the game are as follows. Rulel: Be careful when something eats the Food that belongs to the penguin and also needs the support of the lion because in this case it will surely sing victory song for the whale this may may not be problematic). Rule2: If the gecko needs the support of the aardvark and the kudu prepares armor for the aardvark, then the aardvark will not wink at the oscar. Rule3: If the dog burns the warehouse that is in possessicn of the raven, then the raven shows her cards (all of them) to the hare. Ruled: If you are positive that one of the animals does not knock down the fortress that belongs to the sun bear you can be certain that will wink at the oscar without doubt.
the sea bass. And the rules
Rule5: If the leopard sings a victory song for the whale, then the whale attackk the green fields of the tilapia. Rule4 is preferred over Rule2. Hased the game state and the rules and preferences, does the whale attack the green fields whose owner is the tilapia? This quesion is not provable. Please generate the reascning steps. Make sure your answer is clear and concise. Answer:
We know the whale's attack on the green fields of the tilapis depends whether the leopard sings victory song for the whale, as stated in Rule 5.
For the lcopard to sing the victery song, it must both cat the food of the penguin and need the support of the lion, per Rule 1.
However, the game state does not cenfirm whether the leopard needs the lion's support, leaving this condition unresolved.
As none of the other rules (Rules 2, 3, or 4) are relevant to the whale's attack or the Leopard's pehavier, and there is insufficient information to verity the key condition in Rule the conclusion remains unknown.
So the statement "the whale attack the green fields whose owner the tilapia" is not provable and the answer is "unknown".
Question: () This quesion is not prevable. Please generate the reasoning steps. Make sure your answer is clear and concise.
Answer:
.** n"", format (question)
When using GPT-4 to generate the data, we choose temperature=0.6 and top ttppptppp...... p=0.95. The model we selected is gpt.-4-turbo.
18

---

Published as a conference paper at ICLR 2025
F.2 MATH
3 automatically divide the dataset into training, validation, and test We used all the training data in our experiment and evaluated it using all validation and testing sets to check if the final answer was the same as the labelled data. For more details about the dataset itself, please refer the paper by Hendrycks et al. (2021). One example format of the dataset is shown in the Listing
We download the MATH dataset via Hugging Face
Listing 3: Example data point of MATH dataset
'level': 'Level 4',
'type': 'Counting Probability', 'prompt': "The reality game shew Survivor is played with 16 people divided into two tribes of 8. In the first episode, two people get homesick and quit. If every persen has an equal chance of being one of the two quitters, and the probability that
person quits is independent of the probability that any other person quits, what is the probability that both people who quit are from the same tribe?\n/hink step by step and use the format of *pe* AMfinal answer\\). An##f' to complete your answer.",
'answer'; (1)5
the 16 people to AnThere are $1\binom = 285 ways for them to both
'completion': 'There are WVibinen (161 1205 ways to pick
288 ways for them to both be from the other tribe, for tetal of $28 + +2 565 ways for then both to be from the same tribe. \nse the edds that both people who quit from the same tribe is $56/120 (7)(1)) \n###*
be from the first tribe, and (NNbinom(8)1)
An#### A\rrac
F.3 BLOCKSWORLLD
The Blocksworld dataset can be generated using the code in the GitHub repository by Valmeekam ct al. (2023 4 For our case, we generate domains from 4 blocks to 6 blocks, and randomly samples 500 data points for training. We randomly sample 500 data points for validation and testing sets whose optimal solution length (i.e., ground truth plan) is no longer than ten steps. We uniformly sample the tasks according the ground truth lengths to form the testing set (i.e., 100 two-step tasks, 100 four-step tasks, ... and 100 ten-step tasks).
The original format of the chain of thought for Blocksworld planning unnecessarily long, therefore making it difficult to learn a sufficient policy. We modified the format of the chain of thought make it more concise and informative. The elementary format is shown below: "(problem (prroblem.deccrrpttonn ). Since (precondition .ext). Thus, take action {action) Since {precondition text). text). Thus, take action {action]. Since the goal is GOAL, The goal conditions are satisfied. ### One example of the dataset format is shown in the Listing
Listing 4: Example of Blocksworld dataset
'instance_id" '3',
'query': '1 am playing with a set of blocks where I need to arrange the blocks into stacks. Here are the actions I can doln\nPick up a block), nUnstack a block from top of ancther block(nPut down block\nStack my
block top of another block\ninI have the following restrictions acctions: AnI can only pick up or unstack one block at time. AnI can only pick up or unstack a block if my hand is empty. AnI can only pick up a block if the block is on the table and the block is clear. A block is clear if the block has no other blocks on top of it and if the block is not picked up. AnI can only unstack block from top of another block if the block I am unstacking was really on top of the other block. AnI can
ht http://///gggggieeeccaaaaaa/aaaa http:///hhhhbccco////rra///////2
Link:
19

---

Published as a conference paper at ICLR 2025
only unstack block from top of another block if the block unstacking clear. AnOnce pick up or unstack block, am holding the block. Ani can only put down a block that I am holding. AnI can only stack a block on top of another block if I am holding the block being stacked AnI can enly stack a block en tep of another block if the block onto which am stacking the block is clear. \nOnce put down stack block my hand beccmes empty. AnOnce you stack block on top a second block the second block is no longer clear. \n/nThe plan correctness is defined in terms of states resulting from executing the actiens in the plan. An action is executable in state when all its preconditions hold in that state. The state resulting from the action execution consists of everything in the previous state with the addition and deletion of add and delete effects of the action. Plan correctness is defined as follows: if the first action in the plan is applicable in the initial state, i.e all present there; and the second action is
its precenditions applicable in the state resulting from applying the first action to the initiall state, this process continues until the state resulting from the application of the last action in the last but one state gives rise to satisfied. AnVI[STATMMEET \nAs have that, the red block is clear, the orange block is clear, the yellow block is clear, the hand is empty, the yellow block is on top of the blue block, the red block is on the table, the blue block is on the table and the orange block is on the table\nke goal is to have that the blue block is top of the orange block and the yellow
the final state where all the goals
iniiiaal conditions
blook on top of the red block. \aMy plan is as follows: Anln (PIAN)'
'reply': 'Since the yellow block is clear, the hand is empty and the yellow block is on top of the blue block. Thus, we take action unstack the yellow block from top of the blue block. Since the red block is take
clear and the hand is currently helding yellow block. Thus, action stack the yellow block on tep the red block. Since the blue block is clear, the hand is empty and the blue block is an the table. Thus, we take action pick clear and the hand is currently helding blue block. Thus, we take action stack the blue block on top of the crange block. Since the goal the blue block is on top of the orange black and the yellow block is on top of the red block, The goal conditions are satisfied. \n###', 'ground_truth_plee
the blue block. Since the orange block is
(unstack d b) (stack d a) (pickkup b) \n (staak
b in'
For more details about the dataset, please refer the paper by Valmeekam al. (2023).
G IMPLEMENTATION DETAILS
G.1 AUTO-CEI SETTING
Neural Network Model We use -lama-3.--B-inntruue (Dubey et al.) 2024) as the backbone model and use Lora for Llama-3.1 --ae-...-8--BBeeee Instruns The template is demonstrated in the Listing We fill in the P Prompt by our query and train the LLM to respond by Completion.
128, a 64) fine-tune []
We use the Fastchat template designed
Listing 5: Chat Template
chat between curious human and artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the hunan's questions.
### Human: (Pronpt)
### Assistant: (Complction) ###
'hhe 1.1ma-3.1-4 3.1--B.inmiiii- 1-----Binistrce model is downloaded at httpps//higgiingaacecco//// 11ama/
1.1 11ama- 88- Inatruct
20

---

Published as a conference paper at ICLR 2025
Initialisation As suggested in the paper, used R-Tuning (Zhang et al. 2023 as initiali- sation strategy. Overall, we use LLM to generate various responses via random sampling. will produce correct and incorrect responses with intermediate reasoning steps. We collect those incor- rect responses and attach an expression of acknowledging limitations. In practice we use tempera- ture=1.0 and top-p= = 0.95 sample responses. We sample 16 responses in our experiments for each query. The expression of acknowledging limitation is "Sorry, I am not sure if answer the question correctly. There might be mistakes in my answer".
Expert Iteration In this process, the language model will keep sampling responses, and we select the correct responses or have sufficient reasoning attempts before acknowledging incapability. We use temperature=1.0 .0 and top p 0.95 to sample responses and sample 16 responses for each query.
Expert Iteration resamples the response according to the reward function defined in Equation T In this function, C₁ is initialised by the mean value of reasoning steps produced by the initial LLM policy the validation set, and 02 is computed to make sure that the number of reasoning steps has reached e₁ + 2a (a is the standard deviation of the reasoning steps by initial policy), then the reward should be higher than 0.9. It means the answer is longer than roughly 95% of the reasoning trajectories in the validation set, assuming the distribution of reasoning steps is normal distribution. We use the keywords to identify the length of the reasoning steps. We set a fixed format for different the
reasoning tasks, and each task will have marker for length identification. In general, keyword "Since" to identify the number of reasoning steps for the Blocksworld task, and ". An" for the BoardgameQA and MATH tasks.
Curriculum Update The curriculum updates the reward function gradually. Specifically, it grad- ually increases reduces the value of ej defined Equation [] We use the hill-climbing algorithm to search for the optimal value of the objective function f under the assumption that there is sub- optimal value of f wrt C1. We define the steps updating C1 by min (0.5, 4a /10), where a denotes the standard deviation of the reasoning length produced by the initial LLM policy. We assume that the domain is Iu 2α, μ + 2o]] where μ denotes the mean length of the initial LLM policy, and the domain determines the step size of updating C1. We also do not want to make the step too large, thus capping by a threshold value of 0.5, an empirically selected value. As such, the hill climbing algorithm will search the neighbouring value of с₁ according to the step size d and identify if the neighbouring point has a larger / value; otherwise, we believe ej has reached its optimal point to achieve the highest f.
G.2 TRAINING DETAILS
The experiments are conducted in server with NxNvidia A100 GPUs. We use DeepSpeed Stage 2 conduct the training During the SFT stage, we choose the batch size 256, and epoch number is 5 for initial SFT process, and 2 for SFT in R-Tuning and El process. The learning rate of SFT is We use the HuggingFace SET traine to conduct the SFT training automatically.
1 x 10-4,
H DETAILS OF BASELINE METHODS
H.1 R-TUNING
The setting is the same as AUTO-CEI in the initialisation part of Section G.
H.2 RLKF
Reinforcement Learning from Knowledge Feedback (RLKF) (Xu et al. 2024) is an RL-based post- training method for hallucination mitigation. It aligns the LLM''s behaviours according to the con- sistency and correctness of LLM's sampled responses: it teaches LLM to respond assertively its responses are correct and consistent and acknowledges "I don't know" if its responses mainly wrong or inconsistent. We implemented DPO version of this method as one of our baseline.
'The configuration of DeepSpeed Stage 2 can found ht http://twwww..ooo////g/innnnnnea
trans
21

---

Published as a conference paper at ICLR 2025
Data collection
Xu et al.
2024) define the pairwise preference to define the reward function:
(x, Ye)
(x, yr),
for
All samples are correct;
pair
(xr Ye)
(I, yu
for
There exists correct response in samples;
(x, Mr)
(x,, Vie),
for
All samples are wrong.
In this equation, x refers to the query, and means the response sampled from LLM. denotes the correct response, Vhe denotes the wrong responses, and y,- denotes the refusal response (acknowl- edging "I don't know"). We use this strategy to build up the dataset. Since we do not have a special out-of-domain dataset in our setting, we only build the reliable in-domain preference dataset. We sample the LLM using temperature= .0 and cop-p=0.95, and sample 16 instances for each query. The template refusal responses in RLKF the same as the R-Tuning. If all responses are correct, we will annotate all 16 responses using the refusal template to form 16 pairs of preference data. there are correct and incorrect responses, we sample 16 preference pairs randomly. If samples wrong, we use all of the wrong responses to generate the refusal responses and form 16 pairs of preference data.
Training We use the TRL DPOTraine conduct the training. Similar the previous setting, we use Lora (r = 128, 64) for fine-tuning. The experiment is implemented in 8xNvidia Al A100 GPUs. The batch size of our training 256, and the learning rate 1.0 10-5, and the epoch number is 5.
22


Reviews:
[{"review_id": "eoNSUlJ1nX", "reviewer": "Reviewer_BWMo", "summary": "The paper addresses the problem of hallucinations in reasoning for large language models. Specifically, in methods that use expert iteration for improving this reasoning in LLMs. The authors propose adding a Refusal option to the EI pipeline and rewarding the refusal when the problem has a certain level of difficulty (length of reasoning is used as a proxy for difficulty). Based on the reward that a response gets, the data for the next iteration of EI is selected accordingly. To balance refusal and improvements in reasoning, the authors propose using a curriculum to balance the two objectives. Based on measures of accuracy and refusal rate, the authors show the effectiveness of their method compared to baselines and ablations.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly motivated, the problem of hallucinations in LLM reasoning has not been explored as much.\n- The paper presents the problem, the methods and the baselines clearly.\n- The results on the three datasets, MATH, blocksworld and boardgameQA are comprehensive and provide sufficient evidence of the utility of the method.\n- The authors also present sufficient ablations of their method, with varying versions of the curriculum used.", "weaknesses": "- Missing citations for central EI works: STaR (https://arxiv.org/abs/2203.14465), Training Chain-of-Thought via Latent-Variable Inference (https://arxiv.org/abs/2403.04642)\n- What other measures of difficulty are possible (very recent paper that the authors can choose to ignore since it came out after the paper deadline: https://arxiv.org/abs/2410.04707)? Can a linear probe decode difficulty? A discussion is needed.\n- Similarly, are there other mechanisms of knowing when to say I dont know possible? Is a linear probe enough for refusal? Like in Language models mostly know what they know (https://arxiv.org/abs/2207.05221).\n- Where could the current framework of length based difficulty fail? When sampling multiple times, what is the variation in the number of steps needed to reach the answer for a problem?\n- Are the number of iterations for Auto-CEI and EI etc matched? More generally, how does the amount of training data / compute required vary across the method and baselines? I think some of the implementation details could be moved to main in the revision. This would help readability.\n- How could this method be extended beyond expert iteration, to more online methods of learning like PPO or REINFORCE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldauhLAEQ4", "reviewer": "Reviewer_4yYx", "summary": "This paper proposes an automatic curriculum expert iteration (AUTO-CEI) to enhance LLM reasoning and align responses to the model's capabilities. Enable the model to answer within its limits and decline when tasks exceed them.\nExpert iteration can explore the reasoning trajectories near the LLM policy, guiding incorrect paths back on track to reduce compounding errors and improve robustness.\nAUTO-CEI uses the length of reasoning steps to measure difficulty and designs an automatic curriculum for expert iteration that rewards correct reasoning.\nAUTO-CEI can automatically estimates the boundary of LLMs' reasoning capacity to achieve a reasonable alignment to maximize capacity and control behaviors.", "questions": "Why the process of update R is called curriculum learning? IMO, I think it is more like a search process and curriculum learning is about first learning elementary stuffs and then complex stuffs.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Using expert iteration is a good idea.\n2. The authors consider the trade off between avoiding hallucinations and avoiding laziness, which is important.\n3. This paper aims to avoiding reasoning hallucinations and teaching the model to say IDK to reasoning tasks which beyond its ability.", "weaknesses": "1. Although considering the trade off between helpfulness and laziness, the paper control this trade off by hyper-parameters, instead of proposing some principles or methods to chose the optimal hyper-parameters.\n2. The evaluation metrics in the paper are incomplete; for example, IDK only measures the proportion of times the model outputs \"IDK\" without assessing the accuracy of those IDK responses.\n3. The experiments in the paper may involve unfair comparisons, as AUTO-CEI conducts multiple searches for EI, resulting in significantly more training steps and data compared to other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUMWDTm8TU", "reviewer": "Reviewer_HEc4", "summary": "This paper enhances LLM calibration ability in reasoning tasks by mitigating hallucinations and reducing lazy behaviors. The method uses a curriculum of expert iteration, balancing answering the question and saying 'I don't know' by adapting the reward system according to the difficulty of reasoning tasks. It shows great balancing in assertiveness and conservativeness.", "questions": "1. The MATH dataset has manually labeled question difficulty; it would be helpful to include a figure illustrating the correlation between generated text length and question difficulty as a validation of this signal.\n\n2. Is the method effective across datasets with different distributions?\n\n3. There are many established metrics for measuring uncertainty with different confidence levels, such as AP used in R-tuning. The current paper only reports results for the binary case, i.e., answering or refusing to answer. I am curious about the performance of this method at different confidence levels.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a reasonable motivation that previous approaches to addressing hallucination problems often suffer from overcorrection, leading to overly conservative responses from LLMs on many questions.\n\n2. The paper introduces an innovative reward mechanism that dynamically balances the choice between responding or refusing to answer based on question difficulty. Subsequent experiments demonstrate that this method effectively handles the trade-off between answering and refusing. Figure 3 also illustrates that as the generated text length increases, more questions are declined.\n\n3. The writing is clear, and the figures are well-designed, enhancing the overall readability of the paper.", "weaknesses": "1. The paper’s reward uses the length of the generated text to measure the level of the exploration. This limits the method’s generalizability and scalability, as it’s challenging to find a universal text processing metric for different types of text (e.g., code, tables). Furthermore, reward parameters such as c1 and c2​ also vary across datasets, which makes me concerned about the method’s effectiveness in situations where training and testing distributions are not similar. \n\n2. The experimental results show that even the best method in this paper achieves overall accuracy significantly lower than Vanilla EI. While I understand that incorporating refusal may lead to a drop in accuracy, achieving performance closer to Vanilla EI would be more compelling, as the current method shows that adding a refusal option will significantly reduce the accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IppHhyrNp9", "reviewer": "Reviewer_d7Jy", "summary": "This paper proposes Automatic Curriculum Expert Iteration (AUTOCEI) to enhance LLM reasoning and align responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them, so as to mitigate hallucination and laziness in reasoning tasks. Through experiments on  BoardgameQA, MATH and Blocksworld with Llama-3.1-8B-instruct, The authors demonstrate the effectiveness of AUTO-CEI, achieving superior alignment by effectively balancing assertiveness and conservativeness.", "questions": "See weaknesses.\n\nAdditionaly,\n\n(1) Have you tested the correlation between the number of reasoning steps and difficulty, as this is a key assumption of AUTO-CEI? If not, I suggest conducting a test experiment on the MATH dataset, considering it has manually labeled difficulty tags.\n\ntypos:\nline 286 N -> K", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* This paper is well-written and presents clear ideas.\n\n* The idea of aligning responses to the model's capabilities–assertively answering within its limits and declining when tasks exceed them is novel.\n\n* The  motivation behind AUTO-CEI is straightforward and strong.", "weaknesses": "* AUTO-CEI introduces additianal training overheads. Considering that the process of AUTO-CEI includes EXPERT ITERATION (each iteration needs large amount of resampling), the additional training overhead can not be ignored. I suggest the authors align the sampling cost between different baselines.\n\n* Limited validation across models. The effectiveness of AUTO-CEI is validated only on Llama-3.1-8B-instruct. Further exploration is needed to assess the generalizability to other models.\n\n* The performance is significantly affected by the hyperparameter λ. Table 2 shows that the performance of AUTO-CEI fluctuates greatly under different λ, which can lead to a considerable amount of additional cost during actual usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)