Learning From Correctness Without Prompting Makes LLM Efficient Reasoner
Authors: Yuxuan YAO
Han Wu
Zhijiang Guo
Zhou Biyan
Jiahui Gao
Sichun Luo
Hanxu Hou
Xiaojin Fu
Linqi Song
Venue: COLM (2024)
Keywords: LLM, Reasoning, Self-refine
TL;DR: The proposed framework prioritizes learning from correct reasoning steps and measures confidence for each reasoning step based on generation logits for better multi-step reasoning.

Abstract: Large language models (LLMs) have demonstrated outstanding performance across various tasks, yet they still exhibit limitations such as hallucination, unfaithful reasoning, and toxic content. One potential approach to mitigate these issues is learning from human or external feedback (e.g. tools). In this paper, we introduce an intrinsic self-correct reasoning framework for LLMs that eliminates the need for human feedback, external tools, and handcraft prompts. The proposed framework, based on a multi-step reasoning paradigm \textbf{Le}arning from \textbf{Co}rrectness (\textsc{LeCo}), improves reasoning performance without needing to learn from errors. This paradigm prioritizes learning from correct reasoning steps, and a unique method to measure confidence for each reasoning step based on generation logits. Experimental results across various multi-step reasoning tasks demonstrate the effectiveness of the framework in improving reasoning performance with reduced token consumption.

Published as a conference paper at COLM 2024
Learning From Correctness Without Prompting Makes LLM Efficient Reasoner
Yuxuan Yao", Biyan Zhou, Sichun Luo, Linqi Song*
Department of Computer Science
City University of Hong Kong Tat Chee Avenue, Kowloon, Hong Kong SAR
yuxuanyao3-c0mm cityu. eec.y.u.dduu..e edu.hk, linqi. song@cityu. edu.hk
Han Wu", Zhijiang Guo", Jiahui Gao, Xiaojin Fu, Huawei Noah's 's Ark Lab Hong Kong SAR guozhijiang Chuawei. jaaaaaaaangniiii...... com
Hanxu Hou,
Department of Communication Engineering Dongguan University of Technology thx@dgut.edu edu.cn
Abstract
Large language models (LLMs) have demonstrated outstanding perfor- mance across various tasks, yet they still exhibit limitations such as hallu- cination, unfaithful reasoning, and toxic content. One potential approach to mitigate these issues is learning from human or external feedback (e.g. tools). In this paper, we introduce an intrinsic self-correct reasoning frame- work for LLMs that eliminates the need for human feedback, external tools, and handcraft prompts. The proposed framework, based on a multi-step reasoning paradigm Learning from Correctness (LECO), im- proves reasoning performance without needing to learn from errors. This paradigm prioritizes learning from correct reasoning steps, and a unique method to measure confidence for each reasoning step based on genera- tion logits. Experimental results across various multi-step reasoning tasks demonstrate the effectiveness of the framework in improving reasoning performance with reduced token consumption. The code is available https://github. ctmp///tthbbcoo/e/e com/ wtt.m///taa//auaaaea.....
1 Introduction
Large language models (LLMs; Brown et al. 2020 OpenAl 2023 2023 Touvron et al. 2023 have
exhibited remarkable performance on diverse range of natural language processing bench- marks (Hendrycks et al. | Srivastava et al.) (022) and also showcased promising results on real-world applications (Wu et al. 2023 Uhrunavukaaasuu et al., 2023). However, it is imperative to acknow ledge that LLMs still possess certain limitations, For instance, the occurrence of undesirable behaviors like hallucinations (Rawte et al. 2023), generating harmful content (Bai et al. (2022), and non-adherence to established rules and constraints
Ouyang et al.) 2022 Peng et al. 2023) remains largely unexplored.
One extensively employed approach to address these problems is learning from feedback (Pan et al. 2023). It involves guiding LLMs to improve their responses through a cycle of trial, examination, and correction. During the examination phase, feedback provided to
"Equal Contribution.
t *Corresponding Authors.

---

Published as a conference paper at COLM 2024
While not stop criteria():
TIMMNNSTRA TRATIOONS
Question: heas makdey and dendic
bot and tout half keg tookko dee sence vedk, tie maay
INPUT:
-DEMUNSTRATTURE
OUTPUT:
Quettion:
sel rosh sillin leng He dos hes fimes week Hiwe meny with spenaiie raskig
desdert
Answer: Lat's Bank vico by
Adseren inpplyy
Saggh:
Supl: Jis oper de heas so to redds boox.
Shoph: lin it see with spend of per
OUTPUT:
3tep3: in fen sprno to
speada
Supt Joos
Suppl this teaa wods spadds
beatiely apenslag
INP
Sieple wooks lie teal
Seepth jan zoex the tires work ao opends of hoars Amanguae
Skep. Themmwers (26
1:08
Fng snefiiice
dicrpptes soxr
stop confidence
transition cone divergence
The framework of LECO, LECO first obtains an initial solution for the input problem. Then, we progressively collect the correct steps from the latest solution until the final answer is obtained.
Figure 1:
identify the shortcomings in the trial answer and guide the necessary corrections. Prior efforts (Huang et al., 2023a Gou al. 2023a) have confirmed high-quality feedback can offer valuable insights into further corrections. Although human feedback (Ouyang et al.
2022 Fernandes et al. 2023) and external tools feedback (Gou et al. 2023a b) are generally valuable, they are either expensive to collect or heavily dependent on the abilities of the selected tools. To eliminate external intervention, another popular line of research is self- correction, where the model progressively learns from the feedback it generates internally, without relying on external sources (An et al. 2023). However, Huang et al. (2023b) recently suggests that LLMs do not possess the inherent capabilities to find the errors and rectify their responses just by designing the prompts. More frustratingly, these methods often require creating extensive and elaborate handcraft prompts guide the model in acquiring and understanding the feedback, which is time-consuming and labor-intensive process, finally tuning our researchers into "prompt engineers".
In this work, we present novel intrinsic self-correct reasoning framework that eliminates the need for human feedback, external tools, and handcraft prompts. Different from the existing self-correction methods, which are predominantly based on learning from errors (An et al. 2023 Gou et al. 2023a), we propose a new multi-step reasoning paradigm known as Learning from Correctness (LFCo). As illustrated in Figure [ we begin by assigning confidence score to each reasoning step in the first-round reasoning path. The step with the lowest confidence score will be identified steps before this point are considered to be "correct". Then, the correct steps, considered as "correctness", are appended to the input, and repeat the reasoning process. While the insight of learning from errors comes from the learning process of human students, the motivation behind our method is derived from progressive learning (Wu et al., 2019 Fayek et al.) 2020), where correct reasoning steps are gradually accumulated to ultimately approach the correct answer. Furthermore, we also introduce an efficient method to measure the confidence for each reasoning step based on the generation logits, without the need for additional tokens or external tools. Specifically, we jointly consider the average confidence of each token within a step, the confidence divergence of a step, and the probability of step transmission to calculate the overall step confidence. We surprisingly find our method can identify almost 65% incorrect steps. We conduct experiments with both closed-source models (e.g. GPT-3.5 and GPT-4) and open-source models (e.g. DeepSeek; (Shoo et al. 2024) on various multi- step reasoning tasks, including arithmetic reasoning, commonsense reasoning, and logical reasoning, show that our framework can significantly improve reasoning performance with less token consumption.
the earliest potential error step, and the
2

---

Published as a conference paper at COLM 2024
Our primary contributions include 1) we propose novel multi-step reasoning paradigm pe learning from correctness, dubbed as LECO, which progressively accumulates the correct steps and approaches the final answer; 2) we challenge the conventional belief that high- quality feedback can only come from external sources and propose unique intrinsic method to measure the confidence for each reasoning step, and 3) Both the off-the-shelf and open- source models can benefit from LECO on various multi-step reasoning tasks with reduced token consumption. More excitingly, LECO completely eliminates the need for prompt engineering.
2 Related Work
Learning from Feedback Improving LLMs through learning from feedback has become prevalent strategy, notably through reinforcement learning from human feedback, which seeks to align LLMs with human values by refining their outputs based on feedback (Ouyang et al. 2022 Bai et al. 2022 Touvron et al. 2023). However, this method faces challenges such as high costs due manual labor and a lack of real-time feedback capabilities (Pan al., 2023 Fernandes et al. 2023). An alternative strategy involves using self-correcting LLMs, which rely on automated feedback to iteratively adapt and understand the consequences of their actions without heavy reliance on human intervention. This feedback can be derived
from outside sources such as other models Yang et al. 2022 Lightman et al. 2023 Xiong
et al. 2023), tools (Huang et al.) 2024 Lu et al. 2024b) , knowledge bases (Gao et al. 2023 Yu
et al.
2023), or evaluation metrics Jung et al. 2022 Welleck et al. 2023).
External feedback leverages external perspectives to identify errors and verify factual accuracy, offering insights that may not be recognized by the LLM alone. Conversely, feedback can also be internally generated, where the LLM evaluates and refines its output iteratively until a desired quality is achieved (Madaan et al. 2023 Shinn et al. 2023 Helbling et al. 2023 Xie et al. 2023). This self-improvement mechanism is particularly valuable in scenarios where external feedback is scarce or restricted (Yan et al. 2023 Lu et al. 2024a). However, Hlowever/Huang Huang et al. (202b)) suggests that LLMs struggle to independently identify and correct errors through self-generated prompts. Recent effort (Gonen et al. 2023) show that an LLM's familiarity with a prompt's language predicts its effectiveness, with lower perplexity prompts leading to better performance. Unlike existing efforts, LECO focuses on learning from one's correct reasoning steps, without the need for feedback mechanisms including human intervention, external tools, tailored prompts.
Reasoning without Prompting Recent studies have been focusing on improving the reasoning abilities of LLMs through various methodologies, primarily centered around the enhancement of prompting techniques. These works include few-shot prompting with
intermediate steps augmented demonstrations (Wei et al., 2022 Fu et al., 2023] Yao et al.
2023:
Wang et al.] (202)) or zero-shot prompting with specitic instructions (Kojima et al.
2022
Yasunaga et al. 20033 Although these methods have shown promising results, their
effectiveness is often constrained by their task-specific nature and the labor-intensive process of designing prompts, leading to inconsistent outcomes across different tasks (Ye & Durrett
2022
Zhou et al., 2023).
Another strategy to facilitate reasoning involves instruction tuning, which leverages significant volume of chain-of-thought (CoT) data (Chung et al.) 2022 Mukherjee et al., 2023
Gunasekar et al. 2023 Luo et al.) 2023). Recently, Liu et al. (2024) proposed to tune
LLMs by comparing the logit differences between pair of tuned and untuned smaller
models, showcasing improvements in reasoning without CoT distillation. In contrast to
these methods, our LECO introduces an intrinsic self-correct reasoning mechanism that does not depend on fine-tuning or auxiliary models.
Additionally, there has been an interest in refining decoding algorithms specifically for reasoning. Notably, contrastive decoding (Li et al. 2023) has been developed to enhance model's 's generation quality by adjusting the logits from smaller models, with recent research indicating its potential to boost reasoning performance (O'rrien & Lewis 2023). Wang & Zhou 2024) discovered that CoT reasoning patterns naturally occur within the
3

---

Published as a conference paper at COLM 2024
decoding trajectories of LLMs, leading to the development of CoT-decoding, which aims to identify more reliable decoding paths. Such advancements present a promising avenue to augment the efficacy of LECO. Future work could explore the integration of these decoding algorithms to extend beyond the current use of greedy decoding.
Methodology
We introduce LECO, a learning from correctness framework, designed to enhance multi-step reasoning capabilities. Our core insight that providing the model with more correct reasoning steps helps it narrow down the search space for the solution. This facilitates the process of reaching the final answer. To achieve this, LECO utilizes a prompt-free method to calculate the confidence score of each reasoning step. By identifying the most reliable steps, the model can then leverage these insights to guide its reasoning process.
3.1 Step Confidence
Preliminary In generation tasks, logits represent the log probabilities of candidate tokens being chosen as the next word. Confidence, on the other hand, refers to model's certainty in its prediction. Within reasoning tasks, step confidence specifically measures the model's belief in the correctness or factual basis of each reasoning step. Inspired by Li al. (2023), we propose leveraging logits to estimate step confidence. We further design three logit- based scores that comprehensively evaluate confidence from both intra- and inter-step perspectives.
Algorithm 1 Confidence-based Reasoning Algorithm
Require: input Xn, model M, demonstration Demox, stop condition stop(*)
1: -M (xo, Demox)
b Initial Generation (Eq
2: for iteration E 1,..., do
3: if not stop(y,) then
> Stop Condition
4: for step O,..., yy do
5:
= Lowest end for end X; f- Xt-1 y e)
Lowest Confidence Step (Eq
6:
7:
8:
9: 1/11 = M (x,, Demox)
Rethink Generation
10: end for
return y/+
Formally, we denote the entire reasoning path as S (5,, .,,)).,s,), consisting of n indi- vidual steps. Each reasoning step s; (4, f12. 4. a sequence of tokens. We then apply the Softmax function the logits score to obtain the probabilities Pt,j for each token tij.
Average Token Score A straightforward approach to measure step confidence is by aver- aging the token probabilities within given step. This average reflects the model's certainty in its reasoning during that step. Therefore, we define single-step confidence as:
1
aug score;
Pij
(1)
Step Divergence Score While average token probability seems intuitive, it can be mis- leading. Within a step, most tokens tend to be common words with high confidence scores but carry little information. Conversely, tokens crucial for reasoning, e.g. mathematical
4

---

Published as a conference paper at COLM 2024
calculations, often have lower confidence. This paradox leads to a high average token confidence for the entire step, which contradicts our goal.
To address this issue, distribution uniformity of token probabilities within a step. Ideally, we want the token probabilities to be both high and evenly distributed across all tokens. To achieve this, we formulate the step divergence score based on the Kullback-Leibler Divergence (KLD; Kullback & Leibler 1951) between the normalized distribution P, norm(pi,, Pi2, Pilss)) of the token probabilities and the uniform distribution U:
propose the step divergence score. This metric measures the
diver score; In (KLD" (P;, U) +1),
(2)
where T is the rescaling temperature for the KL divergence value, as the step divergence score is expected to vary between 0 and 1. In this work, T is set to 0.3.
Inter-step Transition Score Following the intra-step measurements, we sought t to quantify the transition between consecutive steps. Our preliminary experiments yielded two key insights: 1) steps with lower overall confidence tend to have lower confidence levels specifically in the initial heading tokens (typically the first three), more dicussions can be found at Section D 2) These initial heading tokens were also the most likely to change across different program runs. Based on these observations, we propose using the probabilities of the heading tokens in step to represent the inter-step transition score between that step and the subsequent one. In other words, the transition score is determined by:
1 K K
Irans score;
Pij
where K is set to 3 here. Further analysis of hyperparameter settings is discussed in Section
Overall, the confidence score score of step is denoted as,
S score ang score; + trans score; diver score;
(4)
3.2 LECO: Learning From Correctness
While leveraging step confidence scores, previous approaches (Gou et al. 2023a Huang et al. 2023a) heavily rely on prompting LLMs to pinpoint and rectity erroneous steps. This dependence on prompts makes them rather sensitive. Our LECO framework tackles this issue by iteratively gathering correct steps and consequently refining the search space for potential reasoning steps. As depicted in Figure LECO operates in a two-stage process.
Initial Stage Given input xo and the corresponding demonstrations Demo,, the model M generates an initial answer yo:
y/ MM (xo, Demox),
(5)
where $1,,, consists of multiple reasoning steps.
Rethink Stage In this stage, we first calculate the confidence score for each step within the initial solution y/ based on Eq. 4 We take the step with the lowest step confidence or the earlier one of the two steps with the lowest step confidence as the earliest error step, which depends on the complexity of the reasoning problems. Denote the selected error step as 5,, S e S lyol we name the steps before 5, as "correctness" (s<e). Then we iteratively append the correctness to the input and repeat the reasoning process with LLMs. At f-th iteration, the workflow can formulated as,
We always use "Let's think step by step." " Kojima et al. 2022 the first step of the reasoning path and we do not consider the step confidence of this sentence.

---

Published as a conference paper at COLM 2024
Commonsense
Arithmetic
Model
Method
Date
Avg.
CSQA
StrategyQA
AQuA
SVAMP
GSMSK
CoT
80.80
79.69
73.25
51.37
84.00
77.86
74.53
Complex
84.20
77.33
54.49
81.25
80.89
74.67
ADPSC
83.60
75.92
68.99
51.97
78.89
79.00
73.06
SC
84.48
77.47
70.37
55.31
81.6
81.03
75.08
GPT-335
RCI
74.97
68.34
51.94
35.30
79.95
75.25
64.33
82.8
79.77
71.15
52.72
85.00
778.24
74.93
LECo+CoT
(+2.00)
(+0.08)
(-2.12)
(+1.15)
(+1.00)
(+0.38)
(+0,40)
84.92
77.68
71.05
56.77
82.35
82.33
75.85
LeCo+Cumplex
(+0.72)
(+0.35)
(+1.21)
(+2.28)
(+1.10)
(+1.44)
(+1.18)
CoT
92.80
87.46
83.63
71.60
93.05
94.84
87.23
Complex
90.40
86.40
82.75
71.94
90.90
95.42
86.30
ADPSC
89.20
70.08
94.09
85.32
SC
90.72
86.81
83,75
72.19
93.49
95.51
86.67
GPT-4
RCI
89,88
86.16
74.62
47.39
90.59
86.23
79.18
93.60
87.63
83.25
71.99
93.55
95.14
87.53
LECo+CoT
(+0.80)
(+0.17)
(-0.38)
(+0.39)
(+0.50)
(+0.30)
(+0.30)
90.80
86.90
83.97
72.33
91.40
95.68
86.85
LeCo+Complex
(+0.40)
(+0.50)
(+1.22)
(+0.39)
(+0.50)
(+0.26)
(+0.55)
Table 1: Performance of GPT models on logical reasoning, commonsense reasoning, and arithmetic reasoning tasks.
Model
Method
Avg.
Algebra
Count
Geometry
Iter
Num
Prealgebra
Precaculus
Complex
58.55
30.80
29.83
17.46
31.93
61.11
15.39
35.01
ADPSC
54.22
28.18
26.89
13.69
28.93
59.70
14.34
32.28
GPT-3-5
SC
56.20
30.87
29.98
17.65
32,25
61.80
18.13
35.27
RCI
49.79
24.25
18.76
10.16
25.09
53.71
13.08
27.83
58.72
34.70
31.89
18.80
33.37
62.21
18.53
36.89
LRCo+Complex
(-0.17)
(43,90)
(+2.06)
(+1.34)
(+1.44)
(+1.10)
(43.14)
(+1.88)
Complex
69.06
50.32
38.62
25,33
46.39
76.98
28.23
47.85
ADPSC
60.13
40.13
30.55
15.84
37.39
21.10
39.23
GPT-4
SC
71.04
52.23
40.48
25.89
30.37
77.84
30.51
49.77
RCI
65.49
46.93
29.71
16.56
43.68
73.99
27.07
43.35
LeCo+Complex
71.92
53.27
41.13
27.49
49.14
78.29
32,02
50.47
(+2.86)
(+3.05)
(+2.51)
(+2.16)
(+2.75)
(+1.31)
(+3.79)
(+2.62)
Table 2: Performance of GPT models the MATH dataset.
- 1 (s e), y1 -M (x,, Demox).
(6)
LECO alternates between input updating and rethink response generation until the stopping condition is met. The process either stops maximum iteration number T or identifies the two consecutive same answers. The algorithm can be found in Algorithm 1
Experiments
Dataset and Baselines We evaluate the performance of LECO using a variety of datasets and baselines. The datasets are categorized into three reasoning types: arithmetic reasoning, commonsense reasoning, and logical reasoning. The arithmetic reasoning datasets include GSMBK (oobee et al. 2021), MATH (Hendrycks et al. 2021b), AQuA (Ling et al. 2017), and SVAMP Patel et al. 2021). For commonsense reasoning, we use CSQA (Saha et al. 2018 and Strategy (Geva et al. 2021). The logical reasoning dataset is represented by Date Understanding (Srivastava et al. al. 2022).
Our evaluation utilizes both off- the--sheleff models, such GPT-3.5- and GPT-4, and open-source models like DeepSeekMatth-RL RL-7B (Shao et al.) 2024). The open-source models are chosen for their superior performance well-known mathematical datasets. We also

---

Published as a conference paper at COLM 2024
MATH
Model
Methods
CSM8K
Aug.
Algebra
Count
Geometry
her
Num
Prealgebra
Precaculus
Comples
79.76
69.50
40.08
38.41
21,39
40.56
68.35
24.18
47.87
DeepSeek
80.14
20.51
22.15
68.52
LeCoi Complex
(+0.38)
(-0.55)
(+0.22)
(-021)
(-0.56)
(+2.15)
(+0.17)
(-0.19)
(+0.50)
Table 3: Performance of DeepSeekMath-7B on GSM8K and MATH, where Count represents counting and probability subset; Iter refers to intermediate algebra subset; Num means number theory subset.
incorporate two suites of public demonstrations, namely exemplars from vanilla CoT (Wei et at. 2022) and exemplars from complex-CoT (Complex; Fu et al. 2023), which are prompts with higher reasoning complexity to improve language models multi-step reasoning ability.
We compare LECO with several baselines, including self-consistency (SC; Wang et al. 2023 adaptive self-consistency (ADPSC; (AgDac,,Agarwaal et al. 2202), and RCI (Kim et al. 2023). SC polls the LLM multiple times and outputs the most frequent solution. ADISC tollows SC manner while conserving iterations via dynamically adjusting the number of samples per question using lightweight stopping criterion. RCI is representative work of learning from errors, which identifies errors and then self-corrects using designed prompts. In most runs, we use greedy decoding with temperature of 0, except for the adaptive self-consistency and self-consistency settings, where a temperature of 0.7 is applied. The iteration number of self-consistency is set to 10. All experiments are run 10 times with different seeds, and the average scores are reported.
Main Results As shown in Table 1 and 3 LECO consistently improves the reasoning performance across the board. Particularly noteworthy is its outstanding performance in arithmetic reasoning, especially evident in the MATH dataset. The MATH dataset is renowned for its challenging nature, like more intricate problems and the need for more reasoning steps, with common CoT approaches demonstrating limited effectiveness this benchmark. However, LECO effectively addresses this complexity by progressively collecting correct steps, thereby reducing reasoning perplexity and achieving substantial improvements. We also find that high-quality demonstrations are preferred when using LECO as larger improvements are consistently observed with LECo+Complex.
For commonsense reasoning tasks, LECO obtains slight improvements or comparable performance against baselines. Except for the StrategyQA dataset, some performance drops are spotted. We think this is because commonsense reasoning necessitates incorporating knowledge concerning events and their relationships. However, LECO primarily focuses on augmenting intrinsic reasoning ability through correctness, hence moderate enhancement deemed reasonable. This finding is also aligned with observations in Lyu et al.) (2023). Conversely, remarkable improvements this task is more similar to mathematical reasoning. It is worth noting that the difficulty of the task correlates positively with the impact of LFCO, as evidenced by the substantial improvements achieved on the AQuA and MATH datasets. The primary reason for this is that the LLM tends to remain their initial reasoning path on the easy problems, offering fewer improvement rooms for LECO. For a comprehensive evaluation, we also apply LECO on the open-source model. We chose DeepSeekMath-RL--75 as demonstrates competitive performance in mathematical reasoning tasks. As shown in Table LECO can consistently improve the reasoning performance on GSMSK and MATH datasets, indicating its effectiveness on open-source models.
obtained in the date understanding dataset since
On the other hand, LECO also exhibits its superiority in reducing token consumption. As shown in Section A.2 although adaptive self-consistency has tried to reduce the iterations and token consumption by settings the early stop criterion, it still needs almost 4.46 rounds to determine the final answer while RCI needs 2.74 rounds. However, using the similar stop criterion of RCI, LECO can reach the final answer just with 2.15 rounds. This phenomenon suggests that learning from correctness is more effective than learning from errors, as it does not necessitate the model's understanding of the error cues. Additionally, during each iteration, LECO reduces APT consumption by alleviating prompting the model to identify
7

---

Published as a conference paper at COLM 2024
Datasets
GSM8K
Exact Correct
Partial Correct
Wrong
Models
Methods
GSM8K
StrategyQA
Only AVG
38
9
53
Complex
82.47
70.17
Only DIV
35
16
49
Only TRANS
42
GPT-3.5
Random
82.09
69.96
AVG+DIV
36
14
50
(-0.38)
(-0.21)
AYG+TRANS
50
16
34
Complex
95.34
82.69
DIV+TRANS
47
37
GPT-4
95.22
83.37
LECO
53
10
37
Random
(-0.12)
(+0.68)
Table 5: Fine-grained level ablation study of the three factors for calculating the step confidence. AVG denotes the average token confidence; DIv denotes the step divergence score; and TRANS de- notes the inter-step transition score.
Table 4:
Coarse-grained level abla- tion study on GSM8K and StrategyQA datasets with GPT--3.5.
and understand the errors and shortening the output length. Therefore, as shown in Section LECO reduces the token consumption by 80% 20% compared to SC/RCI.
A.1
Further Analyses
Ablation Study We conduct ablation studies at two levels of granularity. At the coarse- grained level, we explore the effectiveness of the Barning-from-correeti replacing the selection of correct steps with random choices. Specifically, in the rethink stage, we randomly choose reasoning step as the earliest error step and consider the preceding steps as the "correctness". From Table 4 we can that the random selection of correct steps generally hurt the reasoning performance, suggesting the importance of identifying the true correctness.
framework by
At the fine-grained level, we deeply investigate the design of step confidence, which involves calculating the sum of the average token confidence, step divergence score, and inter-step transition score. To minimize the time and token consumption, we employ the accuracy of identifying the earliest error step as our metric. This measurement has proven to be crucial for enhancing reasoning performance in subsequent rounds, as evidenced by the results in Table 4 To this end, we randomly sampled 100 incorrect solutions on the GSMSK dataset and manually annotated the earliest error step for these solutions. Then, we divide the predicted step into three categories, including exact correct means the predicted step is exactly the labeled earliest step; partial correct means the predicted step an error step but located after the earliest step, and wrong means the predicted step is before the target location. As presented in Table 5 LECO performs best in finding the earliest error step, with accuracy over 50%. We also observe the significant performance drops when separately adopting one of these factors. More interestingly, among the three factors, we find the inter-step transition score affects the final performance most. This finding is also well-aligned with the observations in our preliminary experiments, as stated in Section 3.1 which suggests that the heading tokens of a step warrant more attention.
correct, partial correct and wrong, wherein
Rethink Analysis As LECO and RCI are both the self-refinement framework, distin- guished by their learning mechanisms from correctness or errors, we then compare them regarding the changes in answers after the rethinking stage. As illustrated in Figure 2 on the GSMSK dataset, over 85% of the time, both LECO and RCI retain the original answer. Among the remaining instances, LECO can modify more incorrect answers to correct ones than RCI (3.7% vs. 1.5%). On the Strategy QA dataset, the performance gap between LECO and RCI is more significant, where RCI revises 24.8% correct answers to incorrect. This et al.) 2023b) that LLMs are currently
phenomenon is in line with the recent findings incapable of offfcorrection based on their own teedback. Superior to RCI, LECO cleverly uses the accumulated correct information and avoids meticulous self-evaluation prompts to achieve better reasoning performance.
8

---

Published as a conference paper at COLM 2024
GSMBK-LECO
GSMBK RCI
SQA-LECO
SQA-RCI
733 13%
2.3%
41%
87.3%
18.1%
WZR KZW W2W
No Change
Evaluation of the changes after the rethink stage. We compare our LECO and RCI GSM8K and StrategyQA datasets with GPT-3.5. W2R: the wrong answer is changed to right. R2W: the right answer is altered to wrong. W2W: a wrong answer is changed to another wrong answer. No change: The answer remains unchanged.
Figure 2:
Methods
StrategyQA
Dataset
CSM8K
Models
Methods
Datasets
GSM8K
StrategyQA
Complex
31
10
RANDOM
25
13
GPT-3.5
Complex
81.58
70.94
ORACLE
36
22
Early stop
82.03
69.31
LECO
33
21
(+0.45)
(-1.63)
Complex
95.11
81.25
GPT-4
Table 6: Oracle test on StrategyQA and GSM8K by GPT-3.5-Turbo. RAN-
95.41
81.87
Early stop
(+0.30)
(+0.62)
DOM denotes randomly selecting the earliest error step. ORACLE denotes human annotated earliest error step.
Table 7: Early Stop of LECO on the GSM8K and StrategyQA using GPT-3.5-Turbo and GPT-4.
Oracle Test We also conduct the oracle test to explore the upper bound of learning-from- correctness by directly providing the correct steps to LLMs during the rethink stage. To this end, we sampled 100 incorrect solutions generated by GPT-3.5-Turbo on the StrategyQA and GSM8K datasets, respectively. Subsequently, we manually annotate the earliest error step for these solutions. After collecting the preceding correct steps and appending them to the input, we generate an updated solution. As shown in Table promising results are obtained that 36% and 22% wrong solutions can be amended with the help of correctness. is important to note that these figures do not represent the absolute upper limit of the potential to learn from correctness since the refinement process iterative but can only label the first round. More interestingly, LECO achieves comparable performance (33 vs. 36; 21 vs. 22) with ORACLE and significantly outperforms the random choices, suggesting the effectiveness of LECO in identifying the true correctness.
Early Stop of LECO As discussed above, the majority of initial solutions would not be modified after the rethink stage, which additionally escalates token consumption and ratio of "correct incorrect". To alleviate these problems, we present early stop strategy of LeCO, which dynamically determines whether the initial solution requires refinement based the overall solution score.
Similar to the step confidence, we calculate the overall solution confidence score sin score by jointly considering the average score of step confidence and the inter-step divergence, formulated as,
sin
sIn
si score
sin diver,
(7)
sin
9

---

Published as a conference paper at COLM 2024
Norm distribution of GSM8K test set
Norm distribtiion of GSMBK sample
Count (100)
Count (10)
(8)
Score
(6)
Score
Figure 3: The distribution of correct and incorrect solutions of GSM8K by GPT-3.5-Turbo.
The curve in pink represents incorrect answers, and the curve in blue represents correct answers.
where scorc is the confidence score of i-th step, obtained by Equation sin diver denotes the KL divergence between the normalized step scores S norm score, ..., S|stn| -score) and an equal-length uniform discrete distribution, analogy to the Equation
Firstly, we conducted the test on the GSM8K dataset using GPT-3.5-Turbe and recorded the solution confidence scores following Equation / As shown in Figure 3(a), we observed that the distributions scores for both correct and incorrect solutions consistently tend follow the norm distribution, with the average point of correct answers notably surpassing that of incorrect ones, We aim to employ this discrepancy to early stop the rethink stage. Specifically, first randomly sample subset from the testing data to obtain the distribution of solution scores, approximately 1/6 of the data of the entire test set used. Figure 3/b) illustrates the distribution the GSMSK sample set, which also follows the norm distribution. Then, based on the 3-0 characteristics of the norm distribution, we adopt the positive 1-o value from the score distribution of the incorrect solutions (μ o) 84% incorrect samples while only including around 50% correct instances.
threshold, which covers
As demonstrated in Table consistent improvements can be obtained with early-stop LECO over the vanilla CoT-based method. Compared to the standard LECO, there are slight performance drops since more incorrect instances are filtered and not modified. However, early-stop LECO can still maintain the performance levels intermediate to those of SC and LEčO while using fewer iteration rounds and tokens, approximately further reducing 10% tokens against the standard LECO (More details in Appendix B). We note that early-stop LECO an alternative choice for the users to achieve better trade-off between token consumption and performance.
6 Conclusion and Future Work
This work introduces LECO, intrinsic self-correct reasoning framework designed to enhance LLM reasoning performance without relying on human feedback, external tools, or handcrafted prompts. LECO leverages a multi-step reasoning paradigm, prioritizing learning from successful reasoning steps. It incorporates a novel method for measuring confidence in each step based on generation logits. Our experiments across diverse multi- step reasoning tasks demonstrate LECO's effectiveness in improving reasoning accuracy while minimizing token consumption. This approach represents a distinct pathway for augmenting LLM capabilities, offering promising avenue for advancing their aptitude in reasoning tasks. For future work, a worthy noting point is that LECO, especially its step confidence algorithm, would stand as excellent candidate for pruning the complex reasoning structures, such as Tree-of- Thooghtssss Yao et al. 2023) and -oot-oof--houhtts (Besta et al. 2023).
10

---

Published as a conference paper at COLM 2024
Acknowledgments
This work was supported in part by the Research Grants Council of the Hong Kong SAR under Grant GRF 11217823 and Collaborative Research Fund C1042-23GF, the National Nat- ural Science Foundation of China under Grant 62371411, InnoHK initiative, the Government of the HKSAR, Laboratory for Al-Powered Financial Technologies.
References
Aman Madaan Pranjal Aggarwal, Yiming Yang, and Mausam. Let's sample step by step: Adaptive--onssisency for efficient reasoning and coding with Ilms. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 2237--2336 wwwpantttoooooooooooo/oo or2/2023
Association for Computational Linguistics, 2023.
emnlp-main. main.
Shengman An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes LLM better reasoner. CoRR, abss/230020 2023. doi: 2310.20689
10.48550/ ARRXVV2310020689 URL httpps//s:///o. 0r55/1.44645555//A 48550/arXiv
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen- Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from AT feedback. CoRR, abs/ /bs/2222.08073, 2022. doi: 10.48550/ ARX1V.2222.0007.. URL https:///oi.orrg: 48550/arXi1 http://xxxvv.22.....2. 080733
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. CoRR, abs/ 23088.0877,, 2023. doi: 10.48550/ ARXIV.2308.0666. URL https
//doi.org/10.490 48550/ arXiv. 2308 09687
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/ /bb0//205.1165 2020. URL https: httppp:://arxvv.../// org/ www.:/rr//abb//////// 14165
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. language models. CoRR, abs 2210.11416, 2022. doi: 10.48550/ ARX.V.220.. 11416. U https:///o// org/ 10. 48550/ attp:550aaaiiv...o 2210 11416
Le, and Jason Wei. Scaling instruction-finetune
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/ abs/2110.14168, 2021. http:::::::/ wwwrr/v..or//bb//111..//. / 14168
11

---

Published as a conference paper at COLM 2024
Haytham M. Fayek, Lawrence Cavedon, and Hong Ren Wu. Progressive learning: A deep learning framework for continual learning, Neural Networks, 128:345-3577 357, 2020. doi: 10. 1016/J..EUUNET:220...0111 https://// org/ 10 10.1/.. nwwnet 2020 05. 011
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. T. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation. CoRR, abs 0bb//2305000555 2023. doi: 10.48550/ ARXIV.2305. 00955. URL //tp:: org//////ooooo////////// 48550 arXiv. 00955.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-basse prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://opennreviet wwt. id-yfiiiicci
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: researching and revising what language models say, using language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceed- ings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 16477-16508. Association for Computational Linguistics, 2023, doi: 10.18653 (11886633/1/0 https:///oi.. oraiddio...r//// 10. 18653/v1/2023 acl-long. 910
URL
Mor Geva, Daniel Khashabi, Flad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc, Comput. Linguistics, 9:346-361, 2021. doi: 10.1162/TACL, A) 00370. URL
https://doi.org: 10. 11 we11////aclaa__eee 00370
Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 10136-10148. Association for Computational Linguistics, 2023. doi: otgp::/.8.5../2222....mmnngmegno
URL https:://ooi
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: large language models can self-correct with tool-interactive critiquing. CoRR, abs/2305.1173$ 2023a. doi: 10.48550 2A055//AAXXV.2355117778 URL httpss//:///do org/ 10. 00.44550/arXiv..200 2305. 11738
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. CoRR, abs/ 22tp//3309117555, 2023b. doi: https://doi.org:: www.///.....///a/...
ARXIV.2302.114452 URL
2309.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abs/2306. 2023. doi: 10.48550/ ARXIV.2306.1144 URL https://doi.org::: 19550/arXiv. www..iii.............. 11644
Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. LLM self defense: By self examination, 11ms know they are being tricked. CoRR, als/ 07308, 2023. doi: 00.48550/ARXIV223088 007008. htRpstts::///o// org/10..//1 48550/ 2308. 07308
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. FRLLhttps:
12

---

Published as a conference paper at COLM 2024
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, bttp::///aaattttts-bbbcaattteeea proocccc www.:eiiiinnnnnnnin.....m cctpripppcccpppp/2/211111 htts://255aaaa667760000000062223
Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Jie M.Zhang, Hem- ing Cui, and Zhijiang Guo. Soap: Enhancing efficiency of generated code via self- optimization. ArXio, abss/2405.15889 2024, URLhttps: //api. www.nt org/
(orpuslD:2700452778
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 1051-1068. Association for Computational Linguistics, 2023a. URLLttts://cccanttit .rg. org 2023. emnlp mman
67
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. CoRR, abs/ abs/2310.01799,, 2023b. doi: 10.48550/ ARXIV.2310.01798, URL attps://doi.
48550/arXi: 2::00/arXiv..o 2310 01798
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 1266 1266-1279, Association for Computational Linguistics, 2022. doi: 10.18653/V1 http:////1//222.2ML-MMMMAM 82. URL https //doi. 100 18653/ v1////vv12222222 emn smnlp-mmin... 82
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/paperrs nips wwtpppperfille//ppeeeeepeeeeeeee files/paper/ html
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twa- sawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http: nipsprrrss.pp..p...mpmppp nips cc/ lalpp_rfplllllpppe files//e//eee wttps/feeeeeeeeeee222/11 papprr//22/haahh 9bbod291acc4acf065 wtt.:offfff1119011111201123
ernnerccee html
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-866 1951.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annull Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, 2023, Toronto, Canada, July 94, 2023, pp. 12286-12312. Association for Computational Linguisties, 2023. doi: 10.18653/VV http::://doo..rrg/// wtt..////0.0111/1/2222222
URL
acl long. 687
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step.
13

---

Published as a conference paper at COLM 2024
CoRR, abs//2355200000 2023. doi: 1044555/ARXIV.200222220 URL httpps://doi.orrg
10. 40.44855//rrz http://wwwww...........
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Compu- fational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 158-167. Association for Computational Linguistics, 2017. doi: 10. 10.18653/V1/P1220 URL https:://bi.orgg/// oatpes//ooo.orrr//00116////11111
11653/v1/P17--10
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. 2024. doi: 10.48550/ ARXIV. Tuning language models by proxy. CoRR, http:/avvvv.........
2401.08565. URL.ttss: http://d...ggg//////// org/10 100 48550/ art.v../ariv.vv
Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Yinya Huang, and Zhijiang Guo. Autocv: Empowering reasoning with automated process labeling via confidence variation. ArXiv, abs/ 2205.1006,, 2024a. URL https://api
semanticscholar. orrpa//issccclaa.oooo////rsssss 270063532
Jianqiao Lu, Zhengying Liu, Yingjia Wan, Yinya Huang, Haiming Wang, Zhicheng YANG, Jing Tang, and Zhijiang Guo. Process-driven autoformalization in lean ArXio, abs/ 1ts/2400.01940) 2024b. URL https://api. semanticchoolar org/CorpusID 270226883
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, 2023. doi: 10.48550/ ARXTV URL https://doi.org:: 48550/ arro/v 2308.09583
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful tff reasoning. CoRR, abs/2301.13377 2023. doi: 48550/arXiv 2301 13379
ARXIV.330X113333 URL https://doi.orgg
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bod- hisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. doi: http://00...rg111...666//////111 2303. 17651
Self-refine: Iterative refinement with self-feedback. CoRR, abs 10.48550/ /t1.//AXXXIV21110. 17651. URL
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of GPT-4. CoRR, 0ee//23000227077 2023. doi: 008850//AXXV2230002270000 URL https //doi.org/10 00t.::..r/11.485550///
48550/ 108550/arXiv.. 2306 02707
Sean O'Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. CoRR, abs 2339.09177 2023. doi: 10.48550 0105500/AAXXV222000001177 URL https
1ttpa/orr101.85550// 2309 09117
OpenAL. GPT-4 techniccl report. CoRR, abs/2303.08774, 2023 doi: 10.48550/ARXTV.29 08774. URL
4tt55/1000.8555//a111. arxiv. 23..2
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul E. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: http nips. cc ccppppprpeap//////peepeeeeee.. filess files/
14

---

Published as a conference paper at COLM 2024
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correctionn strategies. CoRR, abs 22t.:/338800333 2023. doi: 10.48550/ ARX00/0/2.. 03188.
40000/8855///ra/ arxiv. 2308 03188
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and
Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 2090-2094. Association for Computational Linguistics, 2021. URL http:::://6oo..gg0001111////////
doi: 10.18653 V1 /2011.2
naacl-main. 168
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, abs/ 2304.0334400 2023. doi: 10.48550/ AAX55000/ARTV2224403377... URL
https://doi.org org/ 10. (01550/arXivv http://awwvv..2.....m 2304.
Vipula Rawte, models. CoRK, abs/ 095922, 2023. doi: 10.48550/ ARXIV.2309.05922.. URL https
Amit Rawte
P. Sheth, and Amitava Das, A survey of hallucination in large foundation
4tte:/0..455//////0 arxiv 2309 05922
Amrita Saha, Vardaan Pahuja, Mitesh M. Khapra, Karthik Sankaranarayanan, and Sarath Chandar. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Shirly-Secone AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 705-713, AAAI Press, 2018, doi: 10.1609/ AAAI. V32I1.1133.. URL https:///oo. org/ 10. 1609 aaaii v3211. 11332
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL htttps://aaate org/ abs//402.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Alice Öh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 - 16, 2023, 2023. Uttpt:p://papet nips. wwtppper_ppppppaeeeeeee serer_ffle__pppeee (a2per/2023/hahh https//iieeeooosssaaat----ooeeee html
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga- Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, An- drew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of modells models. CoRR, abs htts/2200.044115 2022. doi: 10.48550/ ARX1V.220..0461. 04615. URL https://doi. /ttts:///oo.orr/100 org/10. 48550 aat.i//r//vvvv 2206 04615
language
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 19(0):193011940, 2023.
15

---

Published as a conference paper at COLM 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/ 2200 09288, 2023, doi: 10.48550/ 0000//AAXX..2207002222222 URL https
0rw//0 48550/ arxiv
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. CoRR, abs/2402.10200, 2024. doi: 10.48550/ /AA///AAAI22222220000000 URL https://doi.one org//00 18550/arXiv.2440 20w..xx....2..220....
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Rep- resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.neet 2023. URL https:://openrevier net net/ppdffid-1PLIIINMM
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Adoances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http:ttt::/pppeee nips cc/paper f files/ paper Saper/2022/aae http://2222hhhhh/a/e http Sat.a2222aac..taaaaaaaoooaaaaaaa Abrer44-bbtttactt Conf html
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh Inter- national Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
Reveewvi,, 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs abbs/2308.08155 2023. doi: 10.48550/ ARXIV.2308.0815. URL /ttpp:://boi.org/ http://0.4855////////// 2308 08155
Yu Wu, Yutian Lin, Xuanyi Dong, Yan Yan, Wei Bian, and Yi Yang. Progressive learning for person re-identification with one example. JEEE Trans. Image Process., 28(6) 2881, 2019. doi: 0111099/TTI22299228808 2891895
httpss/////doii htt.htttss////ooooo///0000./ 1rg//0.101/////11 2019
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Qizhe Xie. Self-evaluation guided beam search for reasoning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 - 16, 2023, 2023. URL http: //papers nips cc wwtp:cccppppppeeeee httpppaaapeeapppeeelppppppeeepee wwtiiee/paaere//// h0taper//023/haaae 31faa95c4dc791188acceeeeee http:a/4aaaaa00-----rartt------- Abstract- erence.rreeee..m .html
Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, Jing Tang, Chengming Li, and Xiaodan Liang. Dq-lore: Dual queries with low rank approximation re-ranking for in- context learning. CoRR, abs/ 2310.02954, 2023. doi: 0..45500/ARXI22310.00 https://doi. org/10. 488/00 arxiv 2310. 02954
URL
16

---

Published as a conference paper at COLM 2024
Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen-tau Yih, and Ziyu Yao. Learning to simulate natural language feedback for interactive semantic parsing. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 3149-3170. Association for Computational I Linguistics, 2023. URL
doi: 10.18653/ 10.18653/V1 2TA/:///222233LL-LLOOT2 acl-long. 177
Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. CoRR, abs 2210..207,677 2022. doi: /0.48550/ ARXIV.2210.0677.. URL httpp:///oii....o
org/10. artps0//axxv..0. 2210. 06774
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http:////ppaee nips. co/pppr http:/pppe______ fileffieel//le /ttpff/aaaaaape///2///eaaa/ pptppppppp//22/aahhh 27 711662255444477aa4aaaaa////44453
erence. html
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogical reasoners. CoRR, abs/2310.01714, 2023. doi: 10.48550 ARXIV23100011144 URL https:///ci.org///
48550/arXii aar00/rrivv... http:/aiii..........0 20....11102111
Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. In Sanmi Koyejo, Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: nips cc/ wttpp/pppppffsseeappeeppepappeee files/ wwtpffffapeeee//00/0/0/
html
Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. Improving
language models via plug-and-play retrieval feedback. CoRR, abs/ 2etp//2205..000,,, 2023. doi: 2305 14002
a 10.48550/ARXIII ARXIV.2305.1400 URL http:::/t////tt 0885://000000/000//////
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
17

---

Published as a conference paper at COLM 2024
Appendix
A Efficiency of Different Models
A.1 Token Consumption
Dataset
Model
Method
Date
CSQA
StrategyQA
AuQA
SVAMP
GSMBK
CoI
174K/19K
959K/77K
476K/67K
178K/45K
945K/76K
13M/169K
Complex
169K/200
14M//8K
833K/ /10KK
523K/46K
2.5M/96K
3.6M/1958
ADPSC
727K/86K
6.1M/351K
3.6M/49UK
2.7M/247K
8.8M
14.3M/716K
GPT-3.5
SC
1.7M/194K
14.4M/83M
8.3M/1.IM
5,2M/452K
25,5M/700K
36.3M/1.6M
RCI
501K/64K
L.MM/66K
2.4M/214K
1.4M/122K
6.6M/211K
10.2M/
LECo+CoT
386K/35K
2.0M/125K
1.1M/127K
406K/81K
1.9M/136K
2.5M/337K
LECo+Comples
363K/35K
3.0M/15IK
19M/182K
1.2M/104K
5.1M/ 170K
8.2M/394K
CoI
174K/19K
959K/76K
476K/58K
178K/33K
945K/72K
13M/166
Complex
169K/20K
14M/77K
833K/94K
523K/40K
2.5M/92K
3.6M/ 177K
ADPSC
721K/92K
6.2M/350K
3.7M/ 466K
3.0M/244K
108M//18KK
14.1M/ 684K
GPT-4
SC
L7M/209K
14.4M/79KK
1.MMLOM
B25/405K
25.5M/701K
36.3M/ MMMMMMMMM
RCI
393K/42K
3.5M/186K
2.3M/2266
1.7M/13IK
9.1M/261K
9.8M/ 475K
LECoICOT
357K/30K
2.0M/T10K
999K/99K
388K/58K
1.99/126K
2.5M/326K
LeCo+Complex
341K/34K
3.0M/1499 149K
1.8M/167K
12M/85K
5.5M/ 168K
7 AM/334K
Table 8: Average consumed in/out tokens with OpenAl models.
Model
Method
Algebra
Count
Geometry
Prealgebra
Precaculus
Complex
2.9M/25KK
1.2M/96K
12M/113K
2.2M/296K
1.3M/17KK
21M/146213
1.3M/165K
RCI
8.5M/701K
3.7M/305K
4.1M/321K
7.7M/658K
4.1M339KK
6.9M/426K
4.4M/491K
GPT-3.5
ADDSC
15.5M/1.5
6.2M/606K
6.M/744K
15.0M/1.9M LM
7.7M/721K
14.7M/1.IM
116M/1.5M
SC
28.9M/2 6M
11.6M/934K
120M/118M
22,2M/27M
13.1M/1.2M
213M/115M
13.5M/1990 9M
LeCo+Complex
7.4M662KK
3.3M/273K
3AMM309K
6.9M9660K
42M/349K
5.5M/361K
4.IM/183K
Complex
2.9M/210
1.2//8KK
13.1M/104K
2
L3M/144
10.4M/613K
4.3M/27KK
4.M//83KK
49M/323K
7.4/325K
GPT-4
ADPSC
16.7M/1AM AM
84M/692K
8.3M/719K
1933/22M
10/8//88K
12.0/786KK
11.4//13M
SC
28.0M/1.9M.
11,M/895K
120M/11IM
22.2M/25M
13.IM/LIM
21AM/13M
13.5M/1 1.MM
LeCo+Complex
7.M/S55K
3.2M/277K
3.5M/270K
7.2M/720K
3.6M2775K
5.0M/274K
4.2M/432K
Table 9: Average consumed in/ tokens on MATH dataset with OpenAl models.
Math
Models
Methads
GSMSK
Algebra
Count
Gevenetry
Iter
Num
Preuigebrra
Precaculus
DeepSeck
Complex
LIM/159K
L3M/272K
LeCo-Complex
8.7M/589K
2777355K
255T/ 255T/ToK
M.M./T.M
3.MM715KK
A.IMT/TOKK
Table 10: Average consumed in/out tokens MATH and GSM8K datasets with DeepSeek model.
A.2 Average Iterations Numbers by Different Methods and Models
Table and 12 present the average iteration numbers on arithmetic reasoning, common- sense reasoning, logical reasoning, and complex mathematical reasoning using OpenAI models. Table 13 illustrates the average iteration numbers on the GSM8K and MATH datasets using the DeepSeek model.
18

---

Published as a conference paper at COLM 2024
Dataset
Model
Method
Avg.
Date
CSQA
StrategyQA
AuQA
SVAMP
GSM8K
ADPSC
4.31
4.21
4.43
5.13
4.27
4.42
4.46
RCI
2.39
2.90
2.57
3.67
2.56
2.35
2.74
CPT-3.5
LECo+CoT
2.16
2.08
2.18
2.16
2.14
LeCo-Complex
2.11
2.08
2.17
2.43
2.24
2.29
2.22
ADPSC
4.28
4.32
4.56
5.44
4.39
4.21
4.53
RCI
2.08
2.31
2.47
2.9
3.21
2.25
2.54
GPT-4
LECO+COT
2.00
2.02
2.05
2.08
2.05
2.05
2.04
LeCo+Complex
2.01
2.05
2.08
2.24
2.13
2.08
2.10
Table 11: Average iterations on diverse datasets with OpenAl models.
Dataset
Model
Method
Avg.
Algebra
Count
Geometry
Iter
Num
Prealgebra
Precaculus
ADPSC
5.36
5.92
6.21
5.84
6.76
5.59
6.36
6.01
RCI
2.59
2.83
3.00
2.75
2.97
2.58
2.78
2.79
GPT-3.5
LeCo-Complex
2.52
2.83
2.81
2.91
2.78
2.42
2.94
2.74
ADPSC
6.44
7.22
5.91
7.70
8.63
5.03
8.38
7.04
RCI
3.31
3.41
3.51
3.41
3.43
3.27
3.29
3.38
GPT4
L.Co+Complex
2.47
2.75
2.9
2.79
2.63
2.31
2.81
2.66
Table 12: Average iterations on MATH dataset with OpenAl models.
MATH
Models
Methods
GSM8K
Avg.
Algebra
Count
Geometry
Iter
Num
Prealgebra
Precaculus
DeepSeek
LECo+Complex
2,25
2.22
2.44
2.46
2.52
2.45
2.25
2.59
2.40
Table 13: Average iterations on MATH and GSM8K catasetss with DeepSeek model.
Dataset
Models
Methods
GSM8K
Strategy
Early Stop
8.0M/367.66 /36.66K
1.7M/ 132.7K
gpt-3.5-turbo-0613
LeCo
8.2M/393.80
1.9M/181.9K
Farly Stop
7.0M/315.7K
1.7M/162.3K
gpt-4
LeCo
7.4M/334.2K
1.8M/167.3K
Table 14: Average Token Consumption on GSM8K and StrategyQA of Early-stop LECO
B Details of Early Stop LECO
B.1 Algorithm of Early stop LECO
As presented in Algorithm 20 firstly, we sample the entire dataset according to certain proportion, obtaining distributions of correct and incorrect solutions. Leveraging the normal distribution traits of incorrect responses, For the remaining data, if its solution score surpasses the threshold, we accept this answer outright; otherwise, resort to the standard LECO method for reconsideration.
utilize the positive 1-77 value as the threshold.
B.2 Token Consumption and Iteration Number of Early Stop LECO
Table 14 and 15 presents the average token consumptions and average iteration numbers on the GSM8K and StrategyQA datasets using OpenAl models via early-stop LECO.
19

---

Published as a conference paper at COLM 2024
Algorithm 2 Early Stop of LECO
Require: input questions x, model M, demonstration Demox, standard LECO(*), sample amount R, solution score sin score("), normalize function norm(*)
1: sample correct set C Ø, sample incorrect set E = > Initialize sample score set Sample Stage
2: for X3 E O,..., R do
3: Vt, = LECo(xs, M, Demo;)
The subscript, represents the sampling stage
4: if yt, correct then
5:
C + sU
6:
else
E - EU sln score(y:.) 8: end if
7:
9: end for
10: jLincorrect, o_incorrec = norm
11: threshold f = .incorrect + o_incorrect
12: for X ++ 1,... do
> Early Stop Stage > The subscript us represents the remaining part.
13:
y0,zz M (Xns, Demox)
14:
if sln ssllsscoore score > then
15:
you y0as
16:
else
Ytu LECO(XOmar M, Demox, Vo..)
18:
end if
end for
20:
return Y/1
Dataset
Models
Methods
GSM8K
Strategy
Early Stop
2.16
2.11
gpt-3.5-turbo-0613
LeCo
2.39
2.17
Early Stop
2.03
2.06
gpt-4
LeCo
2.08
2.08
Table 15: Average Iterations GSM8K and StrategyQA of Early-stop LECO
K
3
5
Complex
81.8
80.89
83
LeCo + Complex
82.83
82.33
83.87
(+ (+1033)
(+1.44)
(+0.87)
Table 16: Settings of Hyperparameter K
0.1
0.2
0.3
0.4
0.5
Complex
81.16
80.98
80.89
82.46
83.03
LeCo+Complex
82.46
82.24
82.33
83.88
83.84
(+1.3)
(+1.26)
(+1.44)
(+1.42)
(+0.81)
Table 17: Settings Hyperparameter
C Hyperparameter Settings
We compared the experimental results under different settings and found that our method is relatively insensitive to hyperparameters, such K and T. We attach the experimental results of GPT-3.5 on GSMSK as follows.
Table 16 and Table 17 present the settings of hyperparameter K and T.
20

---

Published as a conference paper at COLM 2024
1.0
0.9
Inter-step transition score
0.8
0.7
0.6
0.4
0.8
1.0
Overall Confidence
Figure 4: The relation between overall confidence and inter-step transition scores
In the design of the transition score, the parameter K determines the usage of several initial tokens, hence the value of K can not be very large and we set K varying from 1 to
In the design of the divergence score, the parameter is used to rescale the KL divergence to a reasonable range and helps the divergence score to show significant performance. When exceeds 0.5 in the logarithmic function, the divergence diminishes to negligible values, such as 0.002 or 0.004, which fail to capture the desired differences. Consequently, our study focuses the impact of within the range of 0.1 0.5.
The results, as depicted in the tables, reveal consistent improvement, indicating the robustness of method to these parameter.
D Preliminary Experiments
We draw the scatter plot of the relationship between the overall confidence score and inter- step transition score for 1000 reasoning steps. As shown in Fig4 it's obvious that the overall confidence and inter-step transition scores are highly positively correlated.
E
Case Study of LECO
Table 18 19 and 20 lists the specific cases of reasoning results via different methods on GSM8K, Strategy QA, and MATH datasets.
21

---

Published as a conference paper at COLM 2024
shoe was having weekend sale on a brand of popular tennis shoes. On Friday the store sold 14 pairs of tennis shoes.
The next day they sold double that number shoes. On the last day of the sale they sold one-half the amount that they did the day before. but six people returned their pairs because they didn't fit.
Question
How many tennis shoes were sold by end sale?
Gold Answer
50.0
Let's think step by step
Step On Friday, the store sold 14 pains of tennis shoes.
Step 2:
The next day, they sold double that number, which is 14 2 28 pairs tennis shoes.
CoT
Step On
day, they sold ome-half amount of the day before, Step 4: But 6 pairs were returned, so the total number of tennis shoes sold is 14 14-6 6 6 22 paits.
/ /22 pairs
Step The onswer is \boxed (22).
Let's think step by step
Step On Friday, the store sold 14 paizs of tennis shoes.
Step 2: next day, they sold double number, so they sold x 28 pairs of tennis shoes.
Complex
Step 3: On the last day, they sold one-half the amount they did the day before, they sold 28 2 14 pairs of tennis shoes.
Step However, people returned their pairs, the total number of pairs sold 14 14-6 pairs of tennis shoes.
Step 3: Since each pair of tennis shoes represents one unit, the store sold 22 tennis shoes by the end the sale.
Step 6: The answer is (boxed(22)
The provious answer incorrect because it only calculates the number pairs of tennis shoes sold, not the total number of tennis shoes.
To find total number of tennis shoes, need multiply the number of pairs by
Let's correct the answer:
Step I: On Friday, the store sold 14 pains of tennis shoes.
RCI
Step next day, they sold double number, so they sold
pairs of tennis shoes.
Step 3: On the last day, they sold one one-hall half the amount they did the day before, so they sold 28 - 14 pairs tennis shoes.
Step However, people returned their pairs, the total number of pairs sold 14 + 14- 6 pairs of tennis shoes.
Step 3: Since each pair of tennis shoes represents two units (one for the left foot and one for the right foot).
the store sold 22 2 44 tennis shoes by the end of the sale. The is (44).
Let's think step by step
Step On Friday, the store sold 14 pairs of tennis shoes.
Step 2: On the next day, they sold double that number, so they sold 14 2 28 pairs of tennis shoes.
ADPSC
Step 3: On the last day, they sold half the amount they did the day before, they sold 28 14 pairs of tennis shoes. lotal number of pairs sold
Step However, people returned their pairs,
pairs of tennis shoes.
Step 3: Since each pair consists of two shoes, the total number tennis shoes sold is 22 2 tennis shoes.
Step The answer is 44.
Let's think step by slep
Step I: On Friday, the store sold 14 pains of tennis shoes.
Slep next day, they sold double that number, so they sold 28 pairs of tennis shoes.
SC
Step 3: On the last day, they sold one half the amount as the day before, so they sold 28 / 14 pairs of tennis shoes.
Step However, people returned their pairs, the total number pairs sold is 14 6 22 pairs tennis shoes.
Step Since each pair of tennis shoes consists of shoes, total number of tennis shoes sold 22 x 44 tennis shoes.
Step 6: The answer is 44.
Let's think step by step
Step 1: On Friday, the store sold 14 pairs of tennis shoes.
Step On Saturday, they sold double that number, 14 28 pairs tennis shoes.
COT+LeCo
Step 3: On last day, they sold one-half the amount of the day before, 28 14 pairs tennis shoes. Step 4: But 6 pairs were returned, so 14-6 6 pairs of tennis shoes were sold on the last day.
Step In total, they sold 28 8 50 pairs tennis shoes.
Step 6: The answer is (50))
Let's think step by step
Step On Friday, store 14 pains tennis shoes.
Step 2: The next day, they sold double that number, so they sold 14 x = 28 pairs of tennis shoes.
ComplexiLeCo
p3: On the last day, they sold one-half the amount they did the day before, they sold 29 14 pairs tennis shoes.
Step 4: However, six people retured their pairs, so the total number of tennis shoes sold is 14 29 + 14 6 50 pairs of tennis shors.
Step The answer is (boved (50) pairs of tennis shoes.
Table 18: Case Study of LECO on GSM8K by GPT-3.5-Turbo
22

---

Published as a conference paper at COLM 2024
Question
Could JPMorgan Chase give every American $10?
Answer
Let's think step by slep.
Step JPMMrgan
Step 2: Banks have lot of money.
CoT
Step There approximately 350 million Americans.
Step 10maliiplinllyy3SSiiiiiiiis billion.
Step S: [PMorgan Chase is very large bank, but it may not have enough money to give $10 to every American.
Step Thus, JPMoran Chase may able give every American Step So the answer (no).
Let's think step by step. Step 1: JPMorgan Chase is multinational investment bank and financial services company. Step [PMorgan Chose has market capitalization of $400 billion.
Step 3: The population
States over million.
Complex
Step 4: To give every American $10, JPMorgan Chase would need to distribute over $3.3 billion.
Step JPMorgan Chase's market capitalization significantly smaller than the amount needed give every American $10.
Step Thus, JPMorgan Chase could not give every American S10.
Step 7: So the answer (boved(no)
apologize for the mistake in my previous answer.
Upon reviewing the information, is clear that JPMorgan Chase would not be able to give every American $10. required to give
RCI
each approximately 331 million would which is significant amount even for large financial institution like JPMorgan Chase.
Therefore, the correct answer is (noo).
Let's think step by step.
Step JPMorgan Chose is multinational investment bank and financial services company.
United States approximately Step 3: JPMorgan Chase were to give every American $10, they would need to distribute total of $3.31 billion.
Step current population
people.
ADPSC
Step JPMorgan Chase's total assets of 2020 $3.42 trillion.
Step billion a very small
of JIMMrgan
assets.
Step 6: Thus, JPMorgan Chase could give every American $10.
Step So the
(boxed
Let's think step by slep.
Step JPMorgan Chase multinational investment bank and financial services company.
Step 2: current population United States is approximately 331 million people.
Step 3: JPMMrrga Chase were give every American $10, they would need to distribute total of $3.31 billion.
Step JPMorgan Chase's tolal
2020 Step 3: $3.31 billion is very small fraction of JPMMrgaan Chase's total assets.
trillion.
Step Thus, JPMorgan Chase could give every American $10.
Step
answer booodd
Let's think step by step
Step JPMorgan Chase bank.
COT+LeCo
Step 2:
Banks have lot money.
Step 3: Thus, JPMorgan Chase could potentially give every American S10.
Step
answer (yess
Let's think step by step
Step JPMorgan
multinational
financial company.
Step 2: JPMorgan Chase one the largest banks in the United States
Step The population the United States is 300 million people.
Complex+LeCo
to distribute over Step 5: JPMorgan Chese's total assets are over $3 trillion..
Step To every
every American $10, JPMorgan Chase
Step Thus, JPMorgan Chase could give every American $10.
Step 7: So the answer is (yesed(yes)
Table 19: Case Study of LECO on StrategyQA by GPT-3.5-Turbo
23

---

Published as a conference paper at COLM 2024
Five balls are numbered with the integers through 5 and placed in jar.
Three are drawn without replacement.
Question
the probability
integers on the ballss odd?
Express your answer as a common fraction.
Gold Answer
Let's think step by step
Step There
consider: either all three integers odd, exactly two of the integers odd.
Step 2: Case all three integers are odd, there are () ways to choose the odd integers.
Step 3: There are ( 10 total ways to choose three integers from the five.
Complex
Step
exactly of integers are odd,
ways
odd integers ( ways to even integer.
Step 3: There are ( m total ways to choose three integers from the five.
Step So, probability that the sum of the three integers is odd 1: 4if boxedd
Step answer (boxed ft
A: I et's think stop by step
Step
of (5) 10 ways
balls out
Step 2: We need to count the number ways to choose 3 halls such that the sum of their integers is odd.
Step The sum of three integers will be edd and only if there odd number odd integers among the three chosen balls.
Step
are (+) ways
odd integer and even integers.
RCI
Step S: There are ()) = ways to choose 3 odd integers.
Step There () ways choose 2 odd integers and integer.
Step 7: So, total number of ways choose balls such that the sum of their integers is odd
Step Therefore, the probubility 17 boxel()).
Step 9: answer \boxed
Let's think step by
Step are
consider: either all three integers odd, Step 2: Case all three integers are odd, there are () ways to choose the odd integers.
odd and even.
Step There are 5 odd integers choose from, the probability in this case ( ib-
ADPSC
Step Case
integers odd and
there are ( (3)-)) 3-2 6 choose the integers.
Step are odd integers even integers to from, so the prohability case
Step The total probability is
Step The answer is \boxed
Let's think step by slep
Step There are two cases to consider: either all three integers are odd, two are odd and one even.
Slep Case all three integers are odd, there
ways choose odd inlegers,
Step 3: There are odd integers to choose from, so the probubility in this case (5) tb-
SC
Step 4: Case two integers are odd and one is even, there are (2) ( 3 2 6 ways to choose the integers.
Step There are 3 odd integers and integers to choose from, the probabiltyy this case 10
Step The total probability is
Step 7: The answer is \boxed (
Let's think step step
Step There are two cases to consider: the sum either three numbers are or two numbers
one number even.
Step 2: Case I: All three numbers are odd. There are odd numbers (1, 5) and 2 even numbers (2, 4) choose from.
Step probability choosing odd numbers
Complex+LeCo
Step 4: Case Two numbers are odd and one number even. There are odd numbers and even numbers to choose from.
Step 3: The probability choosing odd numbers and 1 even number
m
Step total probability + &t boxed(()
Step 7: The answer is \boxed $
Table 20: Case Study LECO on the MATH dataset using GPT-3.5-Turbo.
24


Reviews:
[{"review_id": "aYqZ8sW0FW", "reviewer": "Reviewer_SLMi", "summary": "The paper proposes to improve multi-step reasoning capability of LLM by an intrinsic self-correct framework LECO. Specifically, the proposed self-correct method introduces a step confidence method to check the correctness of each step and append the correct steps to the input so as to enhance the context for multi-step reasoning. To calculate the confidence of each reasoning step, the paper uniformly combines three kind of metrics, that is average token score, step divergence score and inter-step transition score. \nThe experimental results reported positive improvement by exploiting LECO with COT/Complex on strong LLMs such as GPT-3.5, GPT-4 and DeepSeek. And the ablation study showed the impact of the three metrics on step confidence by correctness. I have suggestion to list the performance of fine-grained ablation study as those of Table 2. \nGenerally, the paper is well organized and written, easy to follow. \nSome suggestions for more experimental results on:\n1. the choice of tau in Equation (2)  or why 0.3 is the choice\n2. the impact of K in Equation (3) or why 3 is choosen", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive suggestions, which help us to improve the quality of our work, and are pleased that you find our work to be novel and effective.\n\n---\n\n**R1A1**: The choice of hyperparameters\n\n**Response**: We compared the experimental results under different settings and found that our method is relatively insensitive to hyperparameters, such as *K* and *tau*. We attach the experimental results of GPT-3.5 on GSM8K as follows.\n\n* For *K* :\n  \n| K            | 1       | 3       | 5    |\n| -------------- | --------- | --------- | --------- |\n| Complex      | 81.80   | 80.89   | 83.00   |\n| LECO+Complex | 82.83   | 82.33   | 83.97   |\n|             | (+1.03) | (+1.44) | (+0.97) |\n  \n  \n* For *tau* :\n  \n| tau          | 0.1    | 0.2     | 0.3     | 0.4     | 0.5     |\n| -------------- | -------- | --------- | --------- | --------- | --------- |\n| Complex      | 81.16  | 80.98   | 80.89   | 82.86   | 83.03   |\n| LECO+Complex | 82.46  | 82.24   | 82.33   | 83.88   | 83.84   |\n|             | (+1.3) | (+1.26) | (+1.44) | (+1.42) | (+0.81) |\n  \n  \n\nIn the design of the divergence score, the parameter *tau* is used to rescale the KL divergence to a reasonable range and helps the divergence score to show significant performance. When *tau* exceeds 0.5 in the logarithmic function, the divergence diminishes to negligible values, such as 0.002 or 0.004, which fail to capture the desired differences. Consequently, our study focuses on the impact of tau within the range of 0.1 to 0.5. The results, as depicted in the table, reveal a consistent improvement, indicating the robustness of our method to this parameter.\n\n---\n\nWe appreciate the reviewer's insightful feedback and will incorporate this discussion into the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. propose a simple and effective self-correct method to enhance the context for multi-step reasoning by introducing step confidence to check the correctness of each step and append the correct steps to the input", "reasons_to_reject": "1. some experiments supporting the choice of hyper-parameters should be included\n(1) the choice of tau in Equation (2)  or why 0.3 is the choice\n(2) the impact of K in Equation (3) or why 3 is chosen", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qiGL1sdZMM", "reviewer": "Reviewer_2kVG", "summary": "This paper explores the area of aligning language models but without using human feedback (RLHF), external tools handcrafted prompts (SFT). They show that the resulting model improves reasoning performance. The main idea is learning from correct reasoning steps and based on generation logits and estimation of confidence.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you put into reviewing our paper. We appreciate the insightful feedback and address your concerns as follows:\n\n---\n\n**R2A1** Inappropriate and missing citations\n\n**Response:**  Thanks for pointing out the potential problems regarding our reference. We would like to clarify that reference [1] was mentioned in the first paragraph of the Related Works section. We will include reference [2] and other original works in our revised version.\n\nReference [3] hypothesizes that a model's familiarity with a prompt's language predicts its effectiveness, demonstrating that lower perplexity prompts yield better performance. The authors also develop a method to expand a small set of manually written prompts through paraphrasing and back-translation, confirming perplexity as a strong predictor of prompt success. In contrast, our work reduces reliance on prompts by calculating confidence using logits.\n\nReference [4] seeks to enhance the reasoning capabilities of LLMs by exploring different chains of thought and validating individual reasoning steps based on Relevance, Mathematical Accuracy, and Logical Consistency. Constraints are implemented through model verifiers and perplexity checks to ensure high-quality solutions. However, this approach still relies on prompts for verification, similar to our cited works.\n\nIn our revised version, we will integrate the discussed research efforts into the corresponding sections.\n\nWe sincerely thank the reviewer for their insightful feedback again!\n\n---\n\nReferences:\n\n[1] Training language models to follow instructions with human feedback. NeurIPS 2022. \n\n[2] Language Models are Few-Shot Learners. NeurIPS 2020.\n\n[3] Demystifying Prompts in Language Models via Perplexity Estimation. EMNLP 2023, Findings. \n\n[4] General Purpose Verification for Chain of Thought Prompting. ArXiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and easy to follow.\n\n- Their method (even if simple) and that does not require any external knowledge, helps to improve the reasoning capabilities of the LLMs tested in multiple tasks and benchmarks. The improvements are more prevalent in the MATH dataset.\n\n- I think this paper will be of interest to the research community at the conference.", "reasons_to_reject": "- The choice of citations is rather odd. For example, if I would have cite a paper on learning from human feedback, I would probably cite https://arxiv.org/abs/2203.02155 and if I have to cite the precursor of language models as we know them today, I would probably cite https://arxiv.org/abs/2005.14165. The authors, however, cite only 2023 papers thus ignoring the actual original works on the topic they refer to.", "questions_to_authors": "- I believe this paper may merit a mention: https://aclanthology.org/2023.findings-emnlp.679.pdf  - the authors of the reviewed paper calculate the confidence via logits  by averaging the token probabilities within a given step + divergence step, while in this paper link I provide they use ppl as a way to correlate with downstream task performance.\n\n- Note this recent paper, very related https://arxiv.org/pdf/2405.00204 (I believe it was posted to arxiv after the CoLM submission date, but I post it here for their awareness.", "extra_scores": {}}, {"review_id": "vf5pO0XlBI", "reviewer": "Reviewer_Me19", "summary": "This paper aims to address some limitations in LLMs, such as hallucination, unfaithful reasoning, and toxic content. In particular, the authors proposed an intrinsic self-correct reasoning framework for LLMs that eliminates the need for human feedback, external tools, and handcraft prompts. The new framework follows a multi-step reasoning paradigm, learning from correctness.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback. We will carefully address your concerns in the following points.\n\n---\n\n**R3A1**  Eq(4):\n\nThe *avg_score* reflects the general quality of a step, *trans_score* measures consistency between steps. While the *diver_score* assesses the importance of specific tokens within a step, as detailed in our Methodology. High average and transition scores indicate high-quality steps with continuous logical reasoning. Conversely, a high divergence score suggests non-uniform token distribution within a step, indicating that key tokens, such as those in mathematical calculations, may have low probabilities. Therefore, we aim to keep the divergence score low, assigning it a coefficient of -1 to minimize its impact.\n\nTowards the weights of these scores, we did not conduct the grid-search and simply set them to 1 to avoid introducing too many hyperparameters. Based on this vanilla setting, consistent improvements are also observed in our experiments. Therefore, we believe further benefits can be achieved with an elaborated hyperparameter search.\n\n---\n\n**R3A2** Improvements:\n\nWe acknowledge that the improvements against SC are modest on less complex tasks like commonsense and arithmetic reasoning, but more substantial on challenging datasets such as MATH. This is explained in our manuscript: \n>“ the difficulty of the task correlates positively with the impact of LeCo… The primary reason for this is that the LLM tends to remain their initial reasoning path on the easy problems, offering fewer improvement rooms for LeCo.”\n\nLeCo's primary advantage over SC is its **80%** reduction in token consumption while achieving better performance. Our main contribution lies not in attaining SOTA performance, but in proposing a novel paradigm for leveraging feedback—learning from correctness—to enhance reasoning capabilities while reducing computational costs.\n\n---\n\n**R3A3** Costs:  \n\nDue to the limited space, we replace this comparison in **Appendix A.1**. We also provide a brief discussion about the computation cost in the **last paragraph of “Main Results”**.\n\n|      | CSQA       | GSM8K      | \n| ------- | ------------ | ------------ | \n| SC    | 14.4M/8.3M | 36.3M/1.6M | \n| LeCo  | **3M/151K**   | **8.2M/394K**  | \n\nWe can see that LeCo is the **most token-efficient** method among these algorithms, saving about 80% of tokens compared to SC.\n\n---\n\nWe sincerely thank the reviewer again for the valuable suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper studies an important problem, i.e., improving the reliability and reasoning capability of LLMs. Instead of learning from errors as in existing work, this paper presents a new framework by learning from correctness. The idea is well motivated.\n\n2. Overall, the paper is well organized and clearly written. \n\n3. Extensive results on benchmarks are reported. The proposed LECO framework helps improve the performance in most cases. Ablation studies are provided, and case studies are also presented in the appendix.", "reasons_to_reject": "1. The confidence score proposed in Eq. (4) is quite heuristic and thus it requires more justifications. \n\n2. Comparing the proposed LECO framework with baselines (such as SC), the improvements are marginal in many cases, according to the results reported in Table 1 and Table 2. \n\n3. The computational costs of LECO and baselines are not discussed and compared in the experiments.", "questions_to_authors": "----Post Rebuttal----\nSome of my previous concerns have been addressed, and thus I would like increase my score.", "extra_scores": {}}, {"review_id": "rKP4uph1vE", "reviewer": "Reviewer_cvVx", "summary": "This paper introduces a reasoning paradigm called Learning from Correctness (LECO), aiming to enhance the performance of Large Language Models (LLMs) in reasoning tasks. The LECO framework employs a multi-step reasoning paradigm, first computing confidence scores for each reasoning step, which include considerations such as average token scores, step divergence scores, and step transition scores. Subsequently, LECO identifies the step with the lowest confidence as a potential error step and treats all preceding steps as \"correct.\" Then, LECO incorporates these \"correct\" steps as part of the input, refining the search space through iterative reasoning processes until obtaining the final answer or reaching a stopping condition. Experimental results demonstrate that this approach not only reduces reliance on external interventions but also enhances the model's performance in multi-step reasoning tasks while reducing token consumption.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer and address your concerns as follows:\n\n---\n\n**R4A1:**  Comparison with other works\n\nWe would like to clarify that the paper mentioned by the reviewer (DUP) was posted to arxiv after the CoLM submission date, and API versions of GPT may differ. Additionally, there are also key differences between DUP and LeCo. 1) DUP needs elaborate prompts while LeCo eliminates the need for external information; 2) DUP is hard to be used with open-source models.\nMoreover, as stated in R3A2, our goal is not to achieve new SOTA performance.\n\n**R4A2:** Confidence\n\nPrevious works [1,2] also use confidence as the judgment on reasoning tasks. Our experimental results show that 65% of incorrect reasoning steps are the steps with the lowest confidence.\n\n**R4A3:** Step divergence\n\n1. The step divergence score should be evaluated within the overall design of LeCo, as its standalone effect, focusing solely on within-step distribution, is less impactful. \n2. We tested various distributions( Laplace, Rayleigh distribution...), and the normal distribution best matched the data patterns, leading to its selection. \n3. Step divergence evaluates the uniformity within a step rather than individual tokens.  The divergence score for step [.85, .83, .86, .84] is lower than that for [.95, .63, .86, .94]\n\n**R4A4:** Preliminary findings\n\n1. We draw the scatter plot of the relationship between the overall confidence score and transition confidence score, and find they are positively correlated. We will attach the figure to the appendix. \n2. It is concluded through observations.\n\n**R4A5:** Model capacity\n\n1. LeCo prefers stronger LLMs while it also can be used on less-powerful models.\n2. 100 sampled solutions reveal that LeCo identified the first error step 62% of the time with DeepSeek and 65% with GPT-3.5. DeepSeek's slightly lower accuracy and consistent replies, often identical for 'demonstration' and 'demonstration + several correct steps' inputs, may explain its less significant improvement.\n\n**R4A6:** Typo and presentation\n\nThanks for pointing out the typo and presentation problems. We will revise them accordingly in the revised version.\n\n---\n\nReferences:\n\n[1] Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models ArXiv 2023\n\n[2] Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models. ArXiv 2024\n\n[3] Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article differs from previous approaches like learning from errors, as it adopts a method of identifying correct steps in the reasoning process and progressively obtaining the correct answer through these steps. Importantly, it does not require training and is applicable to black-box models that are not publicly disclosed.", "reasons_to_reject": "1.\thttps://arxiv.org/pdf/2404.14963 This research has significantly improved the performance of GPT-4 in GSM8K, achieving up to a 97% enhancement compared to prior work.\n2.\tThe experimental design in the paper is not sufficiently comprehensive. For instance, it lacks adequate baseline tests and comparisons with state-of-the-art methods. Additionally, the statistical significance of the experimental results is insufficient.\n3.\tIs it reasonable to assume that using the level of confidence as a judgment of the correctness of the reasoning process is valid? If a model generates illusions, errors, or unfaithfulness, does its confidence necessarily remain low? Please provide some relevant theoretical explanations.", "questions_to_authors": "1. In Algorithm 1, when the iteration variable t ranges from 0 to t, what does x_(t-1) represent when t=0?\n2. In step 7 of Algorithm 1, what does the procedure mean? Does it involve concatenating the two strings? Further elaboration on this seems needed in this paper.\n3. Figure 3 appears to lack clarity.\n4. It seems that the step divergence score is not effectively functioning across the three confidence scores as indicated in Table 5. Could you provide a more in-depth analysis of this observation? Moreover, could you elucidate why the distribution is expected to approximate a uniform distribution? Additionally, while I understand that the step divergence score may to some extent reflect the distribution of token probabilities, how does it specifically capture higher token probabilities as opposed to lower ones? (the higher the token probabilities, the bigger the step divergence score?)\n5. In the discussion of the inter-step transition score, you mentioned two key insights from your pre-experimentation phase. However, I apologize for not noticing where your preliminary experiment was referenced. It's possible that I missed it, but could you please indicate its location in the paper?\n6. Which models are the green and red markers in the Table compared against? Please mention this in the caption.\n7. I observed that the improvement on DeepseekMath-7B is not as significant compared to the enhancement seen in the GPT series. Does this suggest that your approach requires a higher standard of model performance itself? Given that your methodology entails the model's capability to generate the correct steps as assumed, would it not imply that, with lower model performance, the selection process might involve choosing steps that are less incorrect rather than truly correct?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is an interesting paper that proposes a simple heuristic for progressively improving the accuracy of reasoning step sequences with an LLM.  The technique requires access to the per-token logits, but is otherwise generic, and appears to give useful improvements across datasets and models.  A careful ablation appears to validate the three components of the proposed heuristic score, which otherwise is intuitive but not rigorously justified.

The reviewers gave many helpful suggestions, including relevant references to add.  Moreover, they provide good suggestions to clarify the algorithm exposition, address computational overhead, address hyperparameter sensitivity, and strengthen the justification of the proposed heuristic.  It is expected that the authors will address these in the final version of their paper.
Decision: Accept