Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models
Authors: Chengyu Du
Jinyi Han
Yizhou Ying
Aili Chen
Qianyu He
Haokun Zhao
Haoran Guo
Sirui Xia
Jiaqing Liang
Zulong Chen
Liangyue Li
Yanghua Xiao
Venue: ICLR 2025 Poster (2025)
Keywords: progressive thought refinement
self refine
large language model
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Progressive Thought Refinement, a framework that activates LLMs' internal refinement abilities Our method improves generalization across tasks.

Abstract: Recent advancements in large language models (LLMs) have demonstrated that progressive refinement, 
  rather than providing a single answer, results in more accurate and thoughtful outputs. However, existing methods often rely heavily on supervision signals to evaluate previous responses, making it difficult to effectively assess output quality in more open-ended scenarios. Additionally, these methods are typically designed for specific tasks, which limits their generalization to new domains. To address these limitations, we propose Progressive Thought Refinement (PTR), a framework that enables LLMs to progressively refine their responses. PTR operates in two phases:  (1) Thought data construction stage: We propose a weak and strong model collaborative selection strategy to build a high-quality progressive refinement dataset to ensure logical consistency from thought to answers, and the answers are gradually refined in each round. (2) Thought-Mask Fine-Tuning Phase: We design a training structure to mask the "thought" and adjust loss weights to encourage LLMs to refine prior thought, teaching them to implicitly understand "how to improve" rather than "what is correct." Experimental results show that PTR significantly enhances LLM performance across ten diverse tasks (avg. from 49.6% to 53.5%) without task-specific fine-tuning. Notably, in more open-ended tasks, LLMs also demonstrate substantial improvements in the quality of responses beyond mere accuracy, suggesting that PTR truly teaches LLMs to self-improve over time. Our work is now open-source.  https://github.com/cydu24/Progressive-Thought-Refinement

Published as a conference paper at ICLR 2025
THINK THRICE BEFORE YoU ACT: PROGRESSIVE THOUGHT REFINEMENT IN LARGE LANGUAGE MOD- ELS
Chengyu Du JJnyi Han", Yizhou Ying', Aili Chen Qianyu He',
Haokun Zhao
Sirui Xia 14 Haoran Guo", JJiaiin Liang", Zulong Chen', Liangyue Li", Yanghua Xiao
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University
? Shanghai Institute of Artificial Intelligence for Education, East China Normal University
"School Science, Fudan University "Alibaba Group 'RhineAI
(cydu24, yzying24, alchen24, qyhe21, hkzhao23, srxia24 (.m. fudan. cn. [liangjiaqin, shawyh) 8fudan. edu, en, ijnyyhann99, rhineai lab) @ggma com, [zul (zuleng. czl, liangyue. 1 @a1 ibbabaiinc
ABSTRACT
Recent advancements in large language models (LLMs) have demonstrated that progressive refinement, rather than providing a single answer, results in more accurate and thoughtful outputs. However, existing methods often rely heavily on supervision signals to evaluate previous responses, making it difficult to effectively assess output quality in more open-ended scenarios. Additionally, these methods are typically designed for specific tasks, which limits their generalization to new domains. To address these limitations, we propose Progressive Thought Refinement (PTR), a framework that enables LLMs to progressively refine their responses. PTR operates in two phases: (1) Thought data construction stage: We propose weak and strong model collaborative selection strategy to build a high-quality progressive refinement dataset to ensure logical consistency from thought to answers, and the Phase: We design a training structure to mask the "thought" and adjust loss weights to encourage LLMs to refine prior thought, teaching them implicitly understand "how to improve" rather than "what is correct." Experimental results show that PTR significantly enhances LLM performance across ten diverse tasks (avg. from 49.6% to 53.5%) without task-specific fine-tuning. Notably, in more open-ended tasks, LLMs also demonstrate substantial improvements in the quality responses beyond mere accuracy, suggesting that PTR truly teaches LLMs to self- impfoee over time. Our work is now open-source.
answers gradually refined in each round. (2) Thought-Mast
INTRODUCTION
"Think thrice before you act."
Confucius
Recent advancements in large language models (LLMs) have highlighted that progressive refinement is more important than simply providing a single answer (Yang et al., 2023b; Madaan et al., 2023b). Humans often rely on a combination of two thinking systems to solve problems, known as System and System 2 (Kahneman, 201 System 1 facilitates quick, intuitive responses but often lacks the depth required to handle complex reasoning tasks. In contrast, System 2 engages in progressive refine- ment, gradually improving a solution by starting with rough approximate thought and iteratively adding detail and accuracy. Recent work, such as GPT-ol (OpenAI, 2024), demonstrates that LLMs perform better by adopting progressive thought refinement. This approach leads more accurate and thoughtfully considered outcomes, similar to how the human brain processes complex tasks.
"Corresponding authors.
hhtpps://o /i thub. com wttp.tcomccc//////////////// /ww.:///yydd////rrsssssssi www.rrrgresssseeooooott------eee wwwohhhtgthtto Refi nement

---

Published as a conference paper at ICLR 2025
Progressive refinement ability is imperative for LLMs because it significantly enhances the quality of responses by gradually improving accuracy and depth. Previous methods heavily rely on supervision signals, such correctness assessments, to assess response quality. For example, labeled datasets with feedback are used to fine-tune models as verifiers (Han et al., 2024; Havrilla et al., 2024; Welleck et al., 2023), facilitating self-assessment and iterative improvement. Additionally, Reinforcement Learning (RL) reward functions are also employed to guide models toward generating better answers (Chen al., 2024; Yuan et al., 2024; Rosset al., 2024a; Akyurek al., 2023). However, evaluating answers based on supervision signals has limitations, as annotators often struggle to provide accurate labels without clear, comprehensive criteria. This is particularly challenging in open-ended tasks, such as text generation and summarizing, where the distinction between "correct" and "incorrect" is blurred, making it difficult to define and evaluate response quality.
Due to significant variations in supervision signals and evaluation criteria across tasks, previous self-imprrvemmen approaches have primarily aimed enhance accuracy within specific domains. Examples include enabling LLMs to self-debug for improved code generation (Chen et al., 2023; Tony et al., 2024; Liang et al., 2023) and solving math problems through progressive step validation (Wang et al., 2023a; Lightmmn et al., 2023; Uesato al., 2022a). These methods often rely task- specific pipelines or reward models, making generalization difficult. The key limitation is that errors addressed in one domain may not apply other tasks, since different tasks exhibit varying error types. Consequently, transferring these approaches to new tasks often fails (Tian et al., 2024). and models trained with these methods have limited generalization capabilities, struggling to improve performance beyond their training domains.
To address these challenges, we introduce PTR (Progressive Thought Refinement), framework specifically designed to stimulate the model's intrinsic refinement ability. Our PTR method comprises a progressive refinement dataset construction phase and weighted thought- mask fine-tuning phase. During the progressive refinement dataset construction phase, we obtain queries from open- ppen-domain datasets and employ a weak-strong model collaborative selection strategy to construct high-quality thoughts and refined answers dataset.
This strategy not only ensures improvement from thoughts to answers but also eliminates the need for accurate labels. In the fine-tuning phase, we employ weighted thought-mask fine-tuning to teach LLMs to implicitly understand "how to improve" rather than supervising them with "what is correct". Specifically,
reformulate the masked data structure and redesign the loss of weighting to encourage LLMs to improve responses based on previous thoughts and ensuring logical consistency between the thought process and the final answer.
Our experimental results show that LLMs trained with PTR can improve the quality of their previous answers across ten tasks, including knowledge reasoning, code generation, mathematical reasoning, comprehension, summarizing, and text generation. The average performance across these tasks improved from 49.6% to 53.5%, with significant improvement on the MMLU task, where accu- racy increased from 57.1% 1% to 64.1% 1% for Qwen2-8B. Notably, these improvements occur without task-specific fine-tuning, demonstrating that our method activates the model to learn progressive refinement from the PTR dataset. Moreover, in more open- open-ended tasks, LLMs have also demonstrated further improvements in answer quality and formatting beyond correctness.
Our contributions are threefold:
We propose the PTR method to stimulate models' progressive refinement abilities and enhance generalization across various tasks without additional task task-specific fine-tuning.
We design efficient weak-strong model collaborative selection strategy to construct high-quality PTR datasets without extra feedback.
We introduce a novel weighted thought fine-tunint method to instill general progressive refinement capabilities in LLMs.
2 RELATED WORK
Progressive Refinement with External Feedback Existing work often relies external tools or stronger LLMs to provide feedback for refinement. For example, external tools are used to critique and provide feedback the primary model's responses (Yang al., 2023a; Chen et al.,
2

---

Published as a conference paper at ICLR 2025
(A) Progressive Thought Refinement Dataset
(8) Weiphted Mhoupht-Mask Fine-tuning
Query Proparation
Anought-Answer Preparation
Welghted Loss
Thought Mask Strategy
cpen somait detseos
Piushed
TQUERT
Notso Cleaning
Data Mixing
Data Fittriing
Laat) Tamee
9
Wooindsong Medal Cotloberatine Selettio
Conamnaryy Filring
(HOUUGHTT
A No thought!
Yeestylirr
famubell
Mlrmall- 700
qwerd-W0
Foeek
JANSWERI
Leucl
Siee
Vension
Sposielled
Hamad-Tee
More Corldnn??
5
PR Dasasolle Strphee
Comppared
(C) THINK THRICE BEFORE YOU ACT: Progressive Thought Roftnement inforence Pipeline
Progressive Thought Stage
Refined Answer Stage
coe.eeeeeee..................... cor.eeee........................ Alm gemerate final response with progressive refinement over time,
Q:A rectengle has length motere and wish of 5 meters What the
Iteration 1: Let's think this way....nswer 17.
Itoration 2: Another aporoach is.. answer is 60.
Answer: Let's go through and final solution answer is 60.
In Iteration 2, inputs is formad
with refine instruction and previous thought, asking model thinking further
Iteration n: think further, considering one conditio... another condition... ... answer 60.
PVA n hilen OPPhhe (.hrr Pihr, _.r,..)
More thoughtful!
Figure 1: Illustration Our approaches. (A) Pipeline of our progressive refinement Dataset construction. We first prepare queries from the general open domain datasets, and pre-procsssing queries in three steps. Then we use a strong weak model collaborative selection strategy to generate thoughts and answers for each query. We also implement In-context Learning (ICL) and Consistency Filtering to ensure the quality of the thought process. (B) The illustration of Weighted Thought Masking Fine-tuning. Aiming at training the model to produce a better response in the next attempt and ensure logical consistency during the thought process. The difference between our method and IFT is that we use thought-mask techniques to ask model to generate better responses. (C) Pipeline of our PTR. Given a query Q. L.LMs think progressively and refine their responses based on their own previous thought and refinement instruction. LLMs refined its mistakes on the second attempt, as well as gave a more thoughtful answer a later iteration.
2023; Charalambous et al., 2023; Nijkamp et al., 2023; Yao et al., 2022; Gou et al., 2023). Models have improved their code generation capabilities by leveraging error messages from the Python interpreter (Wang et al., 2023b) and by teaching large language models to debug and explain their own code, allowing them to identify and fix errors without human feedback (Chen et al., 2023). Similarly, compiler feedback has been utilized in code generation (Chen et al., 2024; Olausson et al., 2023). Additionally, some approaches utilize criticisms or constraints generated by stronger models (Pan et al., 2023; Du et al., 2023; Bai et al., 2022; Huang et al., 2023a), such using strong model to verify the correctness of another model's I's math solutions (Wang et al., 2023a), thereby relying on external information sources to guide improvements. Although models self-correct through external feedback (Pan et al., 2023), this approach does not fully tap into their intrinsic progressive refinement capabilities. Moreover, it requires task-specific feedback models or tools, increasing the cost of adapting to broader range of tasks. Furthermore, current LLMs struggle to self-correct reasoning errors without external feedback (Huang et al., 2023b). Our work aims to unlock the model's 's inherent Progressive Refinement ability, enabling it to perform progressive refinement across all domains without relying on external tools.
Prompting for Progressive Refinement Various Prompting methods have been introduced to enhance Progressive Refinement, such as prompting LLMSs to generate explanations and self-correct code (Li et al., 2023a), or encouraging them to generate alternative solutions and revision suggestions (Zhang et al., 2024). Some methods iteratively improve outputs by generating feedback through task-specific prompts (Madaan et al., 2023a), or guide models to generate fine-grained feedback in mathematical problem-solvingg further enhancing solution accuracy and quality (Xue et al., 2023). The Reflexion method enables language models to operate effectively in specific environments by allowing them to reflect and adjust their actions when encountering errors (Shinn et al., 2023b). However, these approaches often require carefully designed, task-specific prompts even oracle ground-truth
3

---

Published as a conference paper at ICLR 2025
answers (Shinn al., 2023a), making LLMs highly sensitive to evaluating response and achieving optimal performance (Wu et al., 2024a). Without external tools. LLMs have limited self-correction capabilities when relying solely prompting (Huang et al., 2023b; Zheng et al., 2024).
Fine- Tane-Tuning for Progressive Refinement In current progressive refinement work, fine-tuning typically relies reward models verifiers assess the accuracy of model outputs based on predefined criteria ( (Wang et al., 2023a; Lightman et al., 2023; Uesato et al., 2022a). For instance, some research focuses improving the model's ability identify and correct mistakes (Han et al., 2024), while others progressively validate solutions, such as in solving math problems (Uesato et al., 2022b). Additionally. reinforcement learning (RL) (Chen et al., 2024; Yuan et al., 2024; Rosset et al., 2024a; Akyurek et al., 2023) has been applied to align model outputs with correct responses. For example, researchers create preference-ben harmful content (Wang et al., 2024; Rosset et al., 2024b). Similarly, ROUGE has been used as reward function in text summarizing tasks to optimize generated summaries (Akyurek et al., 2023). While these methods effectively train models, they focus on building task- datasets and reward functions tailored to particular objectives. In contrast, our approach redefines the fine-tuning objective to bolster the model's 's capacity for progressive refinement. Rather than relying on domain-speeifer datasets, our model is trained to iteratively enhance its responses starting from initial thoughts and evolving toward increasingly refined answers.
datasets to align outputs with human values and reduce
PROGRESSIVE THOUGHT REFINEMENT FRAMEWORK
Our proposed framework, Progressive Thought Refinement (PTR), comprises two stages, as illus- trated in Figure 1: (1) Progressive Thought Refinement Dataset Construction and (2) Progressive Weighted Thought-Mass Fine-tuning. The primary objective of this framework to enhance models' progressive refinement abilities, enabling them handle diverse and unfamiliar tasks without relying on spec-spe fine-tuning. Since tne-tuning models for every task is impractical, our approach uti- lizes general queries, thoughts, and answers to help models comprehend progressive refinement. This strategy gradually improves their capacity to tackle complex tasks through progressive refinement.
3.1 PROGRESSIVE THOUGHT REFNEMMNT DATASET CONSTRUCTION
In the first stage, we construct progressive refinement dataset that includes Queries, Thoughts, and Answers. The Thoughts capture a sequence of different reasoning attempts, which may varied, incomplete, or even incorrect, reflecting the model's s initial exploration of the problem. In contrast, the Answers provide more confident and weal-reasoned responses. This structured approach helps the model implicitly understand the difference between initial thoughts and improved answers, enabling to generate more thoughtful and in-depth responses over time.
3.1.1 QUERY PREPARATION
To enhance the model's 's generalization, we avoid creating domain-specific datasets. Instead, we use queries from open-domain general datasets (details in Appendix B.6), ensuring the model develops general refinement abilities rather than specializing in specific areas. Our data preprocessing involves three key steps. First, we perform data cleaning remove noise and irrelevant content, such images or URLs. Second, to prevent data leakage, we exclude domain-specific testing queries during training. Finally, we incorporate traditional SFT data (queries and answers) into our dataset to mitigate the risk of catastrophic forgetting.
3.1.2 THOUGHT-ANSSE ANSWER PREPARATION
We strategically select weak and strong models generate sequences of thoughts and improved answers from an initial query. The objective is to ensure that the final answer is progressively improved through multiple iterations rather than relying single-step response. We also employ In-Context Learning (ICL) (Dong et al., 2024) and consistency filtering to ensure logical coherence between thoughts and answers.
Weak-Strong Model Collaborative Selection Criteria To ensure the final answer shows signif- icant improvement over the initial thought sequence, adopt weak-strong model collaborative
4

---

Published as a conference paper at ICLR 2025
selection strategy. Let p.: and 9, represent the abilities of the weak and strong models, respectively, with the goal of ensuring 0g > 8w. We employ three key strategies: Model Parameter Strength, Model Version (New Old), and Domain- -peccffc Fine-Tunee Tuning. These selection strategies ensure the quality of the final answer surpasses that of the previous thoughts. Additionally, we validate that the strong model performs significantly better than the weak model through Wilcoxon significance tests, as shown in Appendix B.5.
Thought Generation by the Weak Model The weak model generates a sequence of thoughts based on the input query 9s, with yi.. representing the initial thought at the t-th attempt. We denote the strong model as Tt and the weak model as π w These initial thoughts may contain errors but provide a foundation for further refinement:
St. thought 1o Vi Bh T w g;).
(3.1)
Multiple weak models can be used to generate these thoughts, or a single weak model can produce multiple attempts. Since the weak model's thoughts need not be correct, constructing these thoughts remains cost-effective.
Answer Refinement by the Strong Model To achieve progressive refinement, we leverage the strong model to produce increasingly improved answers. We use ICL to ensure logical coherence between the outputs of the strong and weak models and avoid randomness. This guides the strong model to generate better answers based on prior thoughts. Specifically, the strong model takes the sequence of thoughts St thrught and query qi as input and generates the final answer Di,siet
D Tstrong,,, (. Si thought q:).
(3.2)
Thoughts-Ansser Consistency Filtering To further ensure that the thought process exhibits logical coherence, we apply consistency filtering remove inconsistent outputs. If the consistency score is below a certain threshold, the pair is considered inconsistent and removed, ensuring that only coherent thought sequences are used for the final output (see Appendix A.
3.2 PROGRESSIVE WEIGHTED THOUGHT--MASS FINE-TUNING
In the second stage, we perform weighted thought-mask fine-tuning using the datasets constructed previously, consisting of the input query qis the initial thought sequence S, thought and the final answer 2i,,,i Formally, the dataset is represented as:
D {(gis Si. thoughus Di.s,ie) )]
(3.3)
Thought Mask Mechanism To help the model understand the improvement between the thought process and the watherr than focusing solely on the final answer we introduce a thought mask mechanism. This mechanism selectively hides parts of the thought process during training, as shown in Figure 1 (B). It calculates the loss based only on the accuracy of the refined final answer, ensuring the model focuses on enhancing the quality of its ultimate response. Additionally, we provide refinement instructions (e.g., "Please continue thinking and refine your answer") after each thought process to prompt better refinement in subsequent iterations.
Weighted Supervised Learning We adopt weighted supervised learning approach that enables the model to focus on refining its answers by progressively improving its thought process. Specifically, we perform weighted supervised learning that emphasizes both the accuracy of the final answers and the logical consistency of the thought process. The loss function optimizes the model in three key areas: generating accurate final answers, maintaining consistency in reasoning and ensuring that the model's confidence increases progressively throughout the thought process.
(oIR(O)=
Σ
A1 log Pr(yn I dis St.hough 0)
(g: d.attt JED
12
B, Pr(st gi S. shoughrs 8)) ].
A2
F Feons (Yee vt-1) ) A1 Σ
(3.4)
2
[=]
5

---

Published as a conference paper at ICLR 2025
Unlike standard supervised fine-tuning, which trains the model to produce a single response ÿ given æ; -A1 log Pr(yn I q; St f) focuses exclusively on the accuracy of the final re- sponse generated after the thought refinement process. F (V-)3)) ensures that the current response remains logically consistent with the previous thought sequence by computing the Cosi- neSimilarity with the Sentence- BERT (Reimers & Gurevych, 2019) model (see Appendix A.2). Pryp qú S.houghhi 0)) represents the model's uncertainty or error probability at each reason- ing step, which measures the confidence of the model's predictions. The term represents the confidence score at each reasoning step, which increases as reasoning progresses to encourage higher certainty in later steps (see Appendix A.3). We also investigate sensitivity analysis and ablation study in Appendix A.5.
EXPERIMENTS
The goal of our experiments is to demonstrate the effectiveness of PTR in enabling language models to progressively enhance their responses. Specifically, we aim to answer the following questions: (1) Can the PTR method activate the model's progressive refinement ability? (2) Does our method demonstrate generalization? (3) Does progressive refinement ability emerge during training? (4) Is our method robust across different LLMs and instruction?? (5) How many iterations are required for our method to achieve optimal performance?
PTR dataset Our model has trained on our PTR (Progressive Thought Refinement) dataset, derived queries from the WizardLM dataset (Xu et al., 2023). After thorough cleaning in Section 1. we reduced the original dataset from approximately 50k QA pairs to 40k high-quality QA pairs.
Evaluation Tasks In our experiments, we perform generalization over ten datasets across different tasks. For general tasks, we MMLU (Hendrycks et al., 2020), and for coding tasks, we HumanEval (Chen et al., 2021) (abbreviated as H-Eval). DROP (Dua et al., 2019) is used for comprehension tasks (abbreviated as Comp), and XSum (Narayan et al., 2018) is applied for summary tasks. We use GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) for math-related tasks. For complex reasoning tasks, we use ARC and GPQA (Rein al., 2023). For knowledge reasoning. we utilize Winogrande (Sakaguchi et al., 2019) (abbreviated as Wino) and CommonsenseQAA (Talmor al., 2019) (abbreviated as Comm).
Evaluation Settings We use greedy decoding (with temperature set to 0) for final generation, as lower temperature yields better performance shown Appendix A.4. We utilize zero-shot prompting (Kojima et al., 2023) for both answer sampling and evaluations. All of our experiments are conducted on workstations equipped with eight NVIDIA A800 PCIe GPUs with 80GB memory, running Ubuntu 20.04.6 LTS and PyTorch 2.0.1.
Directly prompting the model to refine its answer (Huang et al., 2023b). (2) IFT: Instruction Fine-Tuning by directly fine-tuning the input-output pairs from strong models on the PRD dataset to show that improvements are not due to knowledge distillation. (3) RL: Perform one reinforcement learning training (Wu et al., 2024b) iteration on the PRD dataset to compare with our method, Specifically, we use the thoughts and answers of the PRD dataset to construct preference data, and prefer model to produce stronger answers through DPO (Rafailov et al., 2024). We compare these methods on the PRD dataset under the same settings in the previous section.
Baselines We compare our model with base models and prior approaches: (1) Prompt:
4.1 CAN THE PTR METHOD ACTIVATE THE MODEL'S PROGRESSIVE REFINEMENT ABILITY?
PTR Activates Progressive Refinement Ability As shown in Table 1, to emphasize the progressive refinement ability, we conduct tests on a broad range of tasks B.6. The result demonstrates that our PTR activate models substantially refine their responses across multiple iterations in the majority of tasks. For instance, in the Qwen2-7B model, accuracy on MMLU increased by 7.0%, from 57. %% (Base model Prompt Iteration 1) to 64.1% (PTR Iteration 2). On several additional tasks, PTR also showed improvements, with the average score across all tasks increasing by 3.9% - from 49.6% to 53.5%. However, the Prompting method results show that both two base models degrade in performance when asked to refine, producing worse answers compared to initial responses. These results indicate that PTR effectively enables base models to improve based on previous thoughts.

---

Published as a conference paper at ICLR 2025
Gene.
Code
Comp.
Sum.
Math
Reasoning Knowledge
Method
Iters
MMLU
H-Eval
DROP Xsum GSMSK Math ARC GPQA
Wino Comm Avg
Acc
Pass@1 1
Acc
Sim
Acc
Acc
Acc
Acc
Acc
Acc
Qwen2-77
Iter.1
57.1
56.1
20.9
47.3
79.1
48.2
60.6
24.6
66.8
55.7
51.6
Prompt
Iter.2
50.1
37.6
18.7
43.2
78.4
47.6
37.9
22.3
50.4
42.1
42.8
Iter.1
57.7
50.2
21.1
45.5
75.4
43.6
54,9
22.3
66.8
46.1
48.5
IFT
Iter.2
52.4
40.2
17.2
37.9
71.0
43.2
36.6
21.9
62.8
40.3
42.3
Iter.1
56.5
48.3
21.7
47.6
71.2
47.3
60.4
20.3
65.0
51.6
48.9
RL
Iter.2
55.1
42.2
20.9
44.3
58.6
44.5
35.3
20.9
63.8
43.5
42.9
Iter.1
59.2
52.3
19.0
45.9
76.7
47.6
58.6
23.2
66.4
47.9
49.6
(TR(our) Iter:2
64.1
57.2
21.2
49.8
79.6
48.6
62.7
25.6
66.4
54.9
53.0
Iter.3
63.2
57.6
21.5
49.6
79.9
48.9
65.2
25.6
66.8
56.5
53.5
Llama3-88
Iter.1
66.4
54.0
47.3
64.5
76.4
25.1
75.1
34.6
67.9
41.6
55.2
Prompt
Iter:2
34.4
50.1
35.9
62.1
70.5
20.9
56.4
30.1
66.8
43.9
47.1
Iter.1
49.1
38.4
52.8
47.9
55.4
21.3
63.0
34.1
63.3
37.1
46.2
IFT
Iter:2
40.4
34.2
24.7
42.9
51.1
18.6
54.4
28.8
62.2
28.7
38.5
Iter.1
51.8
32.4
46.8
65.9
61.3
22.3
67.6
33.7
62.0
60.1
50.3
RL
Iter.2
39.9
30.2
40.7
40.9
57.3
19.3
55.7
30.6
53.8
55.8
42.4
Iter.1
59.6
54.0
49.0
62.4
76.4
21.3
73.0
34.1
68.6
60.0
55.8
PTR(our) Iten.2
68.4
55.2
49.0
65.7
79.2
24.7
77.1
36.2
70.1
60.5
58.6
Iter.3
68.6
55.4
48.6
66.1
79.6
24.9
77.0
36.1
67.9
61.3
58.6
Table Main experimental results about our approach and other baselines across various domains. We experiments on two difference structures LLMs( Qwen2-7b and Llama3-8B ). We also run iteration on different baselines and iteration on our approach. Itertaion means the first answer to the question, and we construct the format of refining instruction with the previous answer which introduced Method 3.2. We denoted Ace: accuracy. Pass l:testing on code. Sim: similarity similared calculated by BGE-m3. These results suggest that PTR excels performing well across multiple attempts. By trading off some accuracy on the first attempt, significantly enhances the model's ability to improve in subsequent iterations.
PTR not Knowledge Distillation We also compare our PTR with simply use strong model answer to the query by IFT. We find that PTR is not equivalent to knowledge distillation. At the first iteration, observe that when models are trained on general datasets rather than domain-specific tasks, its initial performance tends to decline at first. (This performance drop largely stems from supervised fine-tuning amplifying the initial biases of the base model. When trained on general datasets, the base models tend to accumulate biases that may not apply to specific domains, leading to poorer performance on domain-specific tasks.) However, the IFT approach fails to activate the model's progressive refinement ability and does not significantly increase the performance after the first attempts. On CommonsenceQA, The IFT approach does not perform a better response at the second iteration (40.3%) compared their first attempt (46.1%). In contrast, PTR approach improves through iterative attempts without an approach on comain-specifii knowledge. This suggests that our method is not simply distilling knowledge but effectively activating the model to refine outputs and enhance performance through self-driven iterative improvement.
Refinement beyond Correction ground truth, LLMs refine responses to more thoughtful and comprehensive, regardless of correct- ness. For example, in the code task shown in Figure 2, the LLM iteratively improves its response over three iterations, considering additional aspects of the problem. This highlights PTR's 's ability to enhance not just correctness but also the quality and usability of outputs (Shown in Appendix D).
Deeper analysis reveals that in open-ended tasks without clear
7

---

Published as a conference paper at ICLR 2025
def salculation_avvrreee inumbers): iteration total sum (numbers ra)
(numbers) returning the output
def cslculation_ averringeeeeeeeeeeeeeeeee (numbers):
iteration 2
tctal (numbera)
return total/len (numbera) if len inumbers) > else None & Adda empty liats check
calculation aveeeeeeeeeeeeeeeeeeaeaeaaeeeee (numbers): iteration
if isinstance inumbers, list):
raise Typeßrro: Betngut_must__e_aate
introduces type checking Raises specific with informstive messages
if (zumbers)
raise ValueError satt__contt__b_emme $ Adda protection against ompty lists
sum (numbers) /ien (numbera)
Code example shows PTR can refine beyond correction. The PTR goes through three rounds, providing higher quality response for each iterations. first interation, model return with simply output. In second interation, model add more details like considering the empty list. In third interation, model structured the code and futher add type checking and errors information.
Figure 2:
General
Code
Compreh. Summary
Math
Reasoning
Knowledge
Prompt
Iters
MMLLU
H-Eval
DROP
Xsum
GSM8K Math ARC GPQA
Wino Comm QQA
Acc
Pass@1
Acc
Rank
Acc
Acc
Acc
Acc
Acc
Acc
Iter11
58.7
53.9
19.2
46.9
75.1
47.7
50.5
24.6
66.7
46.7
Iter:2
63.4
57.6
20.7
46.6
77.7
48.8
60.6
25.4
Prompt1
Iter.3
63.2
57.6
20.9
49.8
79.2
50.2
61.3
24.9
65.9
54.7
Iter4
63.3
57.6
21.7
49.9
78.1
50.6
62.8
25.6
66.6
55.7
Iter.
58.7
53.0
19.2
46.6
75.1
47.0
44.5
25.4
66.7
46.7
Iter:2
63.4
52.8
22.1
48.9
77.5
47.2
61.1
25.4
68.8
52.1
Prompt2
Iter.3
62.7
57.9
22.5
49.8
76.8
47.9
59.1
25.6
68.2
50.4
Iter4
62.8
57.6
22.4
49.6
77.5
47.8
60.0
25.8
67.5
53.1
Iter.1
58.7
52.3
19.2
48.8
75.1
47.5
47.5
23.6
66.7
46.8
Iter:2
63.4
57.6
21.6
48.2
78.3
48.6
59.4
25.6
66.4
50.2
Prompt3
Iter.3
62.9
57.9
21.9
49.6
78.1
48.7
62.1
25.0
67.0
55.0
Iter4
63.3
57.8
22.3
49.6
77.9
49.6
63.2
25.4
67.2
53.8
Table 2: Results of PTR with different prompts: (1) Assume that this thought could be either correct or incorrect. Carefully review the thought and provide a better answer. (2) Review your previous thought and assess whether it's correct. Then, provide better response based your answer. (3) Regardless of whether your previous thought is correct or not, provide a better answer. Iteration represent the initial answer to the question. Iteration 2 to 4 represent the model's s improvement over the initial answer. Notibly, the model is not trained with these prompts.
4.2 DOES OUR METHOD REMONSTRATE GENERALIZATION??
PTR Other Progressive Refinement methods Unlike previous approaches, our method activates the model's inherent progressive refinement ability rather than merely boosting accuracy in specific domains. To validate PTR's generalization capability, we use datasets with general queries and evaluate whether the model can iteratively refine responses across various tasks. As seen in Table 1, our model refines responses across multiple iterations, significantly improving accuracy across tasks, and demonstrating effective generalization. We also compare PTR with other progressive refinement methods like RL to assess generalization. Our results show that methods like RL, when fine-tuned only general-domain tasks, fails to activate iterative refinement in specialized tasks, often showing decreased accuracy. This suggests that our method is more robust in diverse environments, as it enables the model to iteratively refine its responses without being limited to domain-specinf fine-tuning. By leveraging the model's inherent progressive refinement capabilities, PTR achieves consistent improvements across wide range of tasks.
8

---

Published as a conference paper at ICLR 2025
(A)
Trands
(C) Distribution
80
80
E 70
Perfrrmace (9%)
70
50
60
550
50
60
GPQA-2
Wine
1
40
30
40
Comm rento
MMLU
30
-6
DROP - 1912 ( 3o 1933 2033
20
10-
20
XSum Searee
Huma
5000
10000
15000
20100
25000
30000
ASum Hunna
GSMM1
MATH
GP
Wind
XSum
Huma
Training Steps
Dasks
Tasks
Training Steps
Figure 3: Plot A: Multi-line plot showing the performance trends for 10 tasks, with the average performance (in black) and variance. The overall trend is upward, with the red line highlighting the point where average performance reaches 46%. Plot B: Bar plot comparing initial, baseline, and final performance for each task. While initial performance is lower than the baseline, the final performance surpasses it, indicating that the model has learned and improved from prior iterations. Plot C: Box plot displaying the performance distribution across tasks. The varying lengths of the bars show the performance improvement, with longer bars indicating greater improvements across tasks, Plot D: Heat map representing task performance across training steps. The x-axis represents the training steps, while the y-axis represents the tasks. The color intensity indicates the model's performance at each task and training step, with deeper colors corresponding to better performance.
4.3 Is OUR METHOD ROBUST ACROSS DIFFERENT LLMS AND INSTRUCTIONS?
Prompt Robustness We also evaluated PTR robustness with different prompts and LLMss. Table 2 shows the model's performance using three different prompts across various tasks, refined over four iterations. Across all prompts, we find that PTR achieves iterative improvement across different prompts. Specifically, In the math (GSM8K) tasks, PTR is well-performed(78.1 initial responses (7).1%). On reasoning tasks (ARC), PTR substantial improvements, especially with Prompts (62.8%) and Prompts (63.2%). DROP tasks also improve steadily. with accuracy increasing 22.5% by Iteration in Prompt 2. Our approach enables the model to learn from previous thoughts, rather than relying on the instruction used during training. This PTR enables the model consistently improve its performance on different prompts, demonstrating the robustness of the PTR mechanism.
compared with
LLMs Robustness The table 1 also demonstrates that both Llama3-8B and Qwen2-7B exhibit robustness across different prompts and tasks. While Llama3-8B often outperforms Qwen2-7B, both models show consistent improvements with iterative refinement. This robustness ensures that PTR can be applied effectively to wide variety of open-source LLMs.
4.4 DOES PROGRESSIVE REFINEMENT ABILITY EXHIBIT EMERGENCE DURING TRAINING?
Overall Performance Figures (A) and (B) show a clear upward trend in performance, shown in Figure shown. Notably, after 24.000 training steps (equivalent to 93 million tokens), significant improvements indicate the emergence of inference capabilities. As training continues, we observe that the average performance of PTR increases from 40. 1 to 55.6%, showing an overall improvement across different tasks.
Task Complexity and Learning Curve We also find that tasks of varying difficulty exhibit different emergence timings and improvement rates. Plots (C) and (D) reveal that simpler tasks such MMLU and DROP show early and steady improvements around 22,000 steps. More complex inference tasks such ARC and GPQA exhibit delayed emergence, with ARC improving from 36.3% to 65.2% and GPQA from 23.2% to 25.6% after 24,000 steps. This shows that as training continues, the model's ability to handle complex reasoning and other tasks significantly improves, showing clear emergent behavior in different task types.
4.5
How MANY THINKING STEPS ARE REQUIRED TO ACHIEVE OPTIMAL PERFORMANCE?
We investigate how iterative thinking steps influence performance across tasks by conducting experi- ments over ten iterations using the Qwen2-8B model. Figure 4 illustrates performance trends.

---

Published as a conference paper at ICLR 2025
GSMBK (Math)
ARC (Reasoning)
Comm (Knowledge)
her
80.0
68.0
57.5
GPOA
her 2
Accrecy
79.0
Accurecy
64.0
Acourecy
55.0
Fer:
78.0-
62.0
har
60.0
Comp
Ier
77.0
73450
7
915
9.10
4455 3010
hana
Baratiom
taratorn
DROP (Comprehension)
MATH
XSum (Summary)
21.0
Accurescy (N)
$0.5
Accurac 691
MMLU
Iteration
$0.0
68.0
20.0
48.
HurnanEval
19.0
DROP
Boost After Iteration
910
XSum
Figure 4: Performance of PTR over ten iterations across different tasks. The left plots show accuracy improvements in mathematical reasoning (GSM8K and MATH), reasoning tasks (ARC, GPQA, Winogrande, CommonsenseeAA) comprehension tasks (MMLU, DROP, XSum), and coding tasks (HumanEval). More details are in Appendix B.7. The dashed line is the baseline of the model. We can see the performance of most tasks surpasses the baseline after the two or third iteration. The right plots show performance over six iterations with radar charts, In this chart, we can find the performance saturated after the first two iterations as shown in the blue areas in the figure.
Improvements in the First Three Iterations In the first three iterations, we saw sigmificant improve- ments in model performance. In the mathematical reasoning task GSM8K, the accuracy improved from 75.00 75.0% in the first iteration to 79.9% in the second iteration. Similarly, the ARC dataset improves from 58.6% to 65.2% in the third iteration. This shows that PTR quickly refines its moblem--sllving through progressive refinement.
After the third iteration, the performance improvements for most tasks stabilize. In GSM8K, the accuracy fluctuates slightly between the third and tenth iterations, ranging from 79.9% to 80. 1. In MATH, the accuracy remains around 50.2% to 50.6% after reaching a peak in the second iteration. This indicates that the marginal gains decrease over time, indicating that the performance ceiling of the model is converging.
Sustained Performance Without Overfitting PTR performance is saturated after the first two iterations, and remains stable or improves slightly, with no notable declines. For instance, in DROP and XSum, accuracy increases from 19.0% and 45.9% to 21.6% and 49.7% respectively, over ten iterations.
More Computation for Hard Tasks Complex tasks benefit more from iterative thinking and may require additional iterations for optimal performance. Accuracy in Commonsense improves from 47.9 7.9% to 58.6% by the eighth iteration, suggesting that tasks with higher cognitive demands allow PTR to leverage iterative refinement more effectively. While GSM8K reaches near-optimal performance within few iterations, tasks like MATH require more computation to achieve substantial gains, likely due to the challenging nature of logical reasoning involved.
5 CONCLUSION
We propose PTR, an approach designed to stimulate the progressive thought refinement capabilities inherent in LLMs, allowing them to improve their responses through multiple rounds of iterations. PTR adopts an annotation-free strategy to gradually build refined thoughts and answers through weak and strong models collaborative selection process, and combines thought-answer consistency filtering to ensure logical coherence. Our weighted thought mask fine-tuning further activates the model's internal refinement ability by learning the improvement from initial thoughts to refined answers. Experimental results show that PTR simply trained with general open-domain datasets, but significantly improves the model's 's progressive refinement capabilities in ten different tasks, including knowledge reasoning, code generation, and mathematical reasoning, achieving a generalization level not observed by previous methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like express their sincere gratitude to Alibaba (China) Co., Ltd. and Alibaba Holding Aicheng Geahnology-Ettrpppin Intelligence Business Unit for their support.
REFERENCES
Afra Feyza Akyurek, Ekin Akyurek, Ashwin Kalyan, Peter Clark, Derry Tanti Wijaya, and Niket Tandon. RL4F:
Generating natural language feedback with reinforcement learning for repairing model outputs. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7716-7733, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v1/2000 1/21633a/// 427.
URL httpstttpt.... http://laaaaatotoooooeaa org/2023 acl acl-long.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.00877) 2022.
Yiannis Charalambous, Norbert Tihanyi, Ridhi Jain, Youcheng Sun, Mohamed Amine Ferrag, and Lucas C Cordeiro. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305. 14752, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint S.XXix2334405122 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.0133 30,35, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https: /ttp///xxvvv///////////.. org/ abs/21101 14168.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey in-context learning, 2024. URL https: / http://axxvvvvooo. abs 2301.0000 00234.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 2019.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critie: Large Language Models can Self-Correct with Tote-Interective Critiquing. arXiv preprint arXiv:2305. 17738, 2023.
Haixia Han, Jiaging Liang, Jie Shi, Qianyu He, and Yanghua Xiao. Small language model can self-correct, 2024. URL https:t//// //ta:///a.ii./// perxxivooree org/ abs 2401. 07301.
Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-YY Yu, Maksym Zhuravinskyi. Eric Hambro, and Roberta Railneau. Glore: When, where, and how to improve Ilm reasoning via global and local refinements. arXiv preprint arXiv:2402. 10963, 2024.

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint S0KV::2000003300. 2020.
Dan Hendrycks. Collin Burn,, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: /ttaxiivvv...////////a/////aavv. orgg alss/ 2103, 03874.
Dong Huang, Qingwen Bu, Jie M Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi-agent- based code generation with iterative testing and optimisation. arXiv preprint urXiv:2312.13010, 2023a.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint 1xXiv:2110.01798, 2023b.
Daniel Kahneman.
Thinking, fast and slow. Farrar, Straus and Giroux, 2011.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models zero-shot reasoners, 2023. URL httpp:://axii../oooo/ 11916.
Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. Explaining competitive-level programming solutions using Ilms, 2023a. URL https////aax...oooaaaa////// 05337.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment, 2023b. URL https: //t:://axxa /awpiviiiiv..////////a.a///aiava org/ atds///1gg 05470.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023a. URL https://///
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint 1765,,, 2023b
Shashi Narayan, Shay Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL https: //arxiv. /t/axxiv.rrr/abbbbeeeea wawpg////////aeeeeeeeeeeeeeeeeee htt//bbs/188 /1088 08745.
Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. ICLR, 2023.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Soaar- Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representationss, 2023.
OpenAl. Learning to reason with 1lm.. 2024. URL https: https://openai.com
learning-to- teraaso-o---r--ooo--oo- weto--rraso-wiihheeeee 11ms/.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse celf-correction strategies. arXiv preprint arXiv:2308.033 188, 2023.
12

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL
https://arxiie org/ wttp:///a////////// 0tt.//255.....000 18290.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks, 2019. URL https: h/tp:///aaiivviiaa org/ abs///b/////// 908. 10084.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof qa benchmark, 2023. URL https: arxia. wttp:/wwww.w//b////..1...1
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. arXiv preprint arXiv:2404.037155 2024a.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences, 2024b. URL https: //rxiiv///vvv..a org/ abs 2404. 03715.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019, URL httttps//aaxxivorrge//
1907. 10641.
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023a. URL
wwt.aawwwww...aaaa/oa/oaeea/aaaa org/ 2230/ 11366.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 2023b.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge, 2019. URL abs/1811. 00937.
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self- improvement of Ilms via imagination, searching, and criticizing, 2024. URL BtLLhtss:///axii org/abs/2e0 /4044. 12253.
Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, and Riccardo Scandariato. Prompting techniques for secure code generation: systematic investigation, 2024. URL https: //wrxxiv.oo... org./arrg/aaa abs 2207 07064.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14277 2022a.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022b. URL ht ttsp:: /aaxxiiivvv.////////aa///aaaavvm org/ abs/ 14275.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR, abv/2312.08935. 2023a.
Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, and Heng Ji. Leti: Learning to generate from textual interactions. arXiv preprint arXiv:2305. 10314, 2023b.
Ziqi Wang. Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, and Heng Ji. Enabling language models to implicitly learn sepfoimmprvement,, 2024. URL https: // arxiv.org. abs/2310 0. 00898.
13

---

Published as a conference paper at ICLR 2025
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct, 2022. URL httpp:://aa.i....tttttt.
abs/2211. 00053.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023. URL https: https: http:////ppeeeee..eeom ntt.:/ounttff
id-hH36JeQZDDa
Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. How easily do irrelevant inputs skew the responses of large language models? In First Conference on Language Modeling, 2024a. URL https: h /tpps//veee//eeeee/ee net fww.mmiuuiiooo 1o-.rmmm...-7
Ting Wu, Xuefeng Li, and Pengfei Liu. Progress regress? self- seffiimproeemen reversal in post- / 2407. 05013.
training, 2024b. URL https:
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions, 2023.
Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. Reot: Detecting and rectifying factual inconsistency in reasoning by reversing Mhain-of-thought arXiv preprint arXiv:2305.114P3 11499, 2023.
Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprini arXiv:2306.022244 2023a.
Kaiyu Yang, Aidan M Swope. Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar. LeanDojo: Theorem Proving with Serrival-AAummente Language Models. arXiv preprint arXiv:2306. 15626, 2023b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXtv:2210.03629, 2022.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arX1v:2401.10000, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https: / /awpivv.vvv///////////.../.aaav. org/
07830.
Wenqi Zhang. Yongliang Shen. Linjuan Wu, Qiuying Peng. Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives, 2024. URL https:
//arxiv. /ts///////.aa.aaa.eeeeeeea 02009.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, et al. Natural plan: Benchmarking 1lms on natural language planning. arXiv preprint arXiv:2406.00 04520, 2024.
14

---

Published as a conference paper at ICLR 2025
A METHOD
A.1 SELF-CONSISTENCY FILTERING
In each iteration of thought generation, we apply multiple sampling techniques to generate several candidate thoughts. These candidate thoughts undergo a consistency check against the final answer to ensure logical coherence throughout the thought process.
A.1.1 N-SAMPLING FOR THOUGITT GENERATION
For each query qi we perform n-sampling to generate N candidate thoughts at each step of the thought generation process. These thoughts denoted as imm. represent the m-th query at the t-th attempt, and they collectively form the set of potential thought sequences.
To evaluate the consistency between the thought sequences and the final answer, we vectorize the thoughts and the answer using Sentence-BER' embeddings. Sentence- provides an effective way to embed both the thought sequences and the final answer into a shared vector space, capturing semantic similarities between them.
A.2 CONSISTENCY COMPUTE
To evaluate the consistency between the thought sequences and the final answer, we vectorize the thoughts and the answer using Sentence-BERT embeddings. Sentence-BEi provides an effective way to embed both the thought sequences and the final answer into a shared vector space, capturing semantic similarities between them.
The similarity between each thought Y, and the prior thought Wt-1 is computed using the cosine similarity between their Sentence- BERT embeddings. The consistency score designed to capture how well the current thought Yt consistent with the prior thought yt-1 and how closely it relates to the final answer.
We define Consistency Function Fooms M1-1) as the cosine similarity between the current thought y/2 and the prior thought y--1- The function is computed as:
F Fooms (ye, Mt- 1) CosineSimilarity(BEER)))) BERT BERT()--))
(A.1)
Where CosineSimilarity(4. b) is defined as:
CosineSimilarityya. b) a b
(A.2)
||||||||||
Here, a and are the embeddings of and 1/-1 respectively, generated by the Sentence-BERT model. The cosine similarity measures the angle between he two vectors in the embedding space, with values closer to 1 indicating high similarity and values closer to 0 indicating low similarity.
A.3 DYNAMIC CONFIDENCE ADJUSTMENT:
By introducing a dynamic confidence decay strategy, the model can gradually increase its confidence during the reasoning process. For example, the confidence 3, can increase as the reasoning step t progresses, instead of staying constant throughout the process. This will allow the model gain more confidence as it refines its thoughts and reasoning.
By adjusting the weights each reasoning step, 3r can control how the model's confidence evolves. At the initial steps, the model can have lower confidence since the thought process is still being refined. As the reasoning progresses, the model should gradually increase its confidence and have higher confidence at the final step. This can be achieved by adjusting B, dynamically like so:
= 90
(A.3)
15

---

Published as a conference paper at ICLR 2025
Average Accuracy Comparison for Different Temperature Settings
53.487
53.00%
53.10%
53
52
51.70%
51.50%
51.30%
Accuracy (%)
51.10%
51
50.80%
50 79.60%
49 48.50%
Fixed Temp
Fixed Temp = 0.4
8.10%
Fixed Temp 0.8
48
+7.60%
Temperature Decay
Iterl
Iter2
Iter3
Iteration
Figure 5: performance of model under different temperature settings during inference
where 3o is the initial confidence and is the total number of reasoning steps. This approach ensures that the confidence 3g increases as reasoning progresses.
A.4 TEMPERATURE ADJUSTMENT
This graph illustrates the performance of model under different temperature settings during inference, measured over three iterations in terms of accuracy. The comparison includes four scenarios: fixed temperature O, the fixed temperature at 0.4, fixed temperature at 0.8, and gradually decaying temperature. The main findings are as follows: The graph clearly shows that setting the temperature yields the best performance. temperature of ensures that the model generates deterministic outputs at every step, leading to more reliable and stable results. Higher temperatures (such as 0.4 and 0.8) introduce randomness into the process, reducing overall accuracy. The decaying temperature approach improves accuracy over time but does not surpass the performance of fixed temperature of
A.5 SENSITIVITY ANALYSIS
Table Performance of different hyperparameter configurations across iterations.
Ratio
Iteration MMLU H-Eval GSM8k
ARC Comm Avg
60.9%
51.2%
76.8%
63, 13.1% 48.5% 60. 1%
100
2
62.3%
53.0%
76.7%
25.2% 52.9% 62.0%
62.7%
52.4%
78.3%
66.4%
55.1% 63.0%
60.9%
51.8%
74.8%
61.3%
48.2% 59.4%
0.9 0.05 0.05
62.6%
55.5%
78.4%
64.3%
54.1% 63.0%
62.5%
56.1%
78.9%
64.5%
54.9% 63.4%
58.5%
51.2%
76.2%
62.2%
48.2% 59.3%
0.8 0.1 0.1
63.3%
55.5%
78.4%
54.1% 63.1% % 66.2%
64.3%
63.9%
56.7%
79.7%
54.8% 64.3%
56.3%
51.8%
74.8%
58.6%
45.9% 57.5%
0.7 0.15 0.15
58.3%
52.4%
70.7%
64.3%
50.6% 59.3%
3
57.9%
52.4%
73.2%
64.5%
52.3% 60.1%
54.8%
48.2%
70.6%
61.3%
44.1% 55.8%
0.2
55.6%
47.6%
71.5%
64.3%
50.3% 57.9%
55.7%
48.2%
71.3%
64.5%
50.8%
58.1%
16

---

Published as a conference paper at ICLR 2025
We conduct sensitivity analysis to evaluate the impact of each hyperparameter. Additionally, explore suggestions for setting these parameters to simplify the tuning process and make our method more accessible for broader applications.
The byperparameters X X2 and X balance various aspects of the model's loss function. However, extensive tuning of these hyperparameters could limit the practical adoption of PTR. Setting À = 0.8, 2 = 0.1, and X 0.1 resulted the highest average accuracy of 64.3%, effectively balancing final answer accuracy, reasoning consistency, and confidence distribution. For broader applications, د = 1.0, X 0.0, A 0.0 achieved an average accuracy of 63.0%, simplifying training while maintaining competitive performance.
Hyperparameter Complexity:
Impact of low A1 Setting Å too low negatively impacts final answer accuracy. For instance, at A 0.6, the model's s ability to self-refine diminishes, reducing overall performance. Roles of X2 and A3: While small values of 22 and À3 support reasoning consistency and confidence progression, their contribution final accuracy limited compared to the computational cost of tuning. For tasks requiring high reasoning consistency and confidence optimization, use λ₁ = 0.8, A2 = 0.1, and Ag 0.1. For most real-world applications, A1 1.0, X2 0.0, and Ag = 0.0 is sufficient.
B EXPERIMENT
B.1
SETTINGS
Table 4: Hyperparameter Settings for Training and Inference
Hyperparaantee/Deeeritttoon
Training Values Inference
bf16
TRUE
N/A
epochs
2
N/A
per device train batch size
I
N/A
gpus
2xH8100
2xH800
gradient accumulation steps
256
N/A
learning rate
5e-5
N/A
weight decay
N/A
warmup step
1000
N/A
learning rate scheduler type cosine
N/A
model max length
2048
2048
temperature
N/A
0
top_p max_new_tokens
N/A
N/A
1000
B.2 ITERATION RESULT
Iterations
Math
Code
Reasoning
Comprehension
Overall Avg.
Baseline
62.75
52.3
49.03
41.37
49,86
Iteration 1
65.10 (+2.35)
57.2 (+4.9)
51.23 (+2.20)
45.03 (+3.66)
53.13 (+3.33)
Iteration 2
65.55 (+0.45)
57.6
52.12 (40.89)
45.57 (40.54)
53.65 (+0.52)
Iteration 3
65.00 (.0.55)
57.3
52.06 (.0.6)
45.10 (0.47)
53.33 (.0.32)
Iteration 4
64.75 (-0.25)
58.1 (+0.8)
52.08 (-0.02)
45.46 (+0.36)
53,54 (+0.21)
Iteration 5
65.05 (+0.30)
57.2 (-0.9)
52.05 (-0.03)
45.03 (-0.43)
53.34 (-0.20)
Iteration 6
64.75 (-0.30)
(40.4
51.96 (-0.99
45,03 (40.00)
53.17 (-0.17)
Iteration 7
65.05 (+0.30)
57.9 (40.3)
52.00 (40.04)
45.37 (10.34)
53.65 (+0.48)
Iteration 8
65.20 (+0.15)
57.9 (40.0)
52.05 (40.05)
45.40 (10.03)
53.70 (+0.05)
Iteration 9
65.00 (-0.20)
57.4 (0.55
52.09 (-0.04)
45.03 (-0.77)
53.70 (+0.00)
Iteration 10
65.00 (+0.00)
57.4 (40.))
52.09 (40.00)
45.03 (40.00)
53.62 (-0.08)
Table Averages for Math, Code, Reasoning, and Comprehension datasets over ten iterations, with colored improvements and declines.
17

---

Published as a conference paper at ICLR 2025
Iterations
GSM8k
MATH
ARC
GPQA
Winogrande
Baseline
76.7
48.8
58.6
23.2
66.4
Iteration
(+2.9)
50.6 (+1.8)
62.7 (+4.1)
25.6 (+2.4)
65.6 (-0.8)
Iteration
79.9 (+0.3)
51.2 (+0.6)
65.2 (+2.5)
25.6 (+0.0)
66.2 (+0.6)
Iteration 3
79.2 (.0.))
50.8 (.0.4)
64.5 (.0.7)
25.5 (-0.1)
66.2 (+0.0)
Iteration 4
78.9 (-0.3)
50.6 (-0.2)
65.2 (+0.7)
25.8 (+0.3)
66.3 (+0.1)
Iteration 5
79.9 (+1.0)
50.2 (.0.4)
66.3 (+1.))
25.6 (-0.2)
65.9 (.0.)
Iteration 6
79.3 (-0.6)
50.2 (+0.0)
65.4 (-0.9)
25.3 (-0.3)
65.8 (-0.))
Iteration
80.1 (+0.8)
50.0 (-0.2)
64.3 (-1.1)
24.9 (-0.4)
66.0
Iteration
79.8 (-0.3)
50.6 (+0.6)
65.2 (+0.9)
25.2 (+0.3)
66.2 (+0.0)
Iteration 9
79.5 (-0.3)
50.2 (-0.4)
65.3 (-0.2)
25.4 (+0.2)
66.3 (+0.)
Iteration 10
79.5 (+0.0)
50.2 (+0.0)
65.3 (+0.0)
25.4 (+0.0)
66.3 (+0.0)
Iterations
CommonsenseeQA
MMLJ
DROP
XSum
HvmanEval
Baseline
47.9
59.2
19.0
45.9
52.3
Iteration
54.9 (+7.0)
64.1 (+4.9)
21.2 (+2.)
49.8 (+3.9)
57.2 (+4.9)
Iteration
56.5 (+1.6)
63.2 (40.9)
(+0.3)
49.6 (+0.2)
57.6 (40.4)
Iteration
56.5 (+0.0)
63.1 (-0.1)
21.5 (+0.0)
48.7 (-0.9)
57.3 (-0.3)
Iteration
56,6 (40.1)
(-..))
21.4 (-0.1)
49.5 (+0.8)
58.1 (40.8)
Iteration
(41.)
62.6 (-D.4)
21.2 (.0.2)
49.6 (+0.0)
57.2 (0.9)
Iteration
(40.3
62.5
21.1
49.3 (0.3)
57.6 (40.4)
Iteration
58.6 (43.4)
(-0.6)
21.4 (+0.3)
(+0.0)
57.9 (40.3)
Iteration
57.5 (-1.)
63.3 (-0.2)
(+0.(+
49.7 (+0.1)
57.9 (40.0)
Iteration
57.7 (+0.2)
63.5 (+0.2)
21.6 (+0.0)
49.3 (.044
57.4 (..5)
Iteration 10
57.7 (40.0)
63.5 (+0.0)
21.6 (+0.0)
49.3 (+0.0)
57.4 (40.0)
Table 6: Results iterations for different datasets, with improvements and declines.
B.3 EPFECTIVENESS OF THE THOUGHT-MAST STRATEGY
We conducted experiments with and without the masking strategy. and the results clearly showed that the mask improves performance. The thought-mask guides the model's attention during training. helping it refine answers based prior reasoning steps. Without the mask, the model tends to compute immature or incorrect intermediate thoughts, leading to worse initial responses.
The IFT method removes the thought process and directly fine-tunes the input with the strong model's answer. This approach demonstrates that our data construction format is not merely distillation of the strong model's abilities but instead successfully triggers the model's 's self-refinement capabilities. Basic distillation does not yield significant improvements specific tasks or enable continuous seff-improvementt, further validating the effectiveness of our method.
Table 7: Performance Comparison with and without Thought- Mask Strategy
Method MMLU H-Eval DROP Xsum GSM8k Math ARC GPQA Wino Comm AVE
UnMask 49.9%
48.8%
17.2% 41.1%
67.4%
42.1% 56.3% 19.4% 61.8% 45.1% 44.9%
55.1%
49.4%
19.5%
40.9%
71.1%
43.8% 62.9% 20.8% 58.4% 50.3% 47.2%
Mask
59.2%
52.4%
19.0%
45.9%
76.7%
47.6% 58.6%
23.2% 66.4% 47.9% 49.7%
64.1%
57.2%
21.2%
49.8%
79.9%
48.6% 62.7% 25.6% 66.4% 54.9% 53.0%
B.4 COMPARISON WITH PRIOR WORKS ON REFINEMENT METHODS OR APPROACHES WITH VERIFIERS
B.4.1 REFINEMENT METHODS OVERVIEW
Self-refine Madaan et al. (2023a) Self-refine does not require training but relies standard answers to assist reasoning. uses specific math-refine prompts to guide base model critiquing and revising its mistakes.
18

---

Published as a conference paper at ICLR 2025
Table 8: Performance Comparison on GSM8K and MATH
Method
GSM8K MATH
Self-refine iteration]
79.1
48.7
With ground truth iteration2
81.3
50.1
Without ground truth iteration2
74.7
48.4
Pair Self Correction iteration
77.7
48.2
Pair Self Correction iteration2
80.1
49.5
Reward-mode Verifier iteration
80.0
48.1
Neward-mode Verifier iteration2
81.7
49.2
Ours iterationl
76.7
47.6
Ours iteration2
79.9
48.9
The model checks the correctness of its response only if the initial answer is wrong. If incorrect, it generates a new response in second iteration and stops as soon as the correct answer predicted.
With ground truth:
Without ground truth: The model always refines its answer without verifying correctness.
Reward Model Verifier (ORM) Cobbe et al. (2021) This method requires training reward model (verifier) and using it during inference to evaluate and refine answers. First, the reward model is trained using best-of-n strategy to construct mathematical dataset. For given problem, 10 candidate answers are generated, and the reward model scores these answers based on correctness. The highest-scoring answers labeled correct, while others are labeled incorrect. During inference, the reward model evaluates the responses iteratively, refining incorrect ones until the final output is most likely correct.
Pair Self-Corrective Welleck et al. (2022) This method trains a single model with self-diagnosis and generation capabilities. fine-tunes large model using pairs of correct and incorrect solutions, enabling it to learn to correct mistakes. If the model's 's self-diagnosis determines the output is correct, no changes are made; if incorrect, the model revises the response.
Experimental results show that performance improvements across methods are modest, typically within 1-3%. The self-refine approach relies on ground-truth feedback for slight gains; without it, performance often deteriorates. Similarly, Pair Self-Correctic and Reward-We Modell Verifier achieve comparable improvements but remain limited.
Our method achieves similar improvements and demonstrates effectiveness in non-mathematical and non-reasoning tasks where other approaches struggle due to challenges in dataset construction. Unlike the self-refine structure, which relies heavily external guidance, model consistently improves performance without needing ground truth, showcasing its broader applicability and robustness.
B.5 WILCOXON SIGNED-RAN RANK TEST
In this experiment, we analyzed samples across three dimensions: model parameter strength and model version (new old), and domain-speciff fine-tuning. Using the Wilcoxon signed-rank test, we assessed the differences in inference quality between the strong and weak models across these dimensions verify whether the strong model provides significant improvements. We use human experts and Auto-j Li et al. (2023b) to judge the quality of the generated responses.
To visually present the score differences across the three dimensions, plotted distribution of inference score differences (see Figure 6). The box plot displays score differences in the following three dimensions:
Model Parameter Strength: Differences in inference quality between models with strong parameters and weak parameters.
Model Version (New vs. Old): Score differences comparing the performance new and old versions of the model.
19

---

Published as a conference paper at ICLR 2025
Domain-Specific Fine-Tuning: Score differences between models that have undergone domain-specific fine-tuning and those that have not. In this work, we simply not using this criteria, since the open open-domian datasets relatively various from tasks. However, can be used in future work.
Wilcaxan "igned-ran test
Uomad 700 vs Usma3 So on 6A
Lamaa 70p vo Usmaa 86 Summory
Lioma3 70b vs Liana 5b on Creation
Stree
3emme-7000 Llemaa-773e qA
Liame3-70l Limma-778
Ulema3776 786 Lleme2-726
Creal
Scoees (0-10)
Model weak:
Modil storg
Medel
Modia irron
Vocul weat
Mode itong
Codlegwen2-/t Owenz-rb
Machapeen2-/e va Qeran2-/b on Mach
HutaMatt--' Lamaz-/b
10
30
Modi
Figure 6: Boxplot of score differences across model parameters, model version, and fine-tuning. The boxplots generated from the data compare two sets of models: weaker models (denoted as "Model weak") and stronger models (denoted "Model strong") across nine different tasks. The comparisons involve different models such as Llama3-70b vs. Llama3-8b, Llama3-70b vs. Llama2-70b, and Codeqwen2-7b Qwen2-7b tasks like QA, Summary, Creation, Math, and Code.
While the boxplots provide visual confirmation that Model Strong outperforms Model weak across all tasks, Wilcoxon signed-rank test can further confirm these results statistically. Based on the boxplots, we would expect the p-values from this test to be significantly less than 0.05, indicating that the differences in performance between Model Weak and Model strong are statistically significant.
Table 9: Wilcoxon Signed-
Rank Test Results for Model Comparisons
Task
Weak Model
Strong Model
p-value Significance Sample Size z-score
QA
Llama3-8b
Llama3-70b
0.05
Significant
100
22.96
Summary
Llama3-8b
Llama3-70b
0.05
Significant
100
20.35
Creation
Llama3-8b
Llama3-70b
0.05
Significant
100
21.85
QA
Llama2-708 70b
Llama3-70b
0.05
Significant
110
19.24
Summary
Llama2-708 70b
Llama3-70b
0.05
Significant
110
18.76
Creation
Llama2-70b
Llama3-70b
0.05
Significant
110
19.57
Code
Qwen2-7h
Codeqwen2--7
0.05
Significant
120
23.67
Math
Qwen2-7b
Mathqwen2-78
-h 0.05
Significant
130
21.43
Math
Llama2-7b
MetaMath-7b 0.05
Significant
130
22.02
We find that the Wilcoxon signed-rank test confirms our previous results. 1) Larger model sizes (c.g., Llama3-70b) consistently outperform smaller models across variety of tasks. 2) Fine-tuning for specific domains (such as coding or math) provides significant performance improvements. 3) newer model versions (e.g., Llama3 Llama2) yield better results, though the improvements generally smaller compared to model size differences.
The analysis of the boxplots clearly demonstrates that stronger models significantly outperform their weaker counterparts across all tasks. These findings suggest that both model size and fine-tuning for specific domains play crucial roles in improving model performance. The Wilcoxon signed-rank test,
20

---

Published as a conference paper at ICLR 2025
if conducted, is expected to support these visual findings, confirming the statistical significance of the observed differences.
B.6 SETTING DETAILS
Open-domain Datasets
WizardLM Xu et al. (2023) is an instruction dataset built with the EVOL-INSTRUIC method. EVOI-INSTRUCT utilizes CHATGPT to augment the complexity of the same queries in Alpaca and ShareGPT.
Data Filtering this section, we provide details about the open-domain datasets used for query preparation. These datasets were chosen for their generalizability and diversity of content, ensuring the model is exposed to a wide range topics and query types. Our selection process was guided by the following criteria:
Data Cleaning Pipeline: The cleaning process involved removing noise such as HTML tags, non-alphanumeric characters, and duplicate entries. We applied frequency-basee based filtering to exclude long-tail queries and low-frequency phrases that are unlikely to contribute to the model's refinement abilities.
Final Dataset Size: After applying all filtering and cleaning steps, the final dataset consisted of approximately 40k high-quality, open-domain query-answer pairs.
Eval Tasks Choice We deploy benchmark evaluate whether approach activate the model's progressive refinement capabilities, enabling it to think and iterate across various dimensions. This comprehensive benchmark encompasses eight categories and tasks, rigorously assessing language models on multiple dimensions including basic perception, mathematics, coding, summarization, continuation, question answering, and experimentation.
In our experiments, we utilized widely recognized and diverse datasets from various domains to comprehensively cover different skills and abilities. For general cognitive abilities, we used the MMLU dataset Hendryeks al. (2020), which spans tasks from junior high to professional exams. Code comprehension and problem-solving were evaluated using the HumanEval dataset Chen et al. (2021), while reading comprehension and reasoning were assessed through the DROP dataset Dua et al. (2019). The XSum dataset Narayan et al. (2018) was used for summarization tasks, and mathematical reasoning was tested using the MATH Hendrycks et al. (2021) and GSM8K Cobbe et al. (2021) datasets. Complex reasoning was evaluated with the GPQA dataset Rein al. (2023). For knowledge representation and common-sense reasoning, we utilized Winogrande Sakaguchi et all (2019) and CommonsenseeAA Talmor et al. (2019). Finally, creative reasoning was tested using the HellaSwag dataset Zellers et al. (2019). Unlike other refinement approaches Wang et al. (2024), we do not partition the evaluation datasets for fine-tuning. Instead, perform fine-tuning on general domain data. To verify the generalization of the model's progressive refinement capabilities, we evaluate on 10 unseen evaluation datasets.
answer correctness using the Accuracy metric. For coding problems, we employ the pass @ 1 metric to gauge the effectiveness of solutions. For subjective questions, we utilize GPT-4 for initial analysis and scoring, supplemented by expert evaluation to ensure comprehensive assessment. This approach emphasizes a multi- dimensional evaluation of responses. focusing only on correctness but also on quality and depth of insight.
Metrics In our evaluation framework, for objective questions,
"We use the Mi in http:///gggiif......oooaaaataaaa wi mardIM/Wi ZaráiM_ evol vvol_instruct_w2_ 196k.
21

---

Published as a conference paper at ICLR 2025
GSMBK (Math)
MATH (Math)
ARC (Reasoning)
80
50.5
Accuracy (86)
Accuracy (%)
50.0
Accuracy (86)
64
78-
GSMBK
MATH
60
77
GSMBK Baseline
MATH Baseline
ARC Baseline
10
2
10
Iterations
Iterations
Iterations
GPQA (Reasoning)
CommonsenseCA (Knowledge)
MMLU (General)
25.5
25.0
Accurary (N0
Accuracy 186
63
54
24.5
52
24.0
GPOA GPQA Baseline
Commorsense@A
MMLU MMLU Baseline
23.5
Commorsense@A Baseline
10
6
10
Iteraiion
Ieertiinn
Iterations
DROP (Camprehension)
XSum (Summary)
HumanEval (Coding)
58
Sccuracy PN
49
Accuracy 190)
20.5
20.0
47
19.5
DROP Baseline
XSum Baseline
HumanEval Baseline
19.0
10
2
10
Iterations
Iterations
Iterations
Figure 7: Performance of PTR over ten iterations across different tasks. The plots show accuracy improvements mathematical reasoning (GSM8K and MATH), reasoning tasks (ARC, GPQA, Winogrande, CommonsenseQAA) comprehension tasks (MMLU, DROP, XSum), and coding tasks (HumanEval), Baseline performance indicated by dashed lines.
B.7 ITERATION
C PROMPTS
C.I PROMPTS FOR PROGRESSIVE REFINEMENT
PTRenables LLMs to progressive refinement, asks LLMs to improve responses without any supervi- sion.
(INPUT]
Explain how reinforcement learning works in machine learning. [OUTPUT1]
Reinforcement learning (RL) is type of machine learning where an agent learns to make decisions by performing actions in an environment to maximize some notion of cumulative reward. The agent receives feedback in the form of rewards or punishments based on the actions it takes, and over time, it learns strategy, or policy, to make better decisions.
[REFINE PROMPT]
Please further think about and give me a more precise and professional answer.
22

---

Published as a conference paper at ICLR 2025
[OUTPUT2]
Reinforcement learning (RL) is a type of machine learning where an agent learns through interactions with environment.
classic example is teaching a robot to navigate a maze. The robot, acting the agent, makes decisions at each step--whether to go left, right, forward, or backward. After each move, the robot gets feedback: reward for moving closer to the exit or penalty for getting stuck or moving further away. Over time, it learns to take actions that maximize its reward, thereby finding the optimal path through the maze.
D CASE STUDY
In this section, we present a comprehensive case study analysis of the output generated by various language models on several real-world tasks. Each case is evaluated based on the quality of responses, relevance to the question, and overall utility for the end-user by GPT-4. These cases were randomly selected from our evaluation results.
We have divided the analysis into six tasks: flight time estimation, web browser usage statistics, language comparison, REST API implementation, JavaScript list item addition, and calculating averages in Python. These real-world tasks are often challenging for humans to label accurately. However, observed significant improvements from Iteration to Iteration 4, with the most optimal performance appearing around Iterations and 3.
D.1 TASK 1: FLIGHT TIME ESTIMATION
Question:
How long does it take from Hangzhou New York by air?
This task evaluates the quality of responses related to estimating flight times from Hangzhou New York. The different outputs showcase varying levels of accuracy and relevance:
Outputs
Iteration It depends on various factors such as the type of airplane and weather conditions. However, the estimated flight time from Hangzhou to New York typically ranges from 14 to hours.
Iteration 2 The flight time from Hangzhou to New York approximately 15 hours and 30 minutes. Major airlines operating this route include China Fastern and Delta Airlines. Keep in mind that additional time for security checks, boarding, and customs at both airports should also be considered.
Iteration 3 The direct flight time from Hangzhou to New York is approximately 15 hours and 30 minutes. Airlines like China Eastern and Delta operate this route. Keep in mind that the total travel time may vary depending on factors such as layovers, flight delays, and weather conditions. Additionally, you should plan for extra time for security checks, boarding, and customs, which can add another to hours at each airport. It's advisable check for updated flight schedules and potential stopover options if direct flight is unavailable.
Iteration 4 The flight on December 2, 2021, from Hangzhou to New York took about 15 hours with a stopover in Tokyo. Depending on the route and the specific date, direct flights can be faster, but you should plan for various connection options.
Analysis
Iteration provides general estimate but introduces unnecessary ambiguity by discussing different airplane types without focusing on direct flights.
Iteration 2 is the most accurate, offering a specific duration of 15 hours and 30 minutes, while also considering practical factors such as airport procedures and layovers.
23

---

Published as a conference paper at ICLR 2025
The improved answer not only provides specific flight duration but also mentions the airlines that operate the route. It further addresses factors that might affect the travel time, such delays and weather conditions, which enhances the answer's accuracy and reliability.
Iteration 3
Iteration includes complex and irrelevant details regarding connecting flights, which are not pertinent to the user's query.
Overall, Iteration 3 provides the most relevant and helpful information by giving a clear estimate and addressing additional travel concern.. Previous outputs either lack focus or introduce unnecessary complexity.
Conclusion:
D.2
TASK 2: BROWSER USAGE STATISTICS
Question:
How long does it take from Hangzhou New York by air? Outputs
Output 1: "As of December 2021, Google Chrome is the most widely used browser in the world with a market share of around 63.47%. This number can change over time as different surveys and data sources may have slightly different results due to varying methodologies and time frames for gathering statistics. It's always best check recent reliable sources like StatCounter or SimilarWeb for up-to-date information on web traffic trends."
"As of now (December 2021), according to analytics tools such as StatCounter, Google Chrome is currently the most commonly used browser across the globe. It has dominant market share of approximately 65%, followed by Safari at around 18%, Microsoft Edge at nearly 4%, Firefox at just under 3%, and Internet Explorer/Windoww Web Experience Platform (IE) accounting for less than 1%. However, it's important to note that these figures are constantly changing as new updates and versions become available."
Output 2:
Output 3: "Based on current estimates, Google Chrome is the most popular browser globally. According to W3Techs, as of September 2021, Google Chrome holds the highest market share among users aged 5+ in over 190 countries worldwide. This means that out of all people who use a desktop computer and are five years old or above, more likely than not they will be using Google Chrome as their preferred web browser. However, please remember that internet usage patterns can vary greatly depending various factors such as region, device type, age groups within a country, etc., so these numbers could fluctuate compared to other data sources specific timeframes."
"Google Chrome is currently the most widely used browser across the globe based on recent statistical analysis from reputable tech firms such as W3Techs. strive to provide accurate and up-to-date information, but please remember that technology and user behavior can rapidly evolve. For the most current stats, would recommend checking websites specializing in web analytics."
Output 4:
Analysis
Output 1:
Strengths: Provides a clear and concise answer: Google Chrome is the most used browser with 63.47% market share as of December 2021. Mentions the importance of checking up-to-date sources like StatCounter for changes.
Weaknesses: Lacks depth regarding other browsers' market shares, Could benefit from more contextual information about the browser landscape.
Output 2:
Strengths: Adds more detail by including market share figures for other major browsers (Safari, Microsoft Edge, Firefox, etc.), offering clearer picture of the browser land- scape. Highlights that these figures constantly changing, adding more realistic and dynamic perspective.
Weaknesses: Could further elaborate on the regional or device-based differences in browser usage patterns.
24

---

Published as a conference paper at ICLR 2025
Output 3:
Strengths: Provides not only market share data but also adds important details about user demographics. such as the age group (5+ years) and countries covered. Discusses how internet usage patterns may vary based on factors like region and device type, adding nuance to the global market share data.
Weaknesses: While detailed, the added complexity might overwhelm users looking for simple answer. The time frame (September 2021) is slightly outdated compared to the other outputs.
Output 4:
Strengths:
Confirms the user's query about Google Chrome's dominance and empha- sizes the importance of checking up-to-date sources due to the rapidly changing nature of technology. Simple and engaging, aimed building rapport with the user.
Weaknesses: Doesn't provide specific market share percentages or as much detailed breakdown as earlier iterations. Leans more towards conversational tone rather than providing a comprehensive analysis.
Conclusion:
Iteration 3 provides the most detailed and comprehensive answer by incorporating specific data, user behavior insights, and global perspective. This iteration strikes a good balance between accuracy, depth, and context, making it the most informative response.
D.3 TASK 3: LANGUAGE COMPARISON (C++ vs. PYTHON)
"Please act as a college student majoring in computer science to answer my question. What's the difference between C++ and Python?" Outputs
Question:
Output 1: "Hello! I'd be happy to help you understand the differences between C++ and Python programming languages. Both powerful tools for building software applications, but they have some key differences:
1. Syntax: C++ has more complex syntax than Python, with many reserved words that must be used exactly as written (e.g., int, if, while). In contrast, Python uses indentation to define blocks of code instead of curly braces or []). making its syntax simpler and easier to read. For example, here's 's equivalent code snippets in both languages for calculating the area of circle:
C++: #include <iostream>...
Python:..."
Output 2: "Sure, I'd be happy explain the difference between C++ and Python!
C++ is a low-level language that provides direct control over system resources such as memory management. was designed for systems programming and game development. It supports object-oriented programming features like inheritance, encapsulation, polymor- phism, etc. However, it requires careful handling memory which can lead bugs done correctly.
Python, on the other hand, a high-level language that abstracts away most these low- level details, allowing programmers to focus more on the logic of their programs rather than managing memory themselves. Python also has extensive librarics and frameworks, such as numpy, scikit-learn, and pandas, which make data analysis much faster and casier compared to writing everything from scratch in C++.
In summary, while both languages serve different purposes, Python is generally considered a safer choice for beginners due to its simplicity and ease of use."
Output 3: "Here's brief rundown the differences between C++ and Python: Similarities: Both are general-purpose compiled languages. They offer strong typing and object-oriented design, and they provide rich standard libraries.
Differences: C++ is faster due to its lower-level access to system resources, but Python is easier for beginners with its simple syntax and robust libraries for tasks like data analysis and web development."
25

---

Published as a conference paper at ICLR 2025
Output 4: "C++ is best suited for systems programming and game development where fine-grained control over system resources is necessary. Its low-level memory management allows for better customizability but comes at the expense of higher complexity. On the other hand, Python offers a simplified approach with high readability for beginners, excelling in web development and data science due its comprehensive libraries. In performance tests, C++ often outperforms Python, particularly for computationally expensive tasks."
Analysis
Output 1:
Strengths:
Provides simple comparison of syntax differences, using concrete code examples, making accessible and easy to understand for beginners.
Weaknesses: Only focuses on syntax differences, without addressing broader usage contexts the strengths and weaknesses of each language in real-world applications.
Output 2:
Strengths: Delivers more in-depth comparison, highlighting differences in function- ality and application areas, such systems programming for C++ and data analysis for Python.
Weaknesses: Could provide real-world examples performance comparisons to give users a clearer understanding of when to use each language.
Output 3:
Strengths: Covers similarities and differences between the languages concisely, giving balanced overview of their strengths.
Weaknesses: Introduces too many complex features, potentially overwhelming the reader with excessive technical details.
Output 4:
Strengths: Combines the best of both approaches, discussing performance differences and suitable applications for each language, along with real-world use cases.
Weaknesses: The output becomes too dense and may lose focus due the volume of information provided.
Overall Analysis
Iteration provides simple and clear comparison that is easy to follow but lacks the depth needed for more advanced readers.
Iteration goes deeper into functional differences, making it solid choice for those looking to understand where each language is best applied.
Iteration tries to offer a more comprehensive comparison but overcomplicates the expla- nation, making it difficult to digest.
Iteration attempts to provide a holistic view of both languages, combining strengths from earlier iterations, but can overwhelm users with too much information.
Conclusion: Iteration 2 offers the best balance between simplicity and depth, effectively comparing the functional differencee C++ and Python while providing clear use cases for each language.
D.4
CASE STUDY: IMPLEMENTING A REST API FLASK
Question:
"Describe how to implement a simple REST API in Flask."
Outputs
Output 1: "Sure, can guide you through implementing basic RESTful API using Flask
your framework of choice. Here the steps:
1. Create Your Python Project: First, create a new Python project and install Flask using pip: pip install flask. This will allow to build our server which we can then host any
26

---

Published as a conference paper at ICLR 2025
web server cloud provider like AWS, Google Cloud etc., for production use.
2. Define Your Endpoints: Next, define what endpoints your application needs to handle by creating routes in Flask..."
Output 2: "1. First, you need to set up Python file with the necessary code for your REST API. You can do this by creating a new file called "app.y"' and adding the following basic structure:
from flask import Flask, request, jsonify
import os class MyAPI (Flask) :... The output also discusses other concepts such as session handling and database manage- ment."
Output 3: "Let's break down the implementation into several parts to ensure clarity:
1. Setting Up the Development Environment: Install Flask and essential dependencies, then create project folder and set up a virtual environment.
2. Flask API Structure: Structure your API and use additional libraries such as SQLAlchemy for database interactions and Twilio's API for sending messages."
Output 4: "Here's simplified outline of what you need to cover to build RESTful API in
Flask:
1. Set up your project and environment.
2. Create Flask app.
3. Define resources and endpoints.
4. Implement CRUD operations.
5. Handle errors gracefully and manage high traffic scenarios."
Analysis
Output 1:
-Strengths:
Provides a concise introduction with a clear step-by-step guide on starting a REST API project with Flask.
Weaknesses: While it gives an initial direction, it lacks detailed implementation of each step. making it insufficient for beginners seeking a more complete understanding.
Output 2:
Strengths: Offers a more detailed explanation, including specific code and a compre- hensive example of REST API structure.
Weaknesses: Attempts to explain too many different concepts (e.g., authentication. session handling) without focusing on the core task of building simple REST API, which could overwhelm beginners.
Output 3:
Strengths: Breaks down the process systematically, covering environment setup and API structure with more technical depth.
Weaknesses: Includes extraneous information such Twilio integration and Elastic- search, which may distract from the central task of building a REST API.
Output 4:
Strengths: Provides an overview of the necessary steps for implementing a REST API with Flask and highlights key concepts such CRUD operations and error handling.
WWeaknesses: Lacks specific code examples, making harder for readers translate the steps into an actual implementation.
Overall Analysis
Iteration provides basic step-by-step guide but lacks depth in key areas such CRUD implementation and error handling.
Iteration attempts to give a more complete explanation but introduces too many advanced concepts, which could confuse beginners.
27

---

Published as a conference paper at ICLR 2025
Iteration offers detailed setup process but includes unnecessary information that distracts from the primary goal building a REST API. too high-level and lacks concrete examples to help readers understand how to apply the steps.
Iteration gives a good overview
Conclusion: Iteration 2 offers the most comprchensive explanation but would benefit from focusing more on the key aspects of building a REST API, while Iteration is best suited for those seeking a simpler introduction.
D.5
CASE STUDY: ADDING AN ITEM TO WEB PAGE'S HTML LIST USING JAVASCRIPT
Question:
"Write a JavaScript code snippet to add an dem to web page HTML list."
Outputs
Output 1: "Sure, here is simple JavaScript code snippet that will add an item with the text 'New Item' your web page:
// Select the first unordered list on the page (you can use this TD class)
const ul if (ul)
wuw.mmenntuerySelectorr. (ful));
const li document. createElement (' (
1i Litteet
'New Item';
ul al.appendChild (1i); }"
Output 2: "To add a new item to an HTML list using JavaScript, follow these steps:
1. Get the reference of the parent element where you want insert the new list item. For example:
const myList document gettlementBBIId myList');
2. Create new list item node as child of the selected parent element: const liNode document. www.tereeeeeee.ee.me ement ("li));
1 1iNode: H = NNww List Item';
3. Append the newly created list item node to the specified parent container:
a 1d (1iNode);
Output 3: "Here is a basic implementation of the functionality you described in JavaScript:
// HTML structure with form and list
<form cd=ateeeoooomm <label for=' newItemText' >Enter new item text:
<input sype='rext! id='newItemIext ><br> <button onclick= addToList () >Add to list</bution> </form>
<ul [dd/exxistingttistIImemm
// JavaScript to handle the button click function addToList ()
var inputText document getSSementBBelBeegcce var li document. ceeaterlement ('1i"); 11 li.textContent inputText; document. document.getREeeeeeett
(neetttttty"""eeetteeeeeee"" newItemText' value;
exi skistingTi it st ) appenddchild(11)
Output 4: "Here's an example of how you might implement this functionality using JavaScript:
// HTML struturee for input form and list <div id=' createForm's
<label for=' newItemText' >Enter new item text: text:</label>dpp text' nd='teeIteeeext <<b>> <button onclick= addToList () >Add to list</button> </div>
<input text='
<ul idd/existtingLiittttmm>>>>>
// JavaScript to handle the add button click
28

---

Published as a conference paper at ICLR 2025
function addToList ()
var inputText = document document. Leteeeeettyeee var 1i document ereateBlement (1 (*1in);
newItemText "eweeeeeeettttyidaeeeeeeeeeett vvalue;
11 Conteet = input
document. Ser.mmentt.eeelleeeeteeeeeee itter//yddddxiiiiiiniiiiii siissiingtssttteme .agpendchildd(1)) aappeeeeemaappp
}"
Analysis
Output 1:
Strengths: Provides simple, concise code example for adding an item to an unordered list. The logic is casy to follow and useful for those who need quick solution.
Weaknesses: The example assumes the presence an unordered list without explaining how the list is structured or created. It's the simplest solution but lacks the depth and completeness of the other outputs.
Output 2:
Strengths: Breaks the process into clear steps with explanations. The structured approach helps users follow along easily.
Weaknesses:
The code becomes slightly verbose and introduces unnecessary steps for what could be simpler task. It also lacks the interactive element provided by Output and Output 4.
Output 3:
Strengths: Provides complete implementation with both HTML and JavaScript, show- casing how to dynamically add items to an HTML list using form. demonstrates both code clarity and a practical use case, including event handling.
Weaknesses: While comprehensive, the of forms and event handling could add small amount of complexity for users unfamiliar with those concepts.
Output 4:
Strengths: Similar to Output 3, it provides full implementation with HTML and JavaScript, with clean event handling and form interaction.
Weeakneses: It includes form handling in a slightly clearer structure but doesn' provide much additional value compared to Output 3. The form setup is slightly redundant for simple task like adding a list item.
Overall Analysis
Iteration 1 is the most concise but lacks the depth and flexibility needed for full imple- mentation, making it the simplest but least powerful solution.
Iteration 2 is well-structured but lacks the interactive element of handling input dynamically through form, making it less functional than Iteration and 4.
Iteration 3 provides the most complete and functional solution, offering both HTML and JavaScript for practical, interactive example.
Iteration also strong but somewhat redundant compared to Iteration offering slightly less detail in terms of implementation.
Conclusion:
Iteration 3 offers the best balance between completeness and practical functionality, It provides a clear and interactive solution, making it the most useful for implementing the task of adding an item an HTML list using JavaSeript.
D.5.1 CASE STUDY: CALCULATING THE AVERAGE OF LIST IN PYTHON
Question:
"Write a Python fachtonfune function that takes a list of numbers and returns the average."
Outputs
29

---

Published as a conference paper at ICLR 2025
Output 1: "Sure! Here is simple Python function that does this: def calculate calculate_average (numbers): total sum (numbers) return total / len (numbers)"
Output "Sure, here Python function called 'calculate "alluraat__vverage that takes a list of numbers as imput and returns the average value. It also handles edge cases like empty list: def celccuaat_aveeeee (numbers) return (numbers) None"
len (numbers) if len (numbers)
else
Output 3: "Here is a Python program that implements the 'calculate_averar average function with additional checks and clarity:
def calculate__ aallatt_aveaggeeeee (numbers) : if 1en (numbers) return None
0:
total sum (numbers) return total / len (numbers)
This function includes check for empty lists and provides a clean and understandable implementation with a return of 'None' for edge cases."
Output 4: "Here is a modified version the 'calculate "averaeee_averrge function that adds extra error handling and raises exceptions for certain cases: def calculate_average (numbers): if len (numbers)
raise ValueError (' Input cannot be empty') elif len (numbers)
return float (' nan')
total sum (numbers)
return total / len (numbers) This function handles both empty lists and single-element lists more strictly by raising exceptions and returning NaN value where applicable."
Analysis
Output 1:
Strengths: Provides the simplest and most direct solution for calculating the average of a list. It is easy to understand and quick to implement for basic use cases.
Weaknesses: Does not handle any edge cases, such as empty lists, which may result in errors if used real-world scenarios. It's a good starting point but lacks robustness.
Output 2:
Strengths: Provides simple and practical solution with a basic error check for empty lists. The function is easy to understand and can handle the common case an empty list by returning "None".
Weaknesses: The function only checks for empty lists but does handle other potential issues such as single-element lists or non- nom-numeric input. It is simpler than necessary for users looking for more robust solution.
Output 3:
Strengths: Provides well- rell-balanced solution with clear code and reasonable error handling. It accounts for edge cases such as empty lists and has a clean and readable structure. This output presents a practical and robust solution for calculating averages.
Weaknesses: The implementation straightforward. but it does handle more complex exceptions such as non non-numeric input, which could be useful for certain applications.
Output 4:
Strengths: Adds more advanced error handling by raising exceptions for empty lists and returning NaN for single-clemen lists. This output is ideal for users who want more strict error handling in specific edge cases.
30

---

Published as a conference paper at ICLR 2025
While the function handles more complex scenarios, the added com- plexity may not be necessary for most average calculation tasks, making the function slightly over-engineered for basic purposes.
Weaknesses:
Overall Analysis
Iteration offers the simplest approach, but doesn't handle edge cases. It's a good introductory solution but lacks robustness for more complex situations.
Iteration 2 is a simpler solution that handles empty lists but lacks more advanced error checking. is useful for straightforward applications where minimal error handling is required.
Iteration provides the most balanced solution, combining clarity with practical error handling. It is the best option for well-rounded, everyday case.
Iteration 4 introduces more strict error handling but adds complexity that may not be necessary for basic tasks. It's useful for those who want more control over edge cases.
Iteration strikes the best balance between simplicity and practical error handling. It provides clear code with clean solution for handling basic edge cases. Iteration 4 is stronger in error handling but may be unnecessarily complex for most use cases. Iteration is simple and effective but lacks robustness, and Iteration 1 is the most basic solution for introductory use.
Conclusion:
31


Reviews:
[{"review_id": "ktjpbpmxyl", "reviewer": "Reviewer_9DXQ", "summary": "Previous methods of self-improvement tried to enhance performance within specific domains, because supervision signals are vague and hard to define and evaluate response quality in open-ended tasks. However, their key limitation lies in lack of generalizability due to high reliance on task-specific pipelines or reward models. Hence, the authors introduce PTR method, which consists of “thought data construction phase” and “weighted thought-mask fine-tuning phase”.", "questions": "- To be clear, what do the authors mean about \"extra feedback\" exactly? Using some other tools, human or compiler feedback, or verifiers (reward models) during *inference* means extra feedback? But this method trained models that can refine their answers without any other feedbacks during inference (though they used extra feedbacks from other models during *training*.)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It's easy to follow, and clear writing and presentations (including nice figures.)\n- Performance improvement over iterations looks very good.\n- Extensive experimental results and analysis of the proposed methods, particularly in Sections 4.3 - 4.5, and qualitative examples.", "weaknesses": "I have some concerns, and ask the authors to explain and mitigate these.\n\n- What are the main contributions and novel points of this paper over some prior refinement works? Weak-strong model to construct refinement data without using rewards, and thought-mask strategy?\n- If yes, I wonder why the authors do not compare with prior works on refinement method or approaches with verifiers. There are so many related works (as also mentioned in Related Work sections), but three methods that the authors showed look very naive. If extra feedbacks are not proper for some datasets, I think comparing in math or code reasoning, where we can easily define the rewards or feedbacks by comparing with ground truth, would be needed at least.\n- As a followup question, why do all three baselines showed worse performance after iterations? I believe these baselines are not good and not well-designed, which makes it very hard to confirm the contributions of this paper.\n- As I understood, thought-mask strategy is using some training dataset with masked on thought parts. How were the results when you didn't use this masking strategy, and only used refinement loss parts? I believe IFT baseline is not this case---if yes, what is the IFT method exactly? (in L.304)\n- What were the experimental settings for fine-tuning? If it costs a lot, would it be better to use some extra feedbacks during inference in perspective of test-time compute efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ul6nX9N1HX", "reviewer": "Reviewer_DjDQ", "summary": "This paper introduces Progressive Thought Refinement (PTR), a framework designed to enable LLMs to iteratively improve their responses through self-refinement. PTR operates in two phases: (1) a thought data construction phase, using a weak-strong model collaboration to build high-quality thought-answer pairs, and (2) a weighted thought-mask fine-tuning phase, which encourages models to enhance responses across multiple iterations. Experimental results demonstrate PTR's effectiveness across ten diverse tasks.", "questions": "See above in weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a novel framework, PTR, that enables LLMs to iteratively self-refine. The concept of self-refine/correct is a very important research direction and has attracted many recent research efforts. The paper has shown reasonable effectiveness and performance improvement in experiments, with a new and interesting framework to achieve this. \n\n2. The two-phase methodology, combining weak-strong model collaboration with thought-mask fine-tuning, is robustly validated across ten diverse datasets. While the demand for strong model is costly, it remains a reasonable approach in application.\n\n3. The paper is well-organized, with clear visuals and structured explanations. The paper also provided a thorough discussion on related concepts and papers.", "weaknesses": "- The function $F_{\\text{cons}}$ and $\\beta_t$ in equation 3.4 is not clearly defined. Clear definitions for the components of the equation are needed in the main paper. \n\n- Hyperparameter complexity. while hyperparameters $\\lambda_1$, $\\lambda_2$, $\\lambda_3$ help balance various aspects of the model's loss function, extensive tuning could limit the practical adoption of PTR. The paper would benefit from a sensitivity analysis or heuristics for setting these parameters to make the method more accessible for broader application.\n\n- Computational overhead. The reliance on a 70B model and the multi-iteration inference adds significant computational demands for both training and inference. It would be beneficial for the authors to discuss the cost of this method (the trade-offs in performance versus efficiency).\n\n- Despite the complexity of the PTR procedure, the performance improvement over direct prompting (iteration 1 baseline) is relatively modest. On average, PTR achieves only a 1.88% improvement by iteration 3 for Qwen. For Llama3, the most significant gains are seen in the CommonsenseQA dataset, whereas tasks such as mathematical reasoning show only minimal benefits. \n\n- Minor typos (e.g., correct alignment of PTR and Prompt1/2/3 in Tables 1 and 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACldcrSnev", "reviewer": "Reviewer_uz4P", "summary": "The paper proposes Progressive Thought Refinement (PTR), a novel framework designed to improve the iterative reasoning capability of large language models (LLMs). PTR leverages a dual-phase approach to construct a progressive refinement dataset and then applies a thought-masking fine-tuning technique to teach LLMs to self-improve. PTR aims to enhance the generalization capability of LLMs across diverse tasks, including knowledge reasoning, code generation, and open-ended text generation, without task-specific fine-tuning. Experimental results indicate that PTR improves performance across a wide range of benchmarks, demonstrating its effectiveness in enabling models to progressively enhance response quality over multiple iterations.", "questions": "- Have the authors investigated the effect of different losses of equation 3.4 in an ablation study?\n- What is the \"PRD\" dataset mentioned in the paragraph of line 303?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses a highly relevant and timely topic within the machine learning community. Self-refinement has been perceived as an important approach for improving pre-trained LLMs.\n- The authors have conducted extensive experiments, exploring the performance of their model and the baselines in different domains.", "weaknesses": "- The manuscript lacks sufficient clarity, making it challenging for readers to follow the paper\n- 2 paragraphs of lines 358-467 are exactly duplicated in lines 510-520!\n- Also, the claim in those paragraphs, \"PTR consistently improves across iterations without signs of overfitting.\" is not the case in Figure 4. It looks like the performance is saturated after the first iteration and is just oscillating from iterations 2 to 10, especially in the XSsum experiment.\n- Figure 1 which is supposed to show the core idea/method is unnecessarily complicated and overloaded making it hard to interpret. \n- Figure 3 has 4 plots with lots of information, small numbers w/o proper take-home messages in the caption.\n-  The description for \\textbf{PTR vs. Knowledge Distillation (IFT)} in line 320 is not clear and hand-wavy.\n- Claiming that LLMs have intrinsic/inherence progressive refinement ability is sort of a big claim that needs more detailed investigations. \n- By looking at the average scores in Table 1, the Prompt baseline is already showing a very solid performance in the first iteration. When comparing this baseline with your method you should consider the complexity of your approach and the time analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Av3eEjmC8", "reviewer": "Reviewer_YHbN", "summary": "The paper proposes Progressive Thought Refinement (PTR) for LLMs, focusing on iterative refinement without task-specific fine-tuning. PTR uses a weak-strong model collaboration and a unique thought-mask fine-tuning approach to refine model reasoning, showing generalization across diverse tasks.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ PTR’s approach to iterative response improvement is novel, aiming to boost LLM generalization.\n+ Benchmarking against established methods helps demonstrate PTR’s relative effectiveness.", "weaknesses": "- Critical information, such as data size, prompt details, and filtering strategies, is not presented clearly within the main text. Instead, these details are found only in the appendix, requiring readers to frequently refer back and forth for a complete understanding. Integrating these important details into the main text rather than the appendix will provide readers with immediate context and reduce the need for cross-referencing, enhancing readability and comprehension.\n\n- The paper lacks clear guidelines on the optimal number of iterations for masking and fine-tuning. Accuracy fluctuates across iterations, indicating the model’s sensitivity to iteration count without sufficient explanation/justification. Establish clear guidelines on the optimal number of iterations for masking and fine-tuning.\n\n- The paper would benefit from testing on datasets designed for reasoning and generalization evaluation, such as MMLU-Pro+ (arXiv:2409.02257), which could provide a more robust demonstration of PTR’s reasoning capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)