A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models
Authors: Jinyi Han
Xinyi Wang
Haiquan Zhao
tingyun li
Zishang Jiang
Sihang Jiang
Jiaqing Liang
Xin Alex Lin
Weikang Zhou
Zeye Sun
Fei Yu
Yanghua Xiao
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Self-refine
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advances in self-refinement have demonstrated significant potential for improving the outputs of large language models (LLMs) through iterative refinement. However, most existing self-refinement methods rely on a reactive process with a fixed number of iterations, making it difficult to determine the optimal timing and content of refinement based on the evolving generation context. 
Inspired by the way humans dynamically refine their thoughts during execution, we propose ProActive Self-Refinement (PASR), a novel method that enables LLMs to refine their outputs during the generation process. 
Unlike methods that regenerate entire responses, PASR proactively decides whether, when, and how to refine based on the model’s internal state and evolving context. We conduct extensive experiments on a diverse set of 10 tasks to evaluate the effectiveness of PASR. 
Experimental results show that PASR significantly enhances problem-solving performance. In particular, on Qwen3-8B, PASR reduces average token consumption by 41.6% compared to standard generation, while also achieving an 8.2% improvement in accuracy. 
Our code and baselines used in the paper are available in the GitHub.

Published as a conference paper at ICLR 2026
A STITCH IN TIME SAVES NINE: PROACTIVE SELF- REFINEMENT FOR LANGUAGE MODELS
Jinyi Han Xinyi Wang Haiquan Zhao Tingyun li Zishang Jiang 9, ,Sihang Jiang
Jiaqing Liang 9, Xin Lin Weikang Zhou *, Zeye Sun *, Fei Yu *, Yanghua Xiao 2,
Shanghai Institute of Artificial Intelligence for Education, East China Normal University
School of Data Science, Fudan University
College of Computer Science and Artificial Intelligence, Fudan University Ant Group
ABSTRACT
Recent advances self-refinement have demonstrated significant potential for im- proving the outputs of large language models (LLMs) through iterative refinement. However, most existing self-refinement methods rely on a reactive process with fixed number of iterations, making it difficult to determine the optimal timing and content refinement based on the evolving generation context. Inspired by the way humans dynamically refine their thoughts during execution, we propose ProActive Reff-Reeinnment (PASR), a novel method that enables LLMs to refine their outputs during the generation process. Unlike methods that regenerate entire responses, PASR proactively decides whether, when, and how to refine based on the model's 's internal state and evolving context. We conduct extensive experiments on a diverse set of 10 tasks to evaluate the effectiveness of PASR. Experimental results show that PASR significantly enhances problem-solving performance. particular, on Qwen3-8B, PASR reduces average token consumption by 41.6% compared to standard generation, while also achieving an 8.2% improvement accuracy. Our code and baselines used in the paper available in the GitHub !
INTRODUCTION
Self-refinement, as fundamental cognitive capacity, is essential for effective problem-solving in humans. involves actively monitoring one's thought processes, identifying and correcting errors, and iteratively adjusting responses and behaviors (Dewey, 1986; Kuhl & Beckmann, 2012). Its significance human intelligence highlights a promising direction for developing more autonomous and robust AI agents. Inspired by this powerful cognitive process, recent work has applied the self-refinement to Large Language Models (LLMs).
Existing self-refinement methods for LLMs typically follow patch-after-failure (post-hoc) paradigm, where an initial response is generated and then iteratively improved based on feedback through multiple rounds of refinement Iierations(Madaal Ganguli et al., 2023b). Broadly, these methods fall into two categories. The first employs carefully crafted prompts elicit relf-refinement behaviors. often by explicitly instructing it correct or refine its previous outputs (Ganguli et al., 2023a; Olausson et al., 2024; 2023a). The second leverages Supervised Fine-Tume Tuning (SFT) on synthetic datasets that pair suboptimal responses with improved versions, training the model to refine its outputs automatically (Havrilla et al., 2024; Du et al., 2025). (Tong et al., 2024; Xie et al., 2025; An et al., 2024).
al., 2023; Welleck et al., 2023; Huang et al., 2024;
While these post-hoc self-refinement methods have improved performance various tasks, they remain fundamentally reactive and lack the ability to proactively determine whether, when and how to perform refinement. (Whether:) these methods are often applied blindly after initial generation, requiring multiple iterations whose optimal number is unclear and usually demands extensive tuning (Du et al., 2025; Madaan et al., 2023). (When:) errors arising during initial generation can propagate through subsequent steps (Gan et al., 2025; Bachmann Nagarajan, 2024), making later correction
Corresponding author

---

Published as a conference paper at ICLR 2026
There are lots of boxes: A, C, C..., One box contains gold; the other are emply, Euch box has statement written on ä:: Box As The goid not in Box B,
Exuctly one of these stutements true. Which box contains the gold?
Direct answer
Post-hoc self-refinement
PASR
tivll solve, our
This clessic logic puzzle! So the gold is in
ethinko To def definitively solve this,
methho begins by establishing o
the A box:
X
crefing When the gold ossumed to
systematic strategy: If gold is Statement ('old is olso true...
Please further think about the question and previcus onswer, give
an internal check rapidly confirms that Shifting assumption, the gold </refiiee
precise and professional onawer.
exectly one true statament"... that the Gold in
At point, .//thinko canswers
Okky, I'll ne ee-evvllute the Gold is in Box X
onawer s//nsswere
Figure I: Comparison between the post-hoe refinement method (middle) and our proposed PASR (right). The post-hoe refinement method iteratively refines its initial answer. In contrast, PASR proactively refines reasoning process during the generation.
more difficult. (How:) these methods rely heavily on external feedback mechanisms, such as tool- assisted evaluations and auxiliary models (Gou et al., 2024; Xie al., 2025; Chen et al., 2024), and inappropriate feedback even degrade the performance (Huang et al., 2024).
It is crucial to equip LLMs with proactive self-refinement capabilities during generation, allowing models to autonomously determine the appropriate timing and content for refinement based on the evolving context. While advanced reasoning models like DeepSeek-RI (Guo al., 2025) and OpenAl-ol (Jaech et al., 2024) demonstrate some in in-process refinement behaviors, these mechanisms are neither explicitly designed for proactive self-refinement nor systematically evaluated for their impact on output quality. Furthermore, the underlying mechanisms driving these refinements remain unclear, limiting our understanding of how to develop more effective self-refinement capabilities in LLMs.
straightforward approach for equipping L.LMs with proactive self-refinement training them on demonstrations of adaptive refinement behavior. However, this method faces two significant challenges. First, constructing such demonstration data is non-trivial, as defining the optimal timing for refinement during generation is impractical and distilling it from advanced LLMs is not feasible. Second, merely imitating these demonstrations is insufficient for the model to truly acquire the capability (Kumar al., 2025; Wang al., 2025). Models struggles generalize adaptive self- refinement behavior to unseen tasks, and in some cases, their performance even deteriorates.
Therefore, propose ProActive Self- RefineRefnemett (PASR), Reinforcement Learning (RL) method that trains L.MS to adaptively refine their outputs during generation. Unlike post-hoc refinement, which is applied after generation based on predefined rules, PASR leverages on-policy rollouts to explore whether, when, and how to refine, conditioned on the task and generation state (Figure I). In contrast to SFT, RL shapes the model's behavior through reward signals (Lee al., 2024; Yuan et al., 2024). A key challenge is defining what counts as an effective refinement. If the rewards are misaligned, the model may either miss important refinement opportunities or make unnecessary modifications to already correct outputs. To address this, we introduce a proxy evaluation strategy that compares refinements against standard outputs, encouraging timely, necessary, and contextually appropriate refinement.
In summary, our main contributions are summarized as follows:
We formally define proactive self-refinement as task, allowing models to iteratively decide whether, when, and how to refine.
We introduce ProActive Self- Refinement (PASR), reinforcement learning framework that enables
LLMs to autonomously refine their outputs during generation.
We design comparison-based reward that encourages timely, necessary, and contextually appro- priate refinements.
Extensive experiments show that PASR improves both efficiency and accuracy. Notably, on Qwen3- 8B, it reduces token consumption by 41.6% while increasing accuracy by 8.2%, demonstrating the practical effectiveness of proactive self-refinement.
2

---

Published as a conference paper at ICLR 2026
2
METHOD
2.1 TASK FORMULATION
Unlike existing post-hoc refinement methods, our task is that empowers the model to proactive self- refine its generated content during the generation process. We formalize this in- in-process refinement behavior follows:
Error Correction. Fixing factual inaccuracies, logical fallacies, or computational mistakes introduced in earlier outputs.
ness. Information Complement. Filling in missing yet critical details to ensure completeness and correct-
Solution Improvement. Improving the effectiveness and efficiency of the proposed solution by introducing more advanced strategies refined representations.
Task Alignment. Re-aligning content with the task goal or user intent when divergence is detected.
The model proactively decides whether, when and how refine previously generated parts of its internal reasoning trace, integrating these updates into its ongoing generation process. This sequential decision- decision-making problem is naturally formulated as a Markov Decision Process (MDP) (Bellman, 1957).
Formally, given an input query the goal is to generate a final response This is achieved through an iterative refinement process that constructs an intermediate generation trace (21, .2..... 2T) where 7' is the total number of generation tokens. At each timestep i (from to 7), the model is in the state s,, which is determined by the input and the trace generated 2(1:i-1 so far. It then takes an action a, chosen from an action space A, which consists of two main types of actions: Content Generation dgen and Trace Refinement Orefine- The Content Generation extends the current line of reasoning. The model produces the next reasoning step and appends it directly to the end of the cxisting trace, thereby moving the reasoning process forward. The Truce Refinement focuses on improving the quality of the already generated trace. Instead of advancing the reasoning, the model inspects previously produced content, identifies potential weaknesses, and generates corrective or explanatory additions to enhance clarity, consistency, correctness. The sequence of states, actions, and resulting trace segments ((s), a,, 21),..., (ST, QT, 27)) constitutes an observation. The final response is derived from the complete trace z. The training objective is to learn the optimal policy that maximizes the expected reward of proactive refinement responses. The reward, denoted as R reflects the quality of the response resulting from proactive trace refinement. The objective is formalized as:
max Σ
EB (y
Ryj
2.2 PASR: PROACTIVE SELF-REHNEMENT VIA RL
In this work, we employ Group Relative Policy Optimization (GRPO) algorithm, variant of Proximal Policy Optimization (PPO), specifically designed stabilize training through group-war wise advantage normalization. For each query the policy me samples group of candidate responses G₂ (1/ R.j. (V., R.y,),))))) }}. where each pair contains response and its reward.
We normalize the advantage of each response y, in group G.
A.(víl=) = Ryj Ps 02 +
(2)
where Na and 0g are the mean and standard deviation of rewards in Gg. and is a small constant added for numerical stability avoid division by zero. The GRPO objective function JGRPO (0) is formulated to balance reward maximization and policy stability, which is defined as:
Jauro = Eo G
Σ
Ac(v(=z) min (ri, clip(r:, 1- -c,1+c)) - 3DDu (1=
(3)
where no [z())> (9 [mn Fold is the policy before the update. is hyperparameter controlling the clipping range, and weights the KL-ddivergence penalty. The KI. divergence term, DKL. DKII(To||[TTe)
3

---

Published as a conference paper at ICLR 2026
(-1,1)
Red win: 1 Blue win: -1 Tie: -0.5
<think>
Format
Here is the reasoning process...
Refine score
score
<refine>
Error Correction;
Predefined Jarmad
Evaluation
Equation (6)
Information Complement;
Solution improvement;
Refinement
Average accuracy
Alignment;
</refine>
Oxyy
Golden answer).
Judge LLM
Integrote the refinement and continue reasoning
Judge
goldie unswer)
Accuracy
Comparison
</think>
y1
Y/
3/n
<answer> Here is the final onswer </answer>
Score
[0,1]
Standard answers
Figure 2: Left: Answer format used in PASR. Right: Reward design for a generated answer y during training. The total reward is computed as the of the format score, accuracy score, and refinement score, as defined in Equation 7.
Meed (v/12) log
1, enforces proximity to reference policy Tref, thus preventing
no (x((=))
excessive policy shifts and mitigating the risk of over-optimization.
PASR Rollout. To enable the model autonomously determine both whether, when and how perform refinement during the generation process, we first design a structured output format guided by system prompt. The prompt shown in Table 6.
The system prompt explicitly instructs the model to structure its output using three specialized tags: <think>, <refine> and <answer>, which denote the reasoning trajectory, refinement segments, and final response, respectively. The <think> tag encapsulates the model's 's entire reasoning trajectory. Within this reasoning scope, the <refine> tag identifies specific segments where the model revises or improves previously generated content. The <refin>> tag required to be nested within the <think> tag, indicating that refinement is an integral part the model's reasoning process. After each <refine> segment, the model continues reasoning based on the updated content, allowing refinements to directly influence subsequent steps. The model is encouraged perform recursive refinement, allowing it to invoke the <refin>> action multiple times during a single generation whenever it deems further improvements beneficial.
The use of these specialized tags enforces semantically structured generation process, guiding the model to clearly distinguish and focus on each phase, including reasoning. refinement, and final response, with each phase serving explicit functional role. The refinement output format of PASR illustrated in Figure 2.
2.3 REWARD DESIGN
Rule-based reward mechanisms have demonstrated strong empirical performance and are widely adopted RL settings (Dao & Vu, 2025; Shao et al., 2024). In our training framework, we employ a hybrid reward scheme that combines rule-based and model-base evaluation to guide both generation and refinement behavior. Specifically, we define three types of rewards: the format reward Tformat the accuracy reward Face and the refinement reward Frefine-
Format Reward. This reward evaluates whether the generated output conforms to predefined structural constraints, defined follows:
Constraint 1 (CI): the output must include both <think> and <answer> tag pairs; the <refine> tag optional.
Constraint 2 (C2): if the <refine> tag appears, it must be properly nested within the <think> tag. Constraint 3 (C3): the relative order of the three tags must be preserved and cannot be rearranged.
Let Culy') E 0, indicates whether condition C, is satisfied for given output y. The format reward Tronmat (v) is then defined as:
Tformac (y') 2(Ci(v') Ca(v') Ca(v))
4

---

Published as a conference paper at ICLR 2026
This formulation assigns reward of if and only if all constraints are satisfied; otherwise, penalty of is applied. The strict binary scheme ensures that only fully well-formed outputs are positively reinforced.
Accuracy Reward. is designed to evaluate the quality and correctness of PASR's generated answers. As our training tasks are drawn from open-domain question. many inherently ambiguous or under-specifieed. Consequently, outputs are diverse and expressed in free-form language, making rule-based checks exact string matching ineffective.
To address this issue, following prior work (Zheng et al., 2023), we employ an advanced LLM as a judge model. The evaluation model is prompted with three components: the original question x; the generated answer y and the reference answer y. then outputs continuous score in the range [0, 1]. reflecting the semantic quality and task relevance of the generated response relative to the reference. Let J denote the judgment function, the accuracy reward Taee is defined as:
(ace(y') J [z. 0.9')
Refinement Reward. used to assess whether refinement actions of y are beneficial and timely. Directly measuring the effectiveness of adaptive self-refinement is challenging. we instead employ a proxy evaluation strategy that assesses refinement quality by comparing the refined response y' with set of standard responses y without refinement. Given the stochastic nature of the model's generation, we sample multiple standard responses to estimate the expected accuracy of the model, denoted (y). The refinement reward is designed according to the follows principles:
Reward effective refinements. positive reward is assigned if the refined response achieves signifi- cantly higher accuracy than the baseline average.
Penalize harmful refinements. negative reward given if refinement decreases accuracy relative to the baseline average.
Discourage unnecessary refinements. If the refined response yields comparable accuracy to the baseline average, a small penalty applied to discourage redundant changes.
Formally, the refinement reward is defined as:
1, Laccl))) (y) +
Trefine (g')
1, Tacc( (v') < Face -
(6)
0.5, |racc (a) SC
Here, is tolerance parameter that provides robustness against noise and minor fluctuations. This formulation encourages the model to refine its output only when the refinement yields a measurable gain, while penalizing ineffective unnecessary modifications.
Overall Reward. The final reward for each response generated by my computed as the sum of the three components.
RS format (y')
Tace (v'c Treffine (3')
Unlike prior approaches that rely solely on binary reward signals, our fine-grained reward designed to encourage meaningful and constructive refinement while explicitly discouraging both excessive and insufficient refinement.
EXPERIMENTS
3.1 SETUP
Training Data. Our training data is derived from the alpaca_evol
dataset, general cooruction-oolloiin corpus. We performed thorough cleaning and filtering process based on the following criteria: (1) Removed questions with excessively long ground truth answers to maintain manageable response lengths. (2) Eliminated noise such as HTML tags, non-alphanumeric characters, and duplicate entries. (3) Applied frequency- based filtering exclude rare long-tail queries and
Phttps://huggingfaace wtt./aaattt///aaaaaaaaaaaaaaaaa htwaaa/wwwaaaaae.oooooosstttttt. i___insutttrt_ee
5

---

Published as a conference paper at ICLR 2026
Table 1: PASR vs. other baselines. Compared the base model, PASR achieves an average performance improvement of +4.8 and +8.2 on the two models, respectively.
Math
Reasoning
Knowledge
Comp. Drop
Gene.
Sum.
Methuds
Public
GSMSK MATH AIME24
ARC GPQA
Wino
CSQA
MMLU
Xsum
Qwen2.5-7B
Vanilla
68.4
6.7
25.6
64.7
62.8
78.6
Self-Refinc (Madaan et 2023)
NIP8
89.6
69.4
6.7
89.0
27.7
73.8
67.5
80.2
63.0
56.2
62.3
Selr Refine(Shimi et 2023)
NIPS523
88.7
68.4
16.7
85.3
25.6
64.1
62.3
78.6
49.0
36.0
57.5
PTRiDu al., 2025)
ICLR'25
88.6
61.8
10.0
91.0
27.7
59.0
75.3
75.7
74.0
50.4
61.6
SCaRe|Kumar et al., 2025)
2C
82.4
63.2
3.3
67.2
14.5
48.1
46.4
65.8
56.0
35.0
48.2
STaR(Zelikman
2022)
NIPS'22
70.8
10.0
88.3
19.3
53.7
19.4
72.2
32.9
49.7
ISC et al., 2024)
AAA1224
56.2
56.6
6.7
67.6
19.4
56.3
50.1
57.8
35.0
31.5
43.7
RISE(QU al., 2024)
N4PS
84,9
62.4
13.3
82,9
23.7
60.9
74.5
73.1
45.0
56.6
51.7
PASR(-promp)) PASR(-IFT)
54.4
6.7
46.8
22.5
34.8
30.3
70.6
23.1
40.2
89.2
70.8
3.3
84.6
23.6
62.4
65.4
77.3
51.0
42.0
$7.0
PASRÍ
88.8
73.6
10.0
86.6
29.3
57.0
67.0
79.6
75.0
49.9
61.7
Qwen3-8B
Vanilla Self-Refine (Madaon RMMdun al., 2023)
91.3
80.2
13.3
89.0
25.0
64.5
66.3
71.2
72.0
36.3
60.9
NIPS'23
94.0
83.0
85.0
51.1
Self Refine(Shinn et al., 2023)
NIPS'23
90.5
73.0
10.0
91.1
29.1
76.8
25.8
80.8
73.0
50.2
65.0
PTR(D al.,
ICLR225
88.7
72.0
6.7
80.9
32.3
66.1
46..
65.5
53.0
33.7
54.5
SCoRe(Kumar et al., 2025)
ICLR'25
91.4
81.2
13.3
87.3
36.7
70.7
63.9
78.9
72.0
45.0
64.0
STaR(Zelikman et al., 2022)
NIPS'22
72.7
55.2
0.0
64.2
26.0
55.3
28.8
49,5
22.0
13.7
38.7
ISC(Han et 2024)
AAA1'24
6.7
29.2
63.5
42.5
38.3
RISE/Qu et al., 2024)
NIPS 24
92.5
77.4
16.7
88.3
33.3
70.8
37.2
82.4
44.0
49.3
59.2
PASR(=ppompt)
60.3
67.8
10.0
$7.9
29.4
60.4
74.3
75.1
52.0
26.6
$1.4
PFSR(+IFT)
74.6
6.7
73.6
35,1
68.7
29.3
73.5
36.0
36.3
52.6
PASRI
94.9
81.4
16.7
92.3
24.5
80.0
79.6
85.3
83,0
53.0
69.1
low-frequency phrases that are unlikely to contribute effectively to the model's refinement capabilities. After these preprocessing steps, we obtained approximately 40,000 high-quality, open-domain query- answer pairs for training. We have release the training data in the GitHub.
Benchmarks and Metrics. We evaluate generalization of PASR across ten datasets covering diverse tasks. For general knowledge evaluation, we use MMLU (Hendrycks et al., 202 DROP (Dua al., 2019) is included to assess multi-hop and comprehensive reasoning. Mathematical reasoning is evaluated using GSM8K (Cobbe et al., 2021). I) MATH (Hendrycks et al., 2021b), and AIME24 3 To test complex reasoning abilities, adapt ARC and GPQA Winogrande (Wino) (Sakaguchi et al., 2021) and CommonsenseQA (CSQA) (Talmor et al., 2019) are used for knowledge-basd reasoning. For summarization, we XSum dataset 6, Accuracy is used the evaluation metric for all datasets except XSum, for which we report similarity scores.
Baselines. We use Qwen2.5-7B (Qwen al., 2025) and Qwen3-8B7 as the backbone models, and compare PASR against several existing methods designed to induce self-improvement or self- correction abilities in LLMs. The baselines include: (1) Self-refine (Shinn et al., 2023): Prompts a base model to critique and iteratively revise its own responses in single-turn format. (2) Self-refine" al., 2023): An enhanced version of Self- Self-Refine that uses ground-trute answers as an oracle signal. The model first generates initial response, then identifies errors by comparing with the ground truth and revises them accordingly. (3) PTR (Du et al., 2025): Constructs progressive self-refinement dataset and applies instruction tuning to enable multi-turn, answer-level refinement. (4) SCoRe (Kumar et al., 2025): Employs a multi-turn reinforcement learning framework to train LLMs to self-correct without relying oracle feedback. (5) STaR (Zelikman et al., 2022): Uses few-shot prompting to generate rationales for multiple questions. the answer is incorrect, the rationale is regenerated using the correct answer. The model iteratively fine-tuned on rationales that lead to correct outcomes.
(with oracle) (Madaan
(6) ISC (Han et al., 2024): Builds a self-correction dataset and applies instruction tuning to train the model's intrinsic self-correction ability to detect and amend its errors. (7) RISE (Qu et al., 2024): Creates improvement trajectories showing how a model can refine its own responses under own distribution, and fine-tunes the model on these recursive rollouts.
'https://huggiggt
http:///wgggni.ee.ooooooaaaatat/ co//ataaets/mmth-aa//aimme
ottterr///ngiiiiteeeoooooaeesssd oo/ddatasets/alllemi////a...r
'nttps://huggingfaa wttp//attaeet////aaivireee/////a httpss/s/migiiiicce..ooettetttts
hnttps://hugging:
hhtpp::/huggingffaa

---

Published as a conference paper at ICLR 2026
SCoRc
MMLU
ISC
AMG
roffrcnnce
PASR(HFF)
779
Vanila PASRIIFFI
CS04
GSMSK
Sal refine
GPOA
MMLU
Drrop
Ksum
GSMBK
MATH
AUME
ARC
GPOA
Wind
CSDA
AVG
AIME24
Figure 3: Comparison of average token length across different methods on various tasks. The left figure uses the Qwen3-88 8B backbone, while the right figure uses Qwen2.5-7B.
Detailed descriptions of the prompts, important parameters and implementation settings for all baselines are shown in the Appendix A.
3.2
MAIN RESULTS
3.2.
PERFORMANCE ANALYSIS OF PASR
Unlike prior approaches that perform refinement only after the generation is complete, PASR refines answers adaptively during the generation process. To evaluate its effectiveness, we conduct experi- ments across a diverse set of tasks, with focus generalization capability. For fair comparison, we re-implement representative baselines that are only trained on specific domains under the same training data. The results are shown in Table 1.
PASR consistently outperforms baseline models, with particularly notable gains on more challenging tasks. For example, on the Qwen2.5-7B model evaluated with the MATH dataset, PASR yields 5.2 improvement accuracy compared to the standard method. Similarly, on the Qwen3-8B model tested with the Drop dataset, PASR achieves 14.1 accuracy gain over the standard method. These results suggest that PASR, is capable of dynamically detecting and correcting reasoning errors, leading to effective and domain- performance gains.
PASR achieves high performance without relying on external feedback or task-specific super- vision. We find that only when oracle feedback is available to assist refinement, the self-refine" provides the performance boost. This highlights the limitation the self-refine structure in effec- tively improving model performance without external guidance which is also observed in (Kumar et al., 2025; Qu et al., 2024). However, external supervision signals are often difficult to obtain and introduce additional costs. contrast, PASR performs self-refinement autonomously, relying solely on intrinsic, self-adaptive decisions made during the generation process.
PASR demonstrates strong generalization capabilities. PASR is trained on general tasks and evaluated on domain-specific datasets to assess its generalization ability. Despite this domain shift, PASR achieves the best average performance compared other self self-refinement methods. While PASR does not always outperform all baselines every individual dataset. For instance, its performance on Qwen2.5-7B slightly lower on certain understandable. Domain-specifer tasks often require specialized knowledge exhibit distributional characteristics not present the training data. Moreover, we observe that the effectiveness PASR also vary with the underlying model. Compared to the more advanced Qwen3-8B, Qwen2.5- 7B appears to exhibit a relatively weaker ability to leverage the learned proactive self-refinement mechanism. This suggests that stronger base models provide fundamental proactive self- refinement capability.
tasks. This outcome is expected and
3.2.2 EFFICIENCY ANALYSIS OF PASR
PASR optimizes the output quality with minimal additional token overhead. We compare token consumption across different baselines, illustrated in Figure 3. Compared to standard decoding method, PASR achieves notable accuracy gains with only a slight increase in token usage. This highlights its ability to enhance outputs through targeted, dynamic refinements rather than full
7

---

Published as a conference paper at ICLR 2026
right--rihht wrong-owoong
85-100%
45-85K
wnongoright
right-owering
26%
5%
(1) Refinement Rate
Semantic Consistency Scores
Allgnment Scores
Figure 4: From left to right, the pie charts show: (1) the proportion of answers changed by PASR refinement, (2) the distribution of coherence scores reflecting how well the self-refinement builds upon the initial generation, and, and (3) the distribution of alignment scores measuring the consistency between the refinement process and the final answer. For (2) and (3), each segment represents the proportion of examples falling within specific score range (e.g., (0-0.45), (0.45-0.85), [0.85-1.0]).
rewrites, making it cost-efficient refinement method. Specifically, on the Qwen2.5-7B, PASR yields a 4.8 absolute performance improvement with only an 8.4% increase in token consumption compared to standard generation.
Additionally, while PASR and PTR achieve comparable performance on Qwen2.5-7B. PTR incurs significantly higher token costs. The performance gain of PTR mainly stems from the use of high- quality, answer-level refinement data. However, the effectiveness this data diminishes considerably Qwen3-8B. However, PTR regenerates entire answers at each refinement step, resulting in substantial token overhead.
3.3 DOES PASR GENUINELY EXHIBIT PROACTIVE REFINEMENT CAPABILITIES DURING GENERATION?
We investigate whether PASR performs proactive refinement during the generation process rather than passively correcting outputs after completion. To validate this, conduct a quantitative analysis from three complementary perspectives: (1) whether PASR performs refinement at appropriate moments; (2) whether the refinement behavior modifies earlier reasoning steps or simply regenerates content; (3) whether these refinements contribute causally improving the final output quality. The prompts used in this subsection are shown Table and 8. The results are shown in the Figure
PASR autonomously determine when to refine. We randomly sample 384 questions, among which 267 are initially answered incorrectly by the base model. PASR does not refine all answers indiseriminately: instead, it selectively triggers refinement. Among the 267 incorrect answers, 235 are revised and corrected by PASR. While many originally correct answers nearly remain unchanged. This indicates that PASR is able to identify and act upon potentially flawed generations when refinement is necessary.
PASR shows high coherence between pre- and post-refinement outputs. We randomly sample to evaluate their semantic
300 answers and employ an independent LLM,, Qwen2.5-32BB-mm consistency before and after refinement. Each sample is scored multiple times within in [0, 1Jto ensure the reliability of the assessment. The results indicate that nearly 80% of samples received semantic consistency score excceding 0.9.
PASR's proactive self-refinement process contributes to the answer correctness. We further analyze the 300 samples mentioned above evaluate the alignment between the refinement process and the final answer. Over 85% of the samples achieved aligmment score above 0.9, indicating that refinement leads to the quality of outputs.
3.4 WHAT MAKES PASR EFFECTIVE?
Reinforcement learning enables the model to perform proactive self-refinement. In contrast, prompt-based or supervised signals are insufficiet to elicit proactive refinement capabilities. We explore whether proactive self-refinement can be induced via prompting. The results shown in
8

---

Published as a conference paper at ICLR 2026
Table 1. When the model is explicitly instructed to self-refine during generation via prompt design (PASR+prompU), we observe a consistent performance decline across all tasks, with an average decrease of 16.9 and 9.5 on two backbone models. It indicates that prompt-based guidance alone is insufficient to elicit the model's adaptive self-refinement capability.
Similarly, apply Instruction-followiee finetuning (PASR+IFT) to inject this capability. How- ever, the model shows limited generalization to unseen tasks. On the Qwen3-8B model, per- formance drops by 8.3 compared the base version. These results suggest that proactive self-refinement is not an innate capability and cannot be effectively acquired through SFT.
Comparison- rewards setting help the model learn to perform effective refinements. We use Qwen2.5-7B as the backbone and eval- uate the effectiveness of two alternative reward strategies. The first is Single-reference compar- ison (w/o multi-answer)) computes refinement rewards by comparing the refined output to sin- gle standard answer. The second is Refinement- triggered reward (w/o comparison), assigns coarse positive refinement reward whenever a refinement action is taken, regardless of its cessity or effectiveness, The results are shown in Table 2. This reward strategy offers several key advantages.
Table 2: PASR performance across datasets un- der different refinement reward signals. The comparison-based fine-grained reward better guides the model to learn adaptive and meaningful refinements.
Dataset
PASR
w/o multi-answer
w/o comparison
MMLU
75.0
71.0 (-4.0)
53.0 (-22.0)
Drop
79.6
76.7 (-2.9)
78.6 (-1.0)
Xsum
49.9
44.3 (-5.6)
31.9 (-18.0)
GSM8K
88.8
75.7 (-13.1)
86.0 (-2.8)
MATH
73.6
62.2 (-11.4)
62.2 (-11.4)
AIME24
10.0
10.0 (+0.0)
10.0 (+0.0)
ARC
86.6
83.9
82.9 (-3.))
GPQA
29.3
(-0.4)
(-19)
Wino
57.0
53.4 (-3.6)
65.3 (+8.3)
First, averaging over multiple standard answers reduces the variance introduced by the random- ness of LLM outputs. It provides more robust and stable learning signal for guiding meaning- ful refinements during training. This strategy enables the model to better recognize when refinement yields genuine improvement. Moreover, coarse-grained reward signals are easily exploited by the model, leading to unnecessary refinement in pursuit of high reward (i.e., reward hacking). In contrast, comparison-busee based signal avoids this by rewarding only measurable improvements, leading to more targeted and meaningful refinements.
CSQA
67.0
(-.9
(4.1))
AVG 61.7
57.5
(6.2
4 RELATED WORK
Prompt-basee based self-refinement Prior work on self-refinement typically follows a two-stage paradigm. The model first generates an initial response and then prompted to refine or improve it (Ganguli et al., 2023a). These methods have seen widespread use in complex reasoning tasks. including math (Weng et al., 2023; Wang et al., 2024) and code generation (Olausson et al., 2023b; 2024; 2023a). However, simply prompting model to refine its own output does not consistently yield better results, and there little evidence that prompting alone is sufficient for reliable eeff-improvement Huang al., 2024; Tyen et al., 2024). Success in these settings often relies on the availability of ground truth feedback or external supervision, such as explicit information about the error, its location, and an explanation of why it is wrong (Kim et al., 2023; Shinn et al., 2023). Unfortunately, such ane-ggained feedback is rarely accessible in practical applications (Gou et al., 2024; Pan et al., 2024). Therefore, some studies utilize stronger models or train auxiliary teacher models to evaluate outputs and provide feedback (Xie et al., 2025; Madaan et al., 2023; Uesato et al., 2023; Welleck et al., 2023). While effective, these approaches usually require task-specific annotations to train the feedback models, which significantly increases the cost and limits scalability across diverse tasks (Du et al., 2025).
Sine-tuning for self-refinement. Another line of work focuses on SFT using synthetic self-refinement data. In these settings, initial answers are generated by one model, while refined answers are produced by a stronger model or taken from oracle answers (Havrilla et al., 2024; Du et al., 2025; Han et al., 2024) (Xie al., 2025). The resulting pairs of "bad" to "good" answers are used to train models to imitate the refinement process. However, such methods suffer from either distributional mismatch, where the errors in training data do not reflect the mistakes the model makes during inference (Kang

---

Published as a conference paper at ICLR 2026
al., 2025), behavioral collapse, where the model learns narrow correction pattern that fails to generalize across tasks or domains (Kumar et al., 2025; Qu et al., 2024).
5 CONCLUSION
We propose PASR, novel method that enables large language models to proactively self-refine their responses during generation. PASR leverages an on on-policy reinforcement learning approach explore whether, when, and how perform refinements. We design fine-grained rewards to encourage effective refinements and penalize incorrect or unnecessary ones. Experiments show that PASR achieves strong balance between performance and efficiency. Moreover, even when trained only on general open-domain data, PASR achieves strong self-refinement across ten diverse tasks, demonstrating strong generalization not observed in previous work.
ACKNOWLEDGMENTS
This work was supported by Ant Group and the ECNU Academic Innovation Promotion Program for Excellent Doctoral Students (Grant No. YBNLTT2004-0077)
7 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal experimen- tation was involved. All datasets used were sourced in compliance with relevant usage guidelines, ensuring violation of privacy. We have taken care to avoid any biases or discriminatory out- comes our research process. No personally identifiable information was used, and experiments were conducted that could raise privacy security concerns. We are committed to maintaining transparency and integrity throughout the research process.
8 REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and datasets have been made publicly available in the github repository to facilitate replication and verification. The experimental setup, including training steps, model configurations, and hardware details, described in detail in the paper. Additionally, All datasets are publicly available, ensuring consistent and reproducible evaluation results. We believe these measures will enable other researchers to reproduce our work and further advance the field.
REFERENCES
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes 11m better reasoner, 2024. URL httpp:://aaai http:////a//voooo/a//////// org/abs/231 20688. 20689.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024. URL
https://arre httpxivvorr//bb//4005./ 06963.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevian mett/forum?id-kuiixxtteee
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL abs/2110. 14168.
Alan Dao and Dinh Bach Vu. Alphamaze: Enhancing large language models' spatial intelligence via grpo. arXiv preprint arXiv:2502. 14669, 2025.
John Dewey. Experience and education. The educational forum, volume 50, pp. 241-252. Taylor & Francis, 1986.
10

---

Published as a conference paper at ICLR 2026
Chengyu Du, Jinyi Han, Yizhou Ying, Aili Chen, Qianyu He, Haokun Zhao, Sirui Xia, Haoran Guo, Jiaqing Liang, Zulong Chen, et al. Think thrice before you act: Progressive thought refinement in large language models. In The Twelfth International Conference on Learning Representations, 2025.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2368-222 2378,, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/v1/N19 1246. URL https:
Zeyu Gan, Yun Liao, and Yong Liu. Rethinking external slow-thinking: From snowball errors to probability of correct reasoning, 2025. URL https: /ttp///axxvvv//////////ee hwtp:/iwww..ogg///////////// 15602.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilè Lukošiūte, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.074559 2023a.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hattield-Dodds Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel Bowman, and Jared Kaplan. The capacity for moral self-correction in large language models, 2023b. URL https: // /ttp//xxivvvoo/////////// www.:oogg/bb..... 233//22..2222... 2302 07459.
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujuu Yang. Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference Learning Representations, 2024. URL forum?id=Sxx3Boxjek.
net/
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, and Yanghua Xiao. Small language model can self-correct. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 18162-18170, 2024.
Alex Havrilla, Sharath Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravynski. Eric Hambro, and Roberta Raileanu. Glore: when, where, and how to improve Ilm reasoning via global and local refinements. Proceedings of the 41st International Conference on Machine Learning, ICML'2 '24 JMLR.org, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https: /openreviie wet.eeeweeeeeeeeeeeeeeeeeeeeeeee
net net/forunt d
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifhh Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL http:///ppppeeeeeeeeeeeeererrrra wtt.://orrmmmii///////mmmmmssee
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference Learning Representations, 2024.

---

Published as a conference paper at ICLR 2026
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp- 3600-3612, Albuquerque. New Mexico, April 2025. Association for Computational Linguistics. URL https: / hatps//////laaholoooyoy wwg.orgg///// naacl -log. 183/.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Julius Kuhl and Jürgen Beckmann. Action control: From cognition to behavior. Springer Science & Business Media, 2012.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, al. Training language models to self-correct via reinforcement learning. In The Twelfth International Conference on Learming Representations. 2025.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedhack with AI feedback, 2024. URL http://pppnnrvieeneeeeeeeeeeeeee id=AAxI
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 233, Red Hook, NY, USA, 2023. Curran Associates Inc.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation. CoRR, abs/2306.09896, 2023a. URL https://doi org/10. 48550/ arvi 2306. 09896.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation. CoRR, abs/2306.09886, 2023b. URL .98966.
https://doi 100 48550
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Is self-repair silver bullet for code generation? In The Twelfth International Conference on Learning Representations, 2024.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484-506, 2024. doi: 10. 10.1162/tacl 006600 URL
org org/2024, 2024. 1.cl-1.277
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024, URL https: / htt::///ppeeeeeeeee//e net:/forumidddRRcppZaa
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https://arxils http:://axxiv.oogabb////// 15115.
12

---

Published as a conference paper at ICLR 2026
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381. URL https://doi. org 10t.//11111 1145/344488.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL https: //axxiv org/ httgg//////// absp/202. 03300.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp- 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational URL https: hwtp//olaannotyy.....o org wrg.oogg/////...o. EE/NN91114111
Linguisties. doi: 10.
Yongqi Tong, Dawei Li, Sizhe Wang, Yujia Wang, Fei Teng, and Jingbo Shang. Can LLMs learn from previous mistakes? investigating LLMs' errors boost for reasoning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 3065-3080, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 10.18655/11/20044acc-oom 169, URL https: /tttps:/aaclanteee hwwph/oooooot .aagy..rgg//ee 2cl 169/.
Srikumar(edsss
Gladys Tyen, Hassan Mansoor, Victor Carbune, Peter Chen, and Tony Mak. LLMs cannot find reasoning errors, but can correct them given the error location. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 13894-1390000 13908, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: N0.18653/v1/2022 1/665///1/2224....nnnnga.......
Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Yamamoto Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-based and outcome-based feedback, 2023. URL MND1kmmNy60.
net/forum?id
Peiyi Wang, Lei Li. Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. Thailand, August 2024.
Bangkok,
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning to critique more effective than learning to imitate, 2025. URL https::/arxiv..o org/ 250/ 17703.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. The Eleventh International Conference on Learning Representations, 2023. URL /tttp:://openrevieee net/
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2550-2575 Singapore, December 2023, Association for Computational Lin- guistics. doi: 10.18653/v1/2000 http:3///11222..nnnngnggeee madingg-eemll.1165 URL https: https::/aclaatttoll /awpaaaaaaaaaaaay....oo org/ org/2023. Sindings-emnlp 167/
Zhihui Xie, Jie chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching language models to critique via reinforcement learning. In ICLR 2025 Third Workshop Deep Learning for Code, 2025. URL
net hett/forumidd--UEqg2Z
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve llm-as-a- judge ability as general ability, 2025. URL http:////aaxxixoo////////// 11689.
13

---

Published as a conference paper at ICLR 2026
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024. URL
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar: Bootstrapping reasoning with reasoning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL hwtps: org org/abs. 05685.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENTAL DETAILS
A.1 IMPLEMENTATION DETAILS FOR PASR
Platform. All of our experiments are conducted on workstations equipped with eight NVIDIA A800 PCIe GPUs with 80GB memory, running Ubuntu 20.04.6 LTS and PyTorch 2.0.1. About the training cost, using Qwen2.5-7B as an example, we train PASR with the following setup: 2 GPUs for rollout generation, 1 GPU for policy updates, and 1 GPU for hosting the reference model server. Training for 3,000 steps takes approximately 8 hours in total.
Important Parameters of PASR. The PASR is implemented based on the open-source GitHub repository 8 The KL divergence penalty coefficient 3 is set to 0.04 to balance policy improvement and deviation from the reference policy. The clipping parameter is set to 0.2. For each group, 8 answers are generated, and the training batch size is set to 2.
Distributed training utilizes the DeepSpeed library with the Adam optimizer and learning rate of 1e-6. Gradient accumulation occurs over 4 steps, and with a per-GPU batch size of 2, the effective batch size is 8 NGPUs, where NgPU; denotes the number of GPUs.
Mixed-precision training with BF16 is enabled. Memory optimization employs ZeRO Stage 2, with optimizer state offloading to CPU. Key ZeRO configurations include allgather partitions, an allgather bucket size of 2e8, reduce scatter, and a reduce bucket size of 2e8. Contiguous gradients are enabled, communication overlap is disabled, and 16-bit weights are gathered during model saving. Training loss is logged every 5 steps.
Details on the Judge Model. During training, we employed Qwen2.5-32B-Insse Innsrructt as the judge model, which has been widely adopted for assessing answer correctness (Yu et al., 2025). To ensure reliable and objective evaluation, our prompt design explicitly incorporated three elements: the question, the ground truth, and the model-generated answer. The judge model was instructed to ground its judgment on the provided ground truth rather than on subjective impressions, thereby avoiding inconsistent criteria and yielding more stable evaluations than direct answer-only comparisons. The full evaluation prompts used in both training and testing are shown in Table 6.
Judge
To verify the trustworthiness of the judge model, we randomly sampled 50 evaluation cases from the test set and performed manual verification. Each case was independently reviewed by two human annotators, who compared the generated answer against the ground truth. We observed 91 % agreement rate between the judge model's assessments and human judgments, confirming that the judge model provides consistent and reliable scoring.
For deployment, the judge model runs on four A800 (80GB) GPUs with a batch size of 8, achieving an evaluation speed of approximately 43.27 tokens per second (about 2 seconds per batch).
Table 3: PASR vs. other baselines. Compared to the base model, PASR achieves an average performance improvement of 4.9% on Qwen2.5-14B.
Math
Reasoning
Knowledge Wino
Comp. Drop
Gene. MMLU
Sum. Xsum
Methods
Public
GSMBK
MATH AIME24
ARC GPQA
CSQA
AVG
Qwen2.5-14B
Vanilla
92.9
75.6
20.0
89.0
38.4
81.1
66.4
87.5
$7.0
60.5
66.8
(Madzan al., 2023)
N1PS'23
30.0
92.3
46.3
88.1
74.0
92.3
57.1
72.5
RefineeeeSii et al., 2003)
23
92.1
75.2"
20.0
"9..0
38.5
80.2
65.7
86.9
57.0
"57.5
66.2
PTR al., 2025)
ICLR'25
87.6
63.6
10.0
86.6
37.0
84,5
75.3
83.7
54.0
44.3
62.7
SCaRe(Kumar et al., 2025)
1CI.222
78.2
10.0
86.3
44.1
86.8
70.5
84.6
70.9
70.5
STaR(Zelikmmn et al., 2022)
NIPS
87.0
75.4
6.7
87.0
39.2
78.0
70,2
89.5
72.0
63.2
66.8
ISC(Han et al., 2024)
AAA124
88.1
64.0
23.3
77.9
35.2
71.2
62.9
83.7
75.0
46.2
62.8
PASRi-prempp) PFF
88.7
71.6
26.7
78.9
26.3
71.0
68.0
88.5
66.0
17.7
60.3
75.0
59.4
23.3
86.0
38.4
67. .
69.0
78.9
68.0
61.1
62.7
PASRi
78.0
30.0
88.8
45.1
86.0
78.3
89.9
53.2
15

---

Published as a conference paper at ICLR 2026
A.2 IMPLEMENTATION DETAILS FOR BASELINES
We use the LLaMA-Factory framework' to train all baseline methods. The key parameters are shown in the Table
Table 4: Important parameters for each baseline method
Method
Parameters
PTR
per per_device device train batch Dateh_size:
gradient accumulationn steps:
learning_ rate: 1.0 x 10-5
num tes__taan_eeeeee _eeecpeppss....
t__scheduler_tte tepp:: cosine
warmup_ ratio: 0.1
bf16: true
Dataset: Public GitHub
SCoRe
per jor__evice_itee _at___aiinaaahhhh_eeeeeeaa batch_ _isie:
gradient gradient_accumblne accumulation steps:
learning rate: rate: 1.0 x 10
num simm_trinn_noe train_epochss 2.0
Ir s.chhhullerr type: cosine
warmup_ration 0.1
bf16: true
Dataset: preference pairs form PTR experiment
STaR
per Leevice_trraie aecin_batch_sst size: I
gradient_acene acw.mulatuuuiinnnccmmmm steps:
learning_ rate: 1.0 x 10-5 num_train cum_train_epoche Ir ir_scheduler__ __tp:::yyyppe_y cosine
warmup_ratis ratio: 0.1
bf16: true
Dataset: alpaca_evoll (instruct_7O((ilttet generated pairs))
ISC
per jor_deviee__te tuaiirrinn_i__ batch_ size: 1
aatirmlataccumultiinonnn. _ttppttiioosssss:et_e learning_rad _ate: num_train traii_eppochs: 2.0
10-5
Ir s_schedullr__ye typp: cosine
warmup_raid ratio: 0.1 bf16: true
Dataset: alpaca_evoll instruct__700
RISE
per per_device sevicc__train batch_ size: 1 seafient_accumulatt
steps:
learning_ rate: 1.0 x 10-5
num num_train_ tur_ttrann___eee hri__trri_eeeesss222111 2.0
Ir_scheduler _tpp:: cosine
warmup_ ratio: 0.1
bf True
Dataset: alpaca_cvol wwwoooottooitttttteeeoeeee 70k
PIST(+IFT)
per per_device lel_ddvveee_riiee train batch size:
gradient_accumullat accumulation steps:
learning_raie rate: 1.0 10
num www_ttrinn_eee sokin_epochs: 2.0
Ir rrshcheuuler _tppuullerttye::::e cosine
warmup_ ratio: 0.1
bf16: True Dataset: good refinement paths generated during PASR training
16

---

Published as a conference paper at ICLR 2026
Aligmee
impraeee
05-
033
27.0%
32.0
04
04-
0.3
27.4%
0.3-
02-
9.7%
7.74
0.0
bufirement
methoratical
Reflinemeet behavtor
(4) Rufiremmnt bet sador ine laber
reasordng
stage
generation
stage: generation
Figure 5: The frequency distribution the four refinement types in PASR.
B FURTHER ANALYSIS
B. FURTHER PERFORMANCE ANALYSIS OF PASR
As shown in Table 1, PASR improves the average performance of Qwen2.5-7B and Qwen3-8B by 4.8 and 8.2 points, respectively, compared to standard generation across benchmarks. We further evaluate PASR on Qwen2.5-14B and DeepSeck-PR DeepSeckRR-Dissiil. R-i---iiill-DDDE Distill-Llama-8B the results are reported in Table
Notably, PASR yields larger gains on models with stronger reasoning capabilities, For example, boosts the average accuracy of Qwen3-8B by 8.2 points, whereas the improvement on the larger Qwen2.5-14B 14B model is 4.9 points. This pattern suggests that PASR's effectiveness does not simply scale with model size. Instead, it demonstrates PASR's s intrinsic reasoning ability to generalize across different model families, delivering consistent improvements regardless of model configuration.
B.2 REFINEMENT BEHAVIOR ANALYSIS OF PASR
This experiment aims investigate how PASR autonomously refines its outputs during generation, including the types of refinement behaviors it exhibits and the factors that limit its effectiveness. Specifically, we analyze both qualitative examples and quantitative statistics of refinement types, and examine failure cases to understand the model's strengths and inherent constraints.
Refinement behavior examples of PASR. In the Section 2, we define four intended refinement behaviors of PASR, including Error Correction, Information Complement, Solution Improvement, and Task Alignment. While these four categories guide the design of the system prompt during training, PASR is not explicitly instructed follow specific type when solving tasks. Instead, the model autonomously decides the appropriate refinement behavior based on the task context. We provide concrete example for each the four refinement types to clearly demonstrate how PASR operates. Examples are shown in Table 5.
Statistical analysis of the four refinement types. We sample 2,678 refinement outputs from PASR's training process and used GPT-4o to classify the type of refinement performed. The prompt used shown in Table 6 and the results are shown in Figure 5(1). We find that PASR mainly performs Task Aligument and Information Complement. This pattern is related to the training data, which consists mostly of general instruction-tuni tunnnn corpora. result, the model tends ensure task compliance and complete missing information during generation, rather than focus structural changes or post-hoe error correction.
Morever, to validate PASR's refinement behavior in reasoning tasks, we randomly sampled 300 answers from the math benchmark and analyzed the distribution the four refinement types. The results are show in Figure 5 (2). These results indicate that PASR's behavior distribution varies across task types: Error Correction and Information Complement are predominant mathematical reasoning tasks, whereas Task Alignment and Information Complement are more common in open-ended tasks.
We further investigated how PASR's 's refinement behavior evolves during the generation process by dividing each answer into two halves. The results show in Figure 5 (3) and (4). They show that Error Correction and Information Complement remain the most frequent behaviors throughout the generation. Notably, Information Complement becomes more prominent in the later stage, suggesting
17

---

Published as a conference paper at ICLR 2026
that PASR tends to add supplementary information toward the end of reasoning enhance output quality.
Error Case Analysis. We conducted analysis of PASR's ' failure cases to better understand its limitations. As discussed Section 3.2.2. Among 267 questions initially answered incorrectly, PASR successfully corrected 235 through refinement, while 32 questions remained incorrect. Manual inspection these 32 cases revealed two main reasons for failure. First, questions beyond knowledge boundaries. These involved the question outside the model's existing knowledge, and self-refinement cannot introduce new information, similar to the limitations of human self-correction. This represents an inherent limitation of current models rather than shortcoming of PASR, and identifying such cases can guide future targeted improvements. Second, limited metacognitive ability of existing L.Mss. The model sometimes fails to accurately recognize or locate own errors. This restricts the refinement process, causing it to only partially address or overlook core mistakes.
Summary. PASR's refinement behavior is inherently adaptive: the model chooses how and when to refine outputs based on the specific task, demonstrating the advantage of autonomous, context-aware refinement. However, self-refinement has intrinsic limitations. cannot fully correct errors that stem from gaps in the model's knowledge limitations in its reasoning capabilities.
B.3 DISCUSSION ON How PASR MITIGATES OVER-THINKING.
Over-thinking in LLMs often results from redundant or unfocused thought processes, leading to unnecessarily long outputs. PASR alleviates this issue by enabling the model perform more geted and effective self-refinement, which naturally produces shorter and more purposeful responses. Empirical results support this effect. As shown in Figure 3, PASR generates consistently shorter outputs compared to other self-refinement methods. This behavior is further encouraged by our reward function, which reinforces effective refinements, penalizes detrimental ones, and discour- ages unnecessary refinement actions. Notably, even without explicitly penalizing output length, PASR achieves more concise reasoning by focusing on meaningful refinements, demonstrating that enhancing the quality and efficiency of self-refinement can reduce over-thinking.
Four Refinement Types of PASR
Casel: Exmaple the Error Correction behavior in PASR //
Question:
In the following sentences, what are the commonly confused words that can be corrected using Latex and what is the difference between them?
Answer:
<irsn I need to identify the commonly confused words and then I'll explain the differences between them using LaTeX. After that, I'll provide multiple examples for each word to show their correct usage. <refine>Oops! forgot to specify the words in LaTeX format. Let me fix that.</refine>
( more reasoning tokens omitted brevity)
</think>
<answer>-Thes explanations and examples illustrate the differences between the commonly confused words and demonstrate their correct usage. The words and their differences clearly indicated using LaTeX, and multiple examples prove the model's understanding and application of the correct usage in context. </answer>
** Case2: Exmaple of the lormation Complement beha behavior in PASR #/ Question:
18

---

Published as a conference paper at ICLR 2026
Can you write Python script that scrapes data from PubMed related the relationship between the endocannabinoid system and the immune system?. Answer:
<think>
To complete this task, I will need to use Python's requests and BeautifulSoup libraries to scrape the data from PubMed.
(... more reasoning tokens omitted for brevity
Finally, need to add the necessary enror handling to make sure that the script can handle any errors that might occur during the scraping and parsing process.
<refine>I should also add some code to parse the abstract of each article to extract any relevant information related to the endocannabinoid system and the immune system. This will help to further analyze the current state of research this topic. </refine> </think>
<answer>
Here is the Python code that have written complete the task: (...))
Any articles that were not published within the last five years are filtered out. The data is then stored in CSV file using the csv library.
</answer>
/*Case 3: Exmaple the Solution Improvement beha behavior in PASR #/
Question:
Can you provide an overview the labor movement using a historical timeline?
Answer:
<think>
My initial thought that we can start by researching the key events the labor movement's history.
(... more reasoning tokens omitted for brevity) ) To make more engaging, should include hyperlinks for each event that would lead brief description.
<ugfine-Suggestion include hyperlinks is a good idea. This would make the timeline more interactive. However, we should also consider adding images embedded videos relevant the events for a more visual representation. Let's refine this in our approach. </refine>
</think>
<answer>
(...)
</answer>
19

---

Published as a conference paper at ICLR 2026
C*Cas::: Exmaple of the Task Alignment beha behavior in PASR #/
Question:
Utilizing advanced sentiment analysis in MATLAB, can you determine the nuanced emotional tone conveyed in this quote and provide further insight into Atticus Finch's complex attitude towards the Chief Justice and the United States Constitution?
Answer:
<think>
(... more reasoning tokens omitted for brevity Additionally, should consider the historical and political context of the quote to better understand Atticus Finch's complex attitude towards the Chief Justice and the United States Constitution
<refine>The question asks us to delve deeper into the historical and political context of the novel, particularly in relation to the societal and legal injustices prevalent during that time period. How does the quote reflect these issues, and what can it tell us about the societal norms and legal system of the time? </refine>
</think>
<answer>
(...)
</answer>
All Prompts Used PASR
Prompt Template for PASR */ System:
You are helpful assistant with self-refinement capability. After the user asks a question, you first think carefully and then give the answer.
The thinking process and answer should be enclosed within <think </think> and <answer> </answer> tags respectively. Note that you can only use once these four tags. In the <think> and </think> tag, follow these rules: Start with an initial thought process on how to approach the question.
when you determine that additional clarification, detail, improved reasoning is nec- essary, insert <refine> </refine> tag and then specify what needs to be reconsidered or improved. You Continue to advance your reasoning after each refinement until you feel there is no more room for improvement.
both tags multiple times.
This is how your full response should be structured:
<think>Here is your thinking process, when you think you need to reflect, insert <refine>your refinement</rfinee. Repeat the iterative process as many times necessary before moving
20

---

Published as a conference paper at ICLR 2026
the final auswer/hhink--nnsweelleer is answer the end of the thinking pro- cess.</answer>
1* Prompt Template for ASR evaluation */
You judger, you will judge the correctness of the answer to the question. Below is question, a ground truth answer, and an answer generated by an Al assistant, please rate the Al assistant's 's answers according the question scale from 0 to 1. Your output is just number in the range from 0 to 1.
### Question:
[Question]
#### Ground Truth:
(Ground Truth)
### Answer:
(Answer)
** Prompt for Evaluating the Refinement Behavior of PASR $/
You are a judge of content refinements. will give you sentence that may contain one or more <refinez...</efineee sections. Each <refine> section shows refinement made by model.
Step Analyze each <refine> section one by one.
For each <refine>, classify it into exactly one of the following types:
Error Correction: fixes factual, logical, or calculation mistakes.
2. Information Complement: adds missing important details to make content complete and correct.
Solution Improvement: improves the effectiveness or efficiency of the solution.
4. Task Alignment: adjusts the content to better match the task goal or user intent.
Output only the category name for each <refine>, order, one per line.
Step 2: After all refinements are labeled, create a JSON summary of counts for each type.
Example format:
"Error Correction":
"Information Complement": 0,
"Solution Improvement": O,
"Task Alignment";
Rules:
Do not add any explanations, only output the category names first, then the final ISON summary.
Use the exact category names above.
Here sentence:
21

---

Published as a conference paper at ICLR 2026
/ Evaluation Prompt Template for Summary Questions */
Now, want to test an Al assistant's ' ability to summary. Below is a text (Question), a ground truth summary (Ground Truth Answer), and an answer (Answer) generated by an Al assistant. Please rate the AI assistant's answers according to the ground truth answer. Please score answers according to how relevant they are to the text and ground truth summary. Your output from 0 to 1,which 0 is not similar all, is basically error free. ### Question:
Ground Truth:Ground Truth Answer.Answer
/* Evaluation Prompt Template for Multiple-Choice Questions */
Now, I want to test an Al assistant's ability to answer questions. Below a multi-choice question, ground truth answer(one of the option), and answer generated Please rate the Al assistant's answers according to the question and the ground truth answer. you think the answer is correct, your output I; otherwise, your output is 0. Your output just 0 or 1.
Al assistant.
### Question:
Ground Trouh:Ground Truth
Answer:Answer
/ Evaluation Prompt Template Open Questions */
Now, want to test an Al assistant's ability answer questions. Below is a open question, ground truth answer, and an answer generated by an Al assistant. Please rate the Al assistant's answers according to the ground truth answer. If you think the answer is correct, your output I; otherwise, your output O. Your output is just O or 1. Question:Questee Ground Truth:Ground Truth Answer:Answer 1* Prompt Template for Refinement with Oracle (Math Questions) */
There might be an error in the solution above because of lack of understanding of the question. Please correct the error, if any, and rewrite the solution. Only output the final solution! At the end of the Solution, when you give your final answer, write it in the form Final Answer: The final answer is \box(answer). hope it is correct. ### previous solution:Initial answer
1 Prompt Template for Refinement without Oracle (Open Questions) */ There an error in the previous solution. Please review each step to identify the mistake, and then provide a corrected version of the solution. ### previous solution:Initial answer
** Prompt Template for Refinement without Oracle */
Please review each step of the previous solution to identify any potential errors. If you find any issues. provide revised and corrected version of the solution. there are issues. simply respond with: believe the above solution is correct. ### previous IouutionIIiiiaal answer
22

---

Published as a conference paper at ICLR 2026
/ Standard Prompt for MMLU */
Here is multiple-choice question, which from a dataset tests knowledge across 57 diverse fields such as elementary mathematics, history, computer science, and law, please think step by step and give me your final answer.
/ Standard Prompt for Drop */
Here is passage and question, which requires discrete reasoning over the provided text. Please think step by step and give me your final answer.
/ Standard Prompt for Xsum */
Here passage. please summarize this passage.
/* Standard Prompt Template for Math (GSM8K, MATH, AIME24) %/
Here is problem. please think step by step and give me your final answer.
/ Standard Prompt for ARC
Here is a multiple-choce question, which from collection of questions for the science exam. Please think step by step and give me your final answer.
1* Standard Prompt for Wino */
Here is a question provides two options. Please think step by step and select the correct answer based on the semantics of the sentence.
/* Standard Prompt for Commonsense@A */
Here multiple-choice about commonsense. Please think step by step and give me your final answer.
Prompt for Evaluating the Reasonableness of the Refinement Process
# Role
You are an Al Analyzer specializing in assessing the quality of refinement thinking.
Task
Your task is to evaluate the "reasonableness" of the refinement part within given response. This response typically contains two parts: an initial thought or response (pre-refinement), and part where the user reflects that initial thought (post-refinement). Definition of "Reasonableness"
"Reasonableness" here has specific meaning: it measures the coherence and consistency between the pre-refinemett and post-refinement thought processes.
You need to determine:
1. Is the refinement based the preceding thought content?
2. Does the refinement process logically follow from the previous thinking? Or, if the refinement leads shift in perspective, this shift explained or internally logical and understandable?
23

---

Published as a conference paper at ICLR 2026
3. Does the conclusion or state after refinement form an understandable and coherent thought trajectory with the pre-refinement state?
Crucially: You are not evaluating the depth of the refinement itself, nor the correctness of the final answer. You are evaluating only whether the act of refinement is coherent and consistent with the preceding thought content.
# Evaluation Criteria & Score
Please provide a floating-pone point score between 0.0 and 1.0 based on the following criteria: 0.0: Completely unreasonable. The refinement is entirely unrelated to the previous thinking, or contradicts it without any explanation. The thought process is broken or disconnected. * 0.5: Partially reasonable. The refinement has some connection the previous thinking, but the link is weak, the logical chain is unclear, or shift in perspective seems somewhat abrupt but has a faintly traceable thread.
1.0: Highly reasonable. The refinement is clearly built upon the previous thinking, the logic is coherent, and even if perspectives shift, the reasons and process are clear, demonstrating high consistency in the thought trajectory.
Output Requirements
Strietly output only single number, which must be a floating-pon number between 0.0 and 1.0.
* Do not include any explanations, characters.
justifications, explanations,jssiiicattioss descriptions, units, or any other extra
# Response Text to Evaluate
Box 8: Prompt for Evaluating the Consistency between the Refinement and the Final Answer
# Role
You are an Analyzer specializing in evaluating thought coherence.
# Task
Your task is to evaluate the consistency between a given "Thought Process" (which may include refinement) and the final "Answer".
# Definition of "Consistency"
"Consistency" here measures: The degree which the final answer is direct, relevant, and logical product of the thought process.
You need to determine:
Does the final answer directly address or resolve the problems, dilemmas, or goals
explored in the thought process?
the final answer logically aligned with the thought process, including insights or conclusions derived from refinement?
3. Are the key information, reasoning steps, or refinements from the thought process reflected or applied in the final answer?
24

---

Published as a conference paper at ICLR 2026
You are not evaluating the quality of the thought process itself, nor the correctness or merit of the answer itself. You are evaluating only the degree of relevance and logical connection between the thought process and its final answer. # Evaluation Criteria & Score
Focus:
Please provide a floating-poin score between 0.0 and 1.0 based on the following criteria: * 0.0: Completely inconsisten//rreleaant The final answer has little relation the thought process, appears out of nowhere, or completely ignores the reasoning path.
& 0.5: Partially consisten//relevatt. The final answer has some connection to the thought process, but might only address parts of it, the logical link might be weak, the answer, while related, doesn't seem like the most direct conclusion from the process.
* 1.0: Highly consistent/relevant. The final answer clearly, directly, and logically stems from the provided thought process, serving as its definite conclusion or solution.
# Output Requirements
Strictly output only single number, which must be a floating-poin number between 0.0 and 1.0.
Do not include any explanations,jusifiattiinn text descriptions, units, or any other extra characters.
Response Text to Evaluate
think </think> is thinking process, <answer> </answer> is final answer.
C LLM USAGE
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, we used LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking. and enhancing the overall flow of the text.
It is important to note that the I.I.M was not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuseript, including any text generated or polished by the LLM. We have ensured that the ---eerrrteed text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
25


Reviews:
[{"review_id": "ykGVimnysm", "reviewer": "Reviewer_RVyM", "summary": "The paper introduces ProActive Self-Refinement (PASR), a method that allows large language models to refine their outputs dynamically during generation rather than through fixed, reactive iterations. PASR enables the model to decide whether, when, and how to refine based on its internal state and evolving context, instead of regenerating entire responses", "questions": "1. It would be good to mention what data is PASR trained on?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents strong results using PASR in comparison to other self-refinement methods. The paper compares PASR with 8 other self-refinement baselines.\n2. PASR is token efficient. It does effective refinement using less number of tokens compared to many baselines.\n3. The paper is well written and provides a clear explanation of all the rewards used. It also presents comprehensive ablation study of PASR.", "weaknesses": "1. The paper could also show ablations on different rewards used to understand which ones are most effective and have the most impact on downstream accuracies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fJvPRFbEEv", "reviewer": "Reviewer_3Zjb", "summary": "Current language models often benefit from self-refinement, where they are reflect upon their reasoning traces to come up with a more accurate final answer. The authors argue that this refinement should not come after each explicit reasoning chain, but instead be dynamic and inserted into the reasoning steps, as humans do. This paper proposes PASR, a method that enables LLMs to refine their reasoning during their generation process. Specifically, PASR requires employing reinforcement learning (via GRPO) with rollouts which are encouraged, via a reward function, to include specific tags which correspond with thinking and refinement. Assessed on Qwen models across a variety of datasets, PASR improves over base models and often is the strongest method, even compared to a large number of baselines.", "questions": "Considering the RL training and tagging pipeline, is PASR actually cost-efficient compared to zero-shot or post-hoc self-refinement methods? Some rough compute-hours / GPU specs would help", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A large number of datasets covering diverse reasoning capabilities are explored\n* The authors evaluate against a large set of baselines, making their results much stronger and contextualized\n* The method is written relatively cleanly and explained well. In particular, the reward design section is particularly informative, as it details and justifies each component of the multi-dimensional reward well. The intuition is clear and helpful here.\n* The proposed method consistently outperforms base models across different tasks. Some of the gains on datasets like MATH are significant.\n* The authors perform extra analysis seeing when refinement happens and if it helps with the reasoning, which is helpful. Section 3.4.2 is also well-written.\n* The proposed method generalizes well to tasks that it wasn't explicitly trained on, a great sign and pointing to robustness.", "weaknesses": "* Only Qwen models of similar sizes are tested in this work. It would be important to show that the findings here generalize to other models, especially in light of recent discussion on Qwen's SFT making it perform much different than other base models at inference time. It would also be nice to have a scaling comparison - how do these results change on larger/smaller models of the same architecture? It's unclear if rollouts can be obtained in a zero-shot way with other reasoning models while following the formatting guidelines (as the 0 step for RL)\n* The main issue is that this method requires training a model, while other comparable strategies like Self-Refine emerged as simple, training-free ways to boost performance on outputs. It is unclear whether the performance gains across tasks is worth the high-cost of 1) training a large language model and 2) performing an RL pipeline which requires time to cycle through data collection and scoring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EzW14sbScN", "reviewer": "Reviewer_n7rz", "summary": "This paper introduces ProActive Self-Refinement (PASR), a reinforcement learning framework that enables large language models to refine their outputs during generation instead of after completion. PASR formulates the process as a Markov Decision Process, allowing the model to decide whether, when, and how to refine based on context. Using a GRPO method and a multi-part reward (format, accuracy, and refinement), PASR improves both efficiency and reasoning quality.", "questions": "See weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of moving from reactive post-hoc correction to proactive in-process self-refinement is novel.\n2. MDP formulation and GRPO-based RL training are clearly presented. \n3. Effective reward scheme design: combines structure, correctness, and refinement quality to balance precision and efficiency.\n4. Strong results are shown across diverse tasks and baselines (Self-Refine, PTR, SCoRe, RISE).", "weaknesses": "1. The accuracy and refinement rewards depend on another LLM for scoring, which introduces potential bias and circular evaluation concerns (especially if the same family of models is used for training and evaluation).\n2. Only schematic examples (e.g., the logic puzzle in Figure 1) are shown. More real examples illustrating PASR’s step-by-step refinement and error correction would improve interpretability.\n3. RL overhead compared to SFT methods is not analyzed.\n4. The paper focuses mainly on reward variants without isolating other factors (e.g., tag format, PPO choice).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4hn8Hu1hzV", "reviewer": "Reviewer_zxHQ", "summary": "This paper introduces ProActive Self-Refinement (PASR) as a novel method for enabling Large Language Models (LLMs) to refine their outputs during the generation process, rather than as a post-hoc step. \n\nThe authors formalize this as a MDP and use RL to train models to decide whether, when, and how to refine their reasoning trace. \n\nKey contributions include:\n* A formulation of proactive self-refinement and trained the model with on-policy RL with GRPO.\n* The design of a fine-grained, comparison-based reward function that encourages meaningful refinements while penalizing unnecessary or harmful ones.\n* Extensive experiments on 10 diverse tasks using Qwen models, demonstrating that PASR improves accuracy and problem-solving performance.", "questions": "* Question about the token consumption. How does the 41.6% calculated. What's the setup to achieve this.\n\n* The performance is good but not outstanding, especially when compared with self-refine (row 2 in Table 1). In what types of scenarios does the additional complexity of RL training provides additional advantage. \n\n* It will be good to have more analysis on the pattern of timing of refinement: The model learns when to trigger a refinement. Does the model tend to refine early in its reasoning process to correct its trajectory, or does it make more refinements later once it has generated more context? Understanding these patterns could provide valuable insights into the model's learned metacognitive abilities.\n\n* nit: Page5, Ln 238, duplicated \"Consequently\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The core idea of shifting self-refinement from a reactive, post-hoc process to a proactive, in-process one is a novel and important conceptual leap.\n* The choice of formulating proactive refinement as a Markov Decision Process and using RL is appropriate for the task of learning a sequential decision-making policy.\n* The hybrid reward design, which combines rule-based format rewards with model-based accuracy and refinement rewards, is thoughtful and comprehensive.\n* The evaluation is extensive, spanning 10 different datasets that cover math, reasoning, knowledge, and generation.", "weaknesses": "* Confusing claim about the token consumption: \n  * In page 1, Lines 023-024: the author mentioned \"PASR reduces average token consumption by 41.6%\"\n  * In Page 8, LIne 384 - 386, the author mentioned \"with only an 8.4% increase in token consumption compared to standard generation.\"\n  * It would be good to have a clear claim and explanation about the token consumption. So far, the claim of reduce by 41.6% is a claim not well supported.\n\n* The analysis in the Fig. 6 reveals that the model predominantly learns \"Task Alignment\" and \"Information Complement\" behaviors, with \"Error Correction\" being the least frequent type. The authors attribute this to the nature of the training data, but it may point to a limitation in the model's ability to identify and fix deep logical or factual fallacies. \"Text Alignment\" is a fragile section that there is concern that the approach is simply learning to hack the reward model by overfitting to its stylistic preference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)