Improving Meta Introspection of Small LLMs by Learning Self-Reflection from Self-Generated Data
Authors: Jiaqi Li, Xinyi Dong, Yang Liu, Zhizhuo Yang, Quansen Wang, Xiaobo Wang, Song-Chun Zhu, Zixia Jia, Zilong Zheng
Accepted at: acl25

ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection

Jiaqi Li1, Xinyi Dong2, Yang Liu1, Zhizhuo Yang2, Quansen Wang1,2, Xiaobo Wang1, Songchun Zhu1,2, Zixia Jia1,2, Zilong Zheng1,2
1State Key Laboratory of General Artificial Intelligence, BIGAI
2Peking University
{lijiaqi, jiazixia, zlzheng}@bigai.ai

Abstract
We present a novel pipeline ReflectEvo to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large scale, comprehensive self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs’ reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis on the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.

1 Introduction
Self-reflection involves meditating on, examining, and evaluating one’s behaviors, thoughts, motivations, and desires (Atkins and Murphy, 1993; Von Wright, 1992; Denton, 2011). Typically, it inspects the reasoning process leading to the current solution, identifies errors in each step, generates critiques on the causes of the failure, and offers advice for refining the solution to improve the problem-solving performance of Large Language Models (LLMs) (Welleck et al., 2022; Ferraz et al., 2024; Li et al., 2024a; Tong Wu, 2024). Unlike the paradigm of learning directly from the reasoning process and final answer, we refer to it as the process of human-like meta introspection, which explicitly generates self-reflection, providing textual differentiation and gradients as clear critiques and guidance on what to learn and how to improve based on the current state.

Recent research has demonstrated that LLMs can self-improve through their intrinsic capability of self-reflection (Huang et al., 2022; Renze and Guven, 2024; Guo et al., 2025; Wang et al., 2024b). However, conventional approaches rely closely on LLMs with large model sizes or supervision distilled from a superior model. In this study, we challenge whether the self-reflection capability of SLMs can be learned effectively from reflection data. However, it usually requires high-cost on fine-grained human annotation to acquire high-quality data for fine-tuning and is impractical to scale. Therefore, we are also curious whether it is possible to effectively utilize both high- and low-quality self-generated data from weaker models for reflection learning. With this in mind, we aim to investigate the effectiveness of reflection learning via self-training (Luong et al., 2024; Qu et al., 2024; Pang et al., 2023; Tang et al., 2024) and further validate that the improvement of self-reflection can further strengthen LLM’s inherent reasoning capabilities across various methods and tasks with more interpretability and generalization. We believe that this paradigm can act as a plug-and-play enhancement for various reasoning methods, which emulates human learning through a slower and deeper thought process that iteratively and ultimately derives self-evolution (Li et al., 2023; He et al., 2024; Li et al., 2024b; Tang et al., 2024, 2023).

Therefore, in this paper, we propose a novel pipeline ReflectEvo (Sec. 2), to automatically generate self-reflection data and leverage self-training to enhance LLM’s reflection capability. To the best of our knowledge, we are the first to demonstrate the potential of meta introspection of LLMs that are asked to explicitly generate reflection as an in-
termediate step-by-step process supervision rather than directly mapping an initial solution to a revised solution.

Building on this pipeline, we curate a large-scale, diverse, and unsupervised reflection learning dataset ReflectEvo-460k containing 460k reflection samples derived from 17 source datasets spanning 10 tasks and domains. We explore the diversity of reflection instructions and bootstrap multiple comparative reflections conditioned on the same question and initial solution. Based on the data, we develop reflection learning (Sec. 3) to further improve the self-reflection and self-correction capabilities of LLMs.

The evaluation results validate the effectiveness of reflection learning in boosting the reasoning of weak models. It shows significant improvements on Llama-3, exceeding the original base model by 10% on average tasks and outperforming its strongest counterpart with model size ×8. We conduct a deeper analysis of the self-generated reflection data including various error types identified from the reflection and observe their gains on corrected answers.

In summary, our main contributions are:
• Novel Pipeline for Self-Reflection Generation: We propose ReflectEvo for automatic self-reflection generation and curation, which is the first to explore meta introspection of SLMs.
• Large-Scale and Diverse Self-generated Reflection Dataset: We curate a comprehensive reflection training set ReflectEvo-460K from multiple data sources and tasks including various reflection instructions and comparative samples.
• Learning Reflection Via Self-training: We develop four settings of reflection learning methods to effectively improve self-reflection and self-correction based on SFT and DPO, which significantly boost the reasoning abilities of SLMs as well as surpassing their stronger counterparts.

2 The ReflectEvo Generation Scheme

In this section, we introduce the end-to-end pipeline ReflectEvo for collecting self-generated reflections as the training data for Sec. 3, leveraging the inherent ability of SLMs (see Fig. 1).

2.1 Problem Definition and Preliminary

Given a question \( q \) and its ground truth answer \( a^* \), the answer of the LLM after reasoning is denoted as \( a \) followed by its corresponding verbal feedback \( f \) from the environment, where \( f \) represents the evaluation function that assesses whether an answer is correct or incorrect by comparing it to the reference answer \( a^* \). The self-reflection \( r \) of an LLM explicitly locates and analyzes errors in \( a \) and makes further plans to mitigate the errors. Based on \( r \) and the context provided in the previous stage, the LLM is then asked to revise its original answer \( a \) to obtain \( \hat{a} \) and solve \( q \) as correctly as possible.

2.2 Reflection Generation

Step 1: Collection of instruction pool To enhance the effectiveness and quality of the generated reflections \( r \), we design instructions that target three key stages of reflection and correction, as
<table>
  <tr>
    <th>Feature</th>
    <th>Logical Reasoning</th>
    <th>Mathematics</th>
    <th>Coding</th>
    <th>Commercial QA</th>
    <th>Contract Free QA</th>
    <th>Reading Comprehension</th>
    <th>Commonsense Reasoning</th>
    <th>Social Reasoning</th>
    <th>Causal Reasoning</th>
    <th>Physics Reasoning</th>
    <th>Total</th>
  </tr>
  <tr>
    <td># Reflection training samples</td>
    <td>253,405</td>
    <td>92,967</td>
    <td>9,125</td>
    <td>19,399</td>
    <td>3,624</td>
    <td>32,135</td>
    <td>22,044</td>
    <td>19,175</td>
    <td>8,757</td>
    <td>1,166</td>
    <td>461,799</td>
  </tr>
  <tr>
    <td># Q&A-Reflection samples</td>
    <td>164,746</td>
    <td>106,434</td>
    <td>7,520</td>
    <td>17,448</td>
    <td>2,940</td>
    <td>20,760</td>
    <td>10,012</td>
    <td>11,404</td>
    <td>3,212</td>
    <td>468</td>
    <td>344,944</td>
  </tr>
  <tr>
    <td># Correct after reflection</td>
    <td>16,60</td>
    <td>10,77</td>
    <td>15,31</td>
    <td>4,26</td>
    <td>8,66</td>
    <td>12,39</td>
    <td>33,88</td>
    <td>19,99</td>
    <td>39,98</td>
    <td>38,46</td>
    <td>113,93</td>
  </tr>
  <tr>
    <td>Avg. question length</td>
    <td>140</td>
    <td>77</td>
    <td>99</td>
    <td>335</td>
    <td>200</td>
    <td>148</td>
    <td>219</td>
    <td>52</td>
    <td>118</td>
    <td>39</td>
    <td>130</td>
  </tr>
  <tr>
    <td>Avg. answer length (turn 1)</td>
    <td>131</td>
    <td>267</td>
    <td>187</td>
    <td>91</td>
    <td>118</td>
    <td>82</td>
    <td>158</td>
    <td>110</td>
    <td>138</td>
    <td>112</td>
    <td>888</td>
  </tr>
  <tr>
    <td>Avg. answer length (turn 2)</td>
    <td>163</td>
    <td>299</td>
    <td>202</td>
    <td>119</td>
    <td>135</td>
    <td>116</td>
    <td>159</td>
    <td>133</td>
    <td>130</td>
    <td>124</td>
    <td>237</td>
  </tr>
  <tr>
    <td>Avg. reflection length</td>
    <td>238</td>
    <td>222</td>
    <td>254</td>
    <td>267</td>
    <td>251</td>
    <td>261</td>
    <td>268</td>
    <td>252</td>
    <td>259</td>
    <td>256</td>
    <td>250</td>
  </tr>
</table>

Table 1: Statistics of ReflectEvo-460k. The average length in the table is computed by tokens.

defined below: i. Verify the failed solution. It analyzes the initial solution by tracing and examining the reasoning process with or without step-by-step verification. ii. Locate errors and diagnose potential reasons. It points out errors in specific reasoning steps and identifies the causes (Zeng et al., 2024). We delicately design prompts to mitigate the most common error types (Li et al., 2024c), including mathematical (calculation & algorithm) errors, logic and reasoning errors (flawed rationale & internal inconsistency), instruction violation (context misinterpretation, incomplete or irrelevant response & format discrepancy), factual errors. They are explicitly specified in the instructions for error elimination and accurate fault localization. iii. Outline strategies and plans for error correction and mitigation. It provides strategies and guidance to address the error by proposing a high or low-level plan to mitigate similar issues in the future.

Step 2: Data generation Based on the instructions outlined in Step 1, we introduce two components for reflection generation: a Generator G (reasoning model) that generates the initial answer with its reasoning process and a Reflector R (reflection model) that improves the incorrect answer through self-reflection and self-correction.

Generator G Given a q, G is built upon a base LLM instructed to generate interleaved thoughts and an initial answer G(a|q). It is implemented as described in ReAct (Yao et al., 2022), as the first step for self-reflection. We obtain the external environment feedback f by evaluating the correctness of a as a verifier. f is a binary signal “correct/incorrect” with limited information, which is usually the case in real scenarios, eliminating the need for enriched feedback from humans or more powerful models. If correct, a is directly used as the final answer. If incorrect, R is used to revise the solution iteratively. In this paper, we perform self-reflection once to maximize the efficiency of self-generated data; however, this approach can be extended to multiple iterations in future studies.

Reflector R We use exactly the same base LLM as G for R. The generation process for R is decomposed into two phases: self-reflection and self-correction. Self-reflection generates \( R(r|q, a, f) \) to identify errors in the reasoning process and conduct a deeper analysis of the causes. Self-correction refines \( a \) as \( R(\hat{a}|q, a, f, r) \). To enrich the self-training data, we sample k solution \( \{r_j, \hat{a}_j\}_{j=1}^k \) for each \( \{q, a, f\} \) conditioned on one specific prompt using reject sampling (Liu et al., 2023) to enrich the self-training data. We vary the prompts selected from the instruction pool to generate diverse self-reflection samples.

2.3 Reflection Curation

After the above-mentioned process, we obtain a reflection training set with \( M(N * k * m) \) samples:

\[
\mathcal{D} = \{q_i, a_i, f_i, (r_{i,j}, \hat{a}_{i,j})_{j=1}^{k*m}\}_{i=1}^N,
\]

where N is the number of QA pairs in \( \mathcal{D} \), m is the number of reflection instructions from pool, and k is the value of reject sampling. We aim to further curate the data for reflection learning as follows.

First, we filter r to include those followed by the correct \( \hat{a} \), indicating that these reflections are of high quality for error correction, denoted as \( \mathcal{D}^+ \):

\[
\mathcal{D}^+ = \{(q_i, a_i, f_i, r_i, \hat{a}_i) \mid (\hat{a}_i = a^*)\}_{i=1}^{|\mathcal{D}|}
\]

Subsequently, we leverage GPT-4o (Hurst et al., 2024) as a stronger teacher model to further select preferred reflection data from \( \mathcal{D}^+ \) to create pairwise data, denoted as \( \mathcal{D}^{\text{pref}} \):

\[
\mathcal{D}^{\text{pref}} = \{(q_i, a_i, f_i, [y_i^{\text{cho}}, y_i^{\text{rej}}]) \mid \exists y_i^{\text{cho}}, y_i^{\text{rej}}\}_{i=1}^{|\mathcal{D}^+|},
\]

where \( y = (r, \hat{a}) \) is the reflection and corresponding corrected answer. \( y^{\text{cho}} \) and \( y^{\text{rej}} \) are solutions randomly selected for each \( \{q, a, f\} \) whose r is chosen and rejected, respectively, by GPT-4o.

To fully utilize low-quality reflection data followed by \( \hat{a}_i \) that is still judged to be incorrect, we enrich the self-training data by incorporating both positive and negative samples as pairwise data for each \( \{q, a, f\} \), denoted as \( \mathcal{D}^{\pm} \):

\[
\mathcal{D}^{\pm} = \{(q_i, a_i, f_i, [y_i^+, y_i^-]) \mid \exists y_i^+, y_i^-\}_{i=1}^{|\mathcal{D}|},
\]

where \( y^+ \) and \( y^- \) are solutions whose \( \hat{a} \) is evaluated as correct or incorrect by \( a^* \).
Figure 2: (a) Task-dataset hierarchy distribution of ReflectEvo-460k. (b) Error type distribution of corrected thoughts identified by reflection in the test sets.

Following the above-mentioned steps in Sec. 2. We create a reflection training dataset ReflectEvo-460k by curating examples of 17 carefully selected source subsets from LogiQA (Liu et al., 2020), MATH (Hendrycks et al., 2021), MBPP (Austin et al., 2021), and BIG-bench (bench authors, 2023), spanning diverse domains and categories. The Statistics of the dataset are shown in Tab. 1 and Fig. 2a. We use three commonly used SLMs including Llama-3-8B (Dubey et al., 2024), Mistral-7B (Jiang et al., 2023) and Gemma-2-9B (Team et al., 2024) for the entire process of data generation, training, and test. Implementation details and instructions are provided in Appendices B.1 and C.

3 Reflection Learning on Self-Generated Data

In this section, we further investigate the effectiveness of reflection learning on the reflector \( R \) by adopting self-training on ReflectEvo-460k using supervised fine-tuning (SFT) (Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2024).

3.1 Reflection Learning

We use SFT on \( \mathcal{D}^+ \) in two different settings below. This strengthens the model to better leverage reflections as intermediate thoughts leading to positive \( \hat{a} \) for refinement.

Setting 1: We train the capability of self-reflection and self-correction in one stage:

\[
\mathcal{L}_1 = -\mathbb{E}_{(q, a, f, r, \hat{a}) \sim \mathcal{D}^+} \log R((r, \hat{a}) \mid q, a, f)
\]

Setting 2: We train the capacity of self-reflection and self-correction respectively in two stages:

\[
\mathcal{L}_{2.1} = -\mathbb{E}_{(q, a, f, r) \sim \mathcal{D}^+} \log R(r \mid q, a, f)
\]
\[
\mathcal{L}_{2.2} = -\mathbb{E}_{(q, a, f, r, \hat{a}) \sim \mathcal{D}^+} \log R(\hat{a} \mid q, a, f, r)
\]

Inspired by error-driven learning from humans, we also leverage negative samples \( \hat{a}^- \) that comprise a large portion of \( \mathcal{D}^{\pm} \) and offer valuable insights for model enhancement. In addition, we assume GPT-4o with better self-reflection, which is required for reflection preference annotation as \( \mathcal{D}^{\text{pref}} \), guiding SLMs to continuously refine reflections. We use preference learning through DPO on the aforementioned pairwise data to better judge and distinguish high-quality reflections from suboptimal ones in the following settings.

Setting 3: We train self-reflection only on \( \mathcal{D}^{\pm} \):

\[
\mathcal{L}_3 = -\mathbb{E}_{(x, r^+, r^-) \sim \mathcal{D}^{\pm}} \log \sigma[r_\theta(x, r^+) - r_\theta(x, r^-)]
\]
\[
r_\theta(x, r) = \beta \log \frac{\pi_\theta(r \mid x)}{\pi_{\text{ref}}(r \mid x)}, x = (q, a, f)
\]

Setting 4: We train self-reflection only on \( \mathcal{D}^{\text{pref}} \):

\[
\mathcal{L}_4 = -\mathbb{E}_{(x, r^{\text{cho}}, r^{\text{rej}}) \sim \mathcal{D}^{\text{pref}}} \log \sigma[r_\theta(x, r^{\text{cho}}) - r_\theta(x, r^{\text{rej}})]
\]

where \( R(\cdot) \) is the policy model \( \pi_\theta \) and \( G(\cdot) \) is the reference model \( \pi_{\text{ref}} \). \( \sigma \) is the logistic function and \( \beta \) is a hyperparameter that controls the proximity to the reference policy \( G(\cdot) \) in both settings 3 and 4. The objective is to steer \( R(\cdot) \) towards increasing the likelihood of \( r^+ \) with the correct solutions \( \hat{a} \) or chosen \( r^{\text{cho}} \) and decreasing the likelihood of \( r^- \) with incorrect solutions \( \hat{a} \) or rejected \( r^{\text{rej}} \) for given \( (q, a, f) \). More details can be found in Appendix B.2.
3.2 Inference
During inference, the process follows the same steps as those of reflection data generation in Sec. 2.2. We use the model after reflection learning as a reflector at the inference time for self-reflection and correction. It can be implemented as a multi-turn rollout that terminates either when the current \( a \) is judged to be correct or when it reaches the predefined maximum number of turns (two turns in our setting using twice QA with one intermediate reflection).

4 Experiments

4.1 Performance Learning on ReflectEvo
We measure the performance of self-reflection by adopting the following metrics: 1) Acc@t1: the model’s accuracy in the first turn; 2) Acc@t2: the model’s accuracy in the second turn; 3) \( \Delta(t1,t2) \): accuracy improvement between the first and second turns measuring the efficacy of self-reflection.

We compare three main methods in our experiments, including prompt-based QA with or without reflection without training, SFT training with direct answers, and self-training based reflection learning introduced in Sec. 3 from Setting 1 to Setting 4, noted as one-stage w/ \( \mathcal{D}^+ \), two-stage w/ \( \mathcal{D}^+ \), w/ \( \mathcal{D}^+ \) and w/ \( \mathcal{D}^{pref} \) respectively.

Overall Performance on Different Tasks Tab. 2 illustrates the overall performance on ReflectEvo. We discard the self-generated reflection data by Mistral on MATH due to its extremely low quality. We observe that LLMs gain more from prompt-based reflection, whereas SLMs show either minor improvements or degradation. This is primarily because without specialized training, SLMs inherently generate low-quality reflections and fail to leverage feedback effectively for self-correction. For comparison, experiments on our self-training methods show significant improvements in both models and various tasks. Specifically, it achieves over 20% in \( \Delta(t1,t2) \) for Llama-3 on MBPP and BIG-bench as well as Mistral on LogiQA and BIG-bench. Notably, all three evaluated models outperform their stronger model using ReflectEvo on BIG-bench. This indicates that different models and tasks benefit greatly from the four self-training methods, even surpassing the SFT on answers without step-by-step reasoning process, which paves the way for broader applications and scenarios for various SLMs.

![Bar chart showing performance across different tasks on Llama-3-8B](page_765_120_627_393.png)
Figure 3: Performance training with ReflectEvo across different tasks on Llama-3-8B.

![Line chart showing performance in multi-turn self-reflection on Llama-3-8B after tuning](page_765_563_627_393.png)
Figure 4: Performance in multi-turn self-reflection on Llama-3-8B after tuning.

Fig. 3 provides an in-depth analysis on the reflection learning across tasks. Our method significantly contributes to various tasks, including reasoning, math, QA, and comprehension, with an average of 22% in \( \Delta(t1,t2) \). For coding, it only improves to a certain degree probably due to the lack of fine-grained step-by-step critiques on the erroneous solutions for reflection training on models that are not specialized in coding.

Effect of Reflection from Teacher Model To investigate the influence of reflection sources, we compare different reflections generated by the SLM itself and a more advanced model like GPT-4o which acts as a teacher model with greater knowledge and reasoning capabilities in Tab. 3. Reflections from both models strengthen the \( \Delta(t1,t2) \) of QA performance after tuning under different settings proposed in Sec. 3, while the self-generated data require less cost and resources in practice. To our expectation, reflections from the teacher model yields more obvious improvements underscoring the benefits of high-quality reflection data generation and selection for further improvement.

Scaling Multi-turn Self-reflection We further extend the application of self-reflection to multi-
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">LogiQA</th>
    <th colspan="3">MATH</th>
    <th colspan="3">MBPP</th>
    <th colspan="3">BIG-bench</th>
  </tr>
  <tr>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td colspan="13">Meta-Llama-3-8B-Instruct</td>
  </tr>
  <tr>
    <th>Prompt based</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>30.2%</td>
    <td>38.8%</td>
    <td>+8.6%</td>
    <td>14.4%</td>
    <td>15.0%</td>
    <td>+0.6%</td>
    <td>28.4%</td>
    <td>44.0%</td>
    <td>+15.6%</td>
    <td>38.2%</td>
    <td>52.4%</td>
    <td>+14.2%</td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>36.2%</td>
    <td>46.0%</td>
    <td>+9.8%</td>
    <td>16.0%</td>
    <td>17.6%</td>
    <td>+1.6%</td>
    <td>45.8%</td>
    <td>47.4%</td>
    <td>+1.6%</td>
    <td>51.0%</td>
    <td>51.0%</td>
    <td>+12.8%</td>
  </tr>
  <tr>
    <th>SFT based</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ w/SFT qa pairs</td>
    <td>46.6%</td>
    <td>-</td>
    <td>-</td>
    <td>10%</td>
    <td>-</td>
    <td>-</td>
    <td>31.2%</td>
    <td>-</td>
    <td>-</td>
    <td>61.6%</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th>Self-training based (Ours)</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ one-stage w/ \( D^+ \)</td>
    <td><span style="background-color:#e0e0e0;">43.8%</span></td>
    <td><span style="background-color:#e0e0e0;">+13.6%</span></td>
    <td><span style="background-color:#e0e0e0;">23.6%</span></td>
    <td><span style="background-color:#e0e0e0;">+9.2%</span></td>
    <td><span style="background-color:#e0e0e0;">29.6%</span></td>
    <td><span style="background-color:#e0e0e0;">+1.2%</span></td>
    <td><span style="background-color:#e0e0e0;">71.2%</span></td>
    <td><span style="background-color:#e0e0e0;">+33.0%</span></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>→ two-stage w/ \( D^+ \)</td>
    <td><span style="background-color:#e0e0e0;">49.4%</span></td>
    <td><span style="background-color:#e0e0e0;">+19.2%</span></td>
    <td><span style="background-color:#e0e0e0;">14.5%</span></td>
    <td><span style="background-color:#e0e0e0;">+0.1%</span></td>
    <td><span style="background-color:#e0e0e0;">42.4%</span></td>
    <td><span style="background-color:#e0e0e0;">+14.0%</span></td>
    <td><span style="background-color:#e0e0e0;">45.4%</span></td>
    <td><span style="background-color:#e0e0e0;">+7.2%</span></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>→ w/ \( D^+ \)</td>
    <td>30.2%</td>
    <td>41.8%</td>
    <td>+11.6%</td>
    <td>14.4%</td>
    <td>15.2%</td>
    <td>+0.8%</td>
    <td>28.4%</td>
    <td>48.8%</td>
    <td>+20.4%</td>
    <td>38.2%</td>
    <td>63.0%</td>
    <td>+24.8%</td>
  </tr>
  <tr>
    <td>→ w/ \( D^{pref} \)</td>
    <td>39.2%</td>
    <td>+9.0%</td>
    <td></td>
    <td>14.8%</td>
    <td>+0.4%</td>
    <td></td>
    <td>47.4%</td>
    <td>+19.0%</td>
    <td></td>
    <td>59.6%</td>
    <td>+21.4%</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="13">Meta-Llama-3-70B-Instruct</td>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>42.4%</td>
    <td>64.4%</td>
    <td>+22.0%</td>
    <td>40.8%</td>
    <td>49.6%</td>
    <td>+8.8%</td>
    <td>71.0%</td>
    <td>+4.8%</td>
    <td>48.0%</td>
    <td>67.0%</td>
    <td>+19.0%</td>
    <td></td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>56.8%</td>
    <td>+14.4%</td>
    <td></td>
    <td>48.6%</td>
    <td>+7.8%</td>
    <td></td>
    <td>66.2%</td>
    <td>+6.8%</td>
    <td></td>
    <td>64.4%</td>
    <td>+16.4%</td>
    <td></td>
  </tr>
  <tr>
    <th>Mistral-TB-Instruct-v0.2</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <th>Prompt based</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>28.8%</td>
    <td>31.2%</td>
    <td>+2.4%</td>
    <td>9.2%</td>
    <td>10.6%</td>
    <td>+1.4%</td>
    <td>20.4%</td>
    <td>23.0%</td>
    <td>+2.6%</td>
    <td>36.6%</td>
    <td>43.8%</td>
    <td>+7.2%</td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>34.2%</td>
    <td>+5.4%</td>
    <td></td>
    <td>10.2%</td>
    <td>+1.0%</td>
    <td></td>
    <td>23.6%</td>
    <td>+3.2%</td>
    <td></td>
    <td>44.4%</td>
    <td>+7.8%</td>
    <td></td>
  </tr>
  <tr>
    <th>SFT based</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ w/SFT qa pairs</td>
    <td>28.8%</td>
    <td>-</td>
    <td>-</td>
    <td>7.6%</td>
    <td>-</td>
    <td>-</td>
    <td>17.0%</td>
    <td>-</td>
    <td>-</td>
    <td>37.8%</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th>Self-training based (Ours)</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ one-stage w/ \( D^+ \)</td>
    <td><span style="background-color:#e0e0e0;">38.4%</span></td>
    <td><span style="background-color:#e0e0e0;">+9.6%</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">24.0%</span></td>
    <td><span style="background-color:#e0e0e0;">+3.6%</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">51.6%</span></td>
    <td><span style="background-color:#e0e0e0;">+15.0%</span></td>
    <td></td>
  </tr>
  <tr>
    <td>→ two-stage w/ \( D^+ \)</td>
    <td><span style="background-color:#e0e0e0;">48.8%</span></td>
    <td><span style="background-color:#e0e0e0;">+20.0%</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">20.4%</span></td>
    <td><span style="background-color:#e0e0e0;">+0.4%</span></td>
    <td><span style="background-color:#e0e0e0;">-</span></td>
    <td><span style="background-color:#e0e0e0;">36.6%</span></td>
    <td><span style="background-color:#e0e0e0;">+34.5%</span></td>
    <td></td>
  </tr>
  <tr>
    <td>→ w/ \( D^+ \)</td>
    <td>28.8%</td>
    <td>39.2%</td>
    <td>+10.4%</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>23.2%</td>
    <td>+2.8%</td>
    <td></td>
    <td>50.2%</td>
    <td>+13.6%</td>
    <td></td>
  </tr>
  <tr>
    <td>→ w/ \( D^{pref} \)</td>
    <td>38.0%</td>
    <td>+9.2%</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>22.6%</td>
    <td>+2.2%</td>
    <td></td>
    <td>48.4%</td>
    <td>+11.8%</td>
    <td></td>
  </tr>
  <tr>
    <th>Mistral-22B-Small-Instruct</th>
    <td colspan="12"></td>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>46.4%</td>
    <td>62.8%</td>
    <td>+16.4%</td>
    <td>47.4%</td>
    <td>56.2%</td>
    <td>+8.8%</td>
    <td>63.0%</td>
    <td>68.0%</td>
    <td>+5.0%</td>
    <td>54.4%</td>
    <td>67.2%</td>
    <td>+12.8%</td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>62.0%</td>
    <td>+15.6%</td>
    <td></td>
    <td>52.8%</td>
    <td>+5.4%</td>
    <td></td>
    <td>68.0%</td>
    <td>+5.0%</td>
    <td></td>
    <td>68.0%</td>
    <td>+13.6%</td>
    <td></td>
  </tr>
</table>

Table 2: Performance on Llama-3 and Mistral using ReflectEvo.

<table>
  <tr>
    <th>Dataset</th>
    <th>Method</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td rowspan="4"><b>SR</b></td>
    <td>prompt-based</td>
    <td>36.2%</td>
    <td>+6.0%</td>
  </tr>
  <tr>
    <td>one stage w/ \( D^+ \)</td>
    <td>43.8%</td>
    <td>+13.6%</td>
  </tr>
  <tr>
    <td>two stage w/ \( D^+ \)</td>
    <td><b>49.4%</b></td>
    <td><b>+19.2%</b></td>
  </tr>
  <tr>
    <td>w/ \( D^+ \)</td>
    <td>41.8%</td>
    <td>+11.6%</td>
  </tr>
  <tr>
    <td>w/ \( D^{pref} \)</td>
    <td>39.2%</td>
    <td>+9.0%</td>
  </tr>
  <tr>
    <td rowspan="4"><b>TR</b></td>
    <td>prompt-based</td>
    <td>46.2%</td>
    <td>+16.0%</td>
  </tr>
  <tr>
    <td>one stage w/ \( D^+ \)</td>
    <td><b>52.0%</b></td>
    <td><b>+21.8%</b></td>
  </tr>
  <tr>
    <td>two stage w/ \( D^+ \)</td>
    <td>41.2%</td>
    <td>+11.0%</td>
  </tr>
  <tr>
    <td>w/ \( D^+ \)</td>
    <td>48.8%</td>
    <td>+18.6%</td>
  </tr>
  <tr>
    <td>w/ \( D^{pref} \)</td>
    <td>48.0%</td>
    <td>+17.8%</td>
  </tr>
</table>

Table 3: Performance on LogiQA using different sources of reflections on Llama-3 (Acc@t1=30.2% from Tab. 2). SR and TR indicate self-reflection and teacher-reflection respectively.

turn QA in Fig. 4. To our expectation, the results demonstrate a consistent improvement with increasing turns of reflection on different tasks. BIG-bench exhibits the most significant improvement, surpassing 80% accuracy after six turns and LogiQA also shows a notable upward trend, highlighting the effectiveness of iterative refinement. MBPP and MATH display relatively modest improvements with gradual increase, which suggesting that the impact of self-reflection learning is broadly beneficial but varies between tasks. It is encouraged to investigate the underlying factors that contribute to these differences to further enhance performance in various tasks.

Generalization across Different Tasks and Models We conduct deeper studies on the generalization of the self-reflection capability after tuning across different tasks (Tab. 4) and models (Tab. 5). Our findings reveal that the benefits of reflection learning generalize across tasks, particularly for LogiQA and BIG-bench with 10% increase, which commonly require strong reasoning abilities from LLMs. Due to the divergence of MATH and MBPP, there is merely improvement when trained on reflections generated from the other three datasets. We observe that all the test models in Tab. 5 benefit from the reflector after tuning for error correction in Acc@t2, especially for initial solutions from different generators. For Mistral and Gemma, even with a minor decrease compared with the corresponding results in Tab. 2 and Tab. 9, the result on these two models highlights the potential of our pipeline across different models and demonstrates the effectiveness of reflectors when applied to various generators.
<table>
  <tr>
    <th rowspan="2">Prompt based</th>
    <th colspan="3">LogiQA</th>
    <th colspan="3">MATH</th>
    <th colspan="3">BIG-bench</th>
    <th colspan="3">MBPP</th>
  </tr>
  <tr>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>30.2%</td><td>36.2%</td><td>+6.0%</td>
    <td>14.4%</td><td>16.0%</td><td>+1.6%</td>
    <td>38.2%</td><td>51.0%</td><td>+12.8%</td>
    <td>28.4%</td><td>45.8%</td><td>+17.4%</td>
  </tr>
  <tr>
    <th colspan="13">Self-training based (Ours)</th>
  </tr>
  <tr>
    <td>→ w/ \( \mathcal{D}^+ \) on LogiQA</td>
    <td colspan="3">-</td>
    <td>14.4%</td><td>+0.0%</td><td>-</td>
    <td style="background-color:#e6f2ff">60.0%</td><td style="background-color:#e6f2ff">+21.8%</td><td>-</td>
    <td>30.6%</td><td>+2.2%</td><td>-</td>
  </tr>
  <tr>
    <td>→ w/ \( \mathcal{D}^+ \) on MATH</td>
    <td>30.2%</td><td>36.6%</td><td>+6.4%</td>
    <td>14.4%</td><td>+0.0%</td><td>-</td>
    <td style="background-color:#e6f2ff">54.8%</td><td style="background-color:#e6f2ff">+16.6%</td><td>-</td>
    <td>28.4%</td><td>+0.4%</td><td>-</td>
  </tr>
  <tr>
    <td>→ w/ \( \mathcal{D}^+ \) on BIG-bench</td>
    <td style="background-color:#e6f2ff">52.0%</td><td style="background-color:#e6f2ff">+21.8%</td><td>-</td>
    <td>14.4%</td><td>+0.0%</td><td>-</td>
    <td>38.2%</td><td>40.2%</td><td>+2.0%</td>
    <td>38.2%</td><td>+7.8%</td><td>-</td>
  </tr>
  <tr>
    <td>→ w/ \( \mathcal{D}^+ \) on MBPP</td>
    <td>30.4%</td><td>+0.2%</td><td>-</td>
    <td>14.6%</td><td>+0.2%</td><td>-</td>
    <td>40.2%</td><td>+2.0%</td><td>-</td>
    <td>38.2%</td><td>+7.8%</td><td>-</td>
  </tr>
</table>

Table 4: Generalization across tasks for Llama-3 training one-stage with different task-specific subsets in \( \mathcal{D}^+ \).

<table>
  <tr>
    <th>Different generators</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>28.8%</td>
    <td>45.8%</td>
    <td>+17.0%</td>
  </tr>
  <tr>
    <td>Gemma-2-9B</td>
    <td>47.6%</td>
    <td>57.2%</td>
    <td>+9.6%</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>37.4%</td>
    <td>51.0%</td>
    <td>+13.6%</td>
  </tr>
</table>

Table 5: Generalization across generators using the same reflector Llama-3 training one-stage with subset of LogiQA in \( \mathcal{D}^+ \).

<table>
  <tr>
    <th>Self-training method</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td>one-stage w/ \( \mathcal{D}^+ \)</td>
    <td>28.8%</td>
    <td>40.2%</td>
    <td>+11.4%</td>
  </tr>
  <tr>
    <td>two-stage w/ \( \mathcal{D}^+ \)</td>
    <td>28.8%</td>
    <td>38.0%</td>
    <td>+9.2%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^+ \)</td>
    <td>28.8%</td>
    <td>39.4%</td>
    <td>+10.6%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^{ref} \)</td>
    <td>28.8%</td>
    <td>38.6%</td>
    <td>+9.8%</td>
  </tr>
</table>

Table 6: Generalization of generated reflection data on LogiQA of LLaMA-3.1-8B training on Mistral-7B

<table>
  <tr>
    <th>Self-training Method</th>
    <th>Oracle Groundtruth</th>
    <th>Self-judgement</th>
  </tr>
  <tr>
    <td>one-stage w/ \( \mathcal{D}^+ \)</td>
    <td>43.8%</td>
    <td>32.8%</td>
  </tr>
  <tr>
    <td>two-stage w/ \( \mathcal{D}^+ \)</td>
    <td>49.4%</td>
    <td>50.2%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^+ \)</td>
    <td>41.8%</td>
    <td>40.6%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^{ref} \)</td>
    <td>39.2%</td>
    <td>37.8%</td>
  </tr>
</table>

Table 7: Acc@t2 using different verifiers during inference on LogiQA for Llama-3. (Acc@t1=30.2% and Acc@t2= 36.2% without tuning from Tab. 2)

<table>
  <tr>
    <th>Methods</th>
    <th>Acc@t1</th>
    <th>Acc@t2</th>
    <th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <td>StAR (<a href="https://arxiv.org/abs/2206.04372">Zelikman et al., 2022</a>)</td>
    <td>40.0%</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Re-ReST (<a href="https://arxiv.org/abs/2401.07197">Dou et al., 2024</a>)</td>
    <td>38.8%</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>RISE (<a href="https://arxiv.org/abs/2401.07197">Qu et al., 2024</a>)</td>
    <td>31.4%</td>
    <td>34.4%</td>
    <td>+3.0%</td>
  </tr>
  <tr>
    <td>one-stage w/ \( \mathcal{D}^+ \)</td>
    <td>30.2%</td>
    <td>43.8%</td>
    <td>+13.6%</td>
  </tr>
  <tr>
    <td>two-stage w/ \( \mathcal{D}^+ \)</td>
    <td>30.2%</td>
    <td>49.4%</td>
    <td>+19.2%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^+ \)</td>
    <td>30.2%</td>
    <td>41.8%</td>
    <td>+11.6%</td>
  </tr>
  <tr>
    <td>w/ \( \mathcal{D}^{ref} \)</td>
    <td>30.2%</td>
    <td>39.2%</td>
    <td>+9.0%</td>
  </tr>
</table>

Table 8: Performance on different baselines using LLaMA-3.1-8B on LogiQA.

In Tab. 6, we further explore whether the self-reflection data of one LLM can be beneficial for the other. Compared with Tab. 2 in our paper, we find that the reflection data generated by LLaMA-3.1-8B is helpful for Mistral-7B on reflection learning with comparable or even better performance. It indicates that our dataset ReflectEvo-460k could be reusable for the community for future studies.

Effect of Different Verifiers on Self-Reflection
In this paper, self-reflection is performed only when the model’s answer is verified as incorrect using the ground truth. Another potential approach is to train the model itself as a verifier or use an external reward function to score the model’s answer based on a predefined threshold. We compare the effects of using oracle ground truth and self-judgments generated by the baseline model as verifiers in Tab. 7. For both verifiers, reflection learning improves Acc@t1 by an average of 13+% and enhances Acc@t2 by up to 7% compared with the untuned version. Although the baseline model, without specialized training, exhibits occasional misjudgments, its verification process results in minor performance degradation on the advantage of reflection learning. We leave this a direction for further exploration on the optimized verifiers in an end-to-end pipeline.

In Tab. 8, we make further experiments by comparing with three well-acknowledged baselines for self-improvement through reflection or correction. To make the comparison fair, the external feedback used in all the experiments is only a binary signal “correct/incorrect” without further explanation or given ground truth. We follow the evaluation setting in the original paper. Comparing with StAR and Re-ReST, Ours benefit from improvement over turns. Our methods achieves much higher reasoning performance after self-reflection, which emulates human learning through a slower and deeper thought process that iteratively and ultimately derives self-evolution

4.2 In-depth Analysis on Reflection

Error Types Identified by Reflection
To dissect the intrinsic properties of our reflection data, we analyze the error types in the initial thoughts specified by the reflection across all test sets. Potential error labels are generated heuristically by auto-tagging with GPT-4o and then calibrated by human annotators, achieving Cohen’s kappa of 51.18% with moderate agreement (<a href="https://doi.org/10.1016/S0098-3108(77)80004-6">Landis and Koch, 1977</a>) indi-
Question: Simplify the expression \(5(9x^2+2x+7)+3(x^2+7x+5)+2\). Express your answer as a polynomial with terms arranged in decreasing order of degree.

Thought: We can combine like terms in the given expression. The terms with \(x^2\)’s are \(59x^2+23x^2=12x^2\), the terms with \(x\)’s are \(5x+3x+57x=58x\), and the constant term is \(7+2+98\). Therefore, the simplified expression is \(12x^2+58x+98\).

Reflection: The failure occurred because I incorrectly combined the terms with \(x^2\) by adding their coefficients, resulting in an invalid result. I also mistakenly added 9, not recognizing that there was no constant term in the original expression. New plan: Moving forward, I will first combine the terms with the highest power of x, then handle the terms with the next highest power, and finally deal with the constants. I will ensure terms are arranged in decreasing order of degree to maintain accuracy.

Corrected Thought: To simplify the expression \(5(9x^2+2x+7)+(3x^2+7x+5)+2\), I first identified the terms with the highest powers of x, \(59x^2\) and \(50x^2\), and kept them separate since they have different coefficients. Then, I added the \(5x+25\) and \(33x+25\) terms, which resulted in \(12x^2+58x+98\). For the constant terms, \(7+5+2\), I added them to get \(98\). Finally, I arranged the terms in decreasing order of degree: \(7x^2+12x^2+3x+98\).

Thought & reflection pathway
<table>
  <tr>
    <th>Key snippets of reflection in pathway</th>
    <th>Correct constituents in thought / reflection</th>
    <th>Erroneous constituents in thought / reflection</th>
  </tr>
  <tr>
    <td>incorrectly combined the terms with \(x^2\) by adding their coefficients</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>mistakenly added 9, not recognizing that there was no constant term in the original expression</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>arranged in decreasing order of degree to maintain accuracy</td>
    <td></td>
    <td></td>
  </tr>
</table>

Question: \(5\left(\frac{1}{4}\right)^4\)

Thought: We have \(5\left(\frac{1}{4}\right)^4 = \left(\frac{1}{2}\right)^4 = \left(\frac{1}{16}\right) = \boxed{\frac{1}{16}}\).

Reflection: A possible reason for failure is that I did not consider the fact that \(5^2=25\), but instead used the property \(5^2=15\). This led me to incorrectly simplify the expression as \(5\left(\frac{1}{4}\right)^4=\left(\frac{1}{16}\right)=\boxed{\frac{1}{16}}\). New Plan: To avoid this mistake, I will carefully evaluate each power of I separately before simplifying the expression. Specifically, I will use the property \(5^n\equiv n\mod 4\), where n mod 4 gives the remainder when n is divided by 4. Then, I will substitute the value of I raised to the appropriate power into the original expression.

Corrected Thought: The mistake in the initial approach was using the incorrect property \(5^2=15\) instead of \(5^2=18\), leading to an incorrect simplification. To fix this, we use the fact that \(5^n\equiv n\mod 4\) and evaluate powers of 5 correctly. To find \(5\left(\frac{1}{4}\right)^4\), 45, we can simplify step by step:
\[
\begin{align*}
\left(\frac{1}{4}\right)^4 &= \frac{1}{16} \\
\left(\frac{1}{16}\right) &= \frac{1}{16} \\
\end{align*}
\]
Thus, the correct result is \(\boxed{\frac{1}{16}}\).

Figure 5: Qualitative examples from the MATH. “False to True” and a “False to False” stand for successful and failed correction in the second turn respectively. The key snippets highlighted in green, red and yellow indicate correct, erroneous thought and reflection respectively.

![Social IQA and StrategyQA performance vs correlation coefficient plots](page_1012_1042_377_377.png)

Figure 6: Task performance (Acc@t2) versus the correlation between reflection and the second-turn thought.

Reasoning Errors (88.4%) and Instruction Violation (47.9%), indicating that math and logic issues were the primary causes. We also provide detailed error distributions for the different subsets. MATH has a higher percentage of Calculation Errors (20.8%) than the other subsets, whereas COQA has more Context misinterpretation (43.1%). This shows that our method provides tailored reflections for specific domains rather than superficial or general advice.

Correlation Analysis in Reflection We calculate the correlation between reflection and second-turn corrected thoughts, and we assess the association between the correlation and Acc@t2 after self-correction. Empirically, we hypothesize that they have a linear relationship, and we select the Pearson correlation coefficient by computing the semantic similarity for each pair of data (see the details in Fig. 7). As we have seen, reflection learning can improve the ability of models to correct errors; we argue that if reflection is indeed specific to the error in thought, then task performance should intuitively be enhanced as the correlation between reflection and corrected thought increases.

Measuring with the Pearson coefficient, Fig. 6 and Fig. 7 show that StrategyQA, Social IQA, VitaminC, and SQuAD all have a clear linear relationship between the performance and the correlation of reflection – second-turn thought, while MATH

cating high annotation quality.

Fig. 2b shows five coarse-grained and nine fine-grained error types identified through human calibration. The most common errors are Logic and
and MBPP exhibit irrelevant tendency or show a slightly negative correlation implying their desire data of fine-grained reflection. Comparing the blue and red correlation curves, we find that more similarity between the reflection and corrected thought, more effective correction (i.e., higher performance) that outperforms the vanilla model.

Case Studies We perform case studies to see how reflection interacts with the thought process by making critiques and refinements in Fig. 5. We random sampled 100 cases from the MATH test set and display two of them. In the case “False to True”, reflection precisely recaps the key causes of error and explicitly bridges the logical pathway between the initial thought and the corrected one, which finally results in the correct answer. In contrast, we find that even tiny erroneous constituent in the reflection may lead to a false reasoning thought and final answer. It validates that high-quality reflection is helpful for incentivizing the model to generate thought with correct answer while flawed reflection still lead to repeated errors after self-correction.

5 Related Work

Self-training and Self-Improvement Self-training allows a model to learn from its own outputs, reducing its reliance on human-annotated data or superior models (Zelikman et al., 2022; Yuan et al., 2024; Chen et al., 2024). Previous research has primarily concentrated on enhancing models’ reasoning abilities through SFT (Yuan et al., 2023) with positive samples or preference learning using both positive and negative samples to potentially leverage valuable information in incorrect solutions and recent advances also extend self-training to agentic scenarios (Wang et al., 2024a; Wallace et al., 2024; Gulcehre et al., 2023; Song et al., 2024; Motwani et al., 2024; Li et al., 2024b). We further advocate reducing reliance on resource-heavy rationale annotations via self-training for SLMs.

Learning for Self-reflection Recent research highlights the significant benefits of integrating self-reflection into LLMs to enhance their reasoning and problem-solving capabilities, by iteratively refining their responses (Kumar et al., 2024; Cheng et al., 2024; Qu et al., 2024; Yao et al., 2023; Zhou et al., 2024; Liang et al., 2024; Moskvoretskii et al., 2025). Shinn et al. (2024) reinforces the language agent to verbally reflect on task feedback and induce better plans in subsequent trials. Dou et al. (2024) employs the low-quality outputs generated from the weak model iteratively by fine-tuning the reflection module for self-refinement. Zhang et al. (2024) further validates that SLMs have the ability of self-correction on reasoning tasks by accumulating high-quality critique-correction data. We pioneer the exploration of reflection learning on self-generated data.

6 Conclusion

We propose ReflectEvo to enhance SLMs through reflection learning by iteratively generating self-training data, which achieves substantial performance improvements, even surpassing much larger models highlighting its generalization across various models and tasks for future research.

Limitations

Despite the promising results of ReflectEvo through reflection learning, there are several limitations to our work. The quality of the self-generated reflection data is highly dependent on the initial reasoning ability of the SLMs. Models with inherently weak reasoning capabilities may struggle to produce high-quality reflections, which in turn limits the effectiveness of the self-training. While our pipeline demonstrates significant improvements in certain tasks, tasks such as coding and mathematics require more specialized knowledge and step-by-step critiques than reasoning and comprehension tasks. Future work could explore more sophisticated feedback mechanisms with optimized verifiers or reward functions during inference to enhance the reflection learning process.

Acknowledgement

The authors thank the reviewers for their insightful suggestions on improving the manuscript. This work presented herein is supported by the National Natural Science Foundation of China (62376031).

Ethics Statement

We adhere to ethical principles to ensure the responsible development and application of our proposed techniques. Our work focuses on enhancing the self-reflection abilities of SLMs without directly involving human subjects or sensitive information. We acknowledge the potential broader impacts of our research, recognize the environmental and computational costs associated with LLM training, and strive to optimize our methods for efficiency.
References

Sue Atkins and Kathy Murphy. 1993. Reflection: a review of the literature. Journal of advanced nursing, 18(8):1188–1192.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335.

Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. 2024. Vision-language models can self-improve reasoning via reflection. arXiv preprint arXiv:2411.00855.

David Denton. 2011. Reflection and learning: Characteristics, obstacles, and implications. Educational Philosophy and Theory, 43(8):838–852.

Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, and Nanyun Peng. 2024. Re-rest: Reflection-reinforced self-training for language agents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15394–15411.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Thomas Palmeira Ferraz, Kartik Mehta, Yu-Hsiang Lin, Haw-Shiuan Chang, Shereen Oraby, Sijia Liu, Vivek Subramanian, Taeyoung Chung, Mohit Bansal, and Nanyun Peng. 2024. Llm self-correction with decrim: Decompose, critique, and refine for enhanced following of instructions with multiple constraints. arXiv preprint arXiv:2410.06458.

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. 2024. Self-correction is more than refinement: A learning framework for visual and language reasoning tasks. arXiv preprint arXiv:2410.04055.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.11610.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AI Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yu Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. 2024. Training language models to self-correct via reinforcement learning, 2024. URL https://arxiv.org/abs/2409.12917.

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference on Learning Representations.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024a. Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Jiaqi Li, Xiaobo Wang, Wentao Ding, Zihao Wang, Yipeng Kang, Zixia Jia, and Zilong Zheng. 2024b. Ram: Towards an ever-improving memory system by learning from communications. arXiv preprint arXiv:2404.12045.

Ming Li, Lichang Chen, Jiuhai Chen, Shuai He, Heng Huang, Jiuxiang Gu, and Tianyi Zhou. 2023. Reflection-tuning: Data recycling improves llm instruction-tuning. arXiv preprint arXiv:2310.11716.
Youquan Li, Miao Zheng, Fan Yang, Guosheng Dong, Bin Cui, Weipeng Chen, Zenan Zhou, and Wentao Zhang. 2024c. Fb-bench: A fine-grained multi-task benchmark for evaluating llms’ responsiveness to human feedback. arXiv preprint arXiv:2410.09412.

Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. 2024. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm. arXiv preprint arXiv:2408.08072.

Jian Liu, Leyang Cui, Hannmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logica: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124.

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. 2023. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657.

Trung Quoc Luong, Xinbo Zhang, Zhanning Jie, Peng Sun, Xiaoran Jin, and Hang Li. 2024. Reft: Reasoning with reinforced fine-tuning. arXiv preprint arXiv:2401.08967.

Viktor Moskvoretskii, Chris Biemann, and Irina Nikishina. 2025. Self-taught self-correction for small language models. arXiv preprint arXiv:2503.08681.

Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Markian Rybchuk, Philip HS Torr, Ivan Laptev, Fabio Pizzati, Ronald Clark, and Christian Schroeder de Witt. 2024. Malt: Improving reasoning with multi-agent llm training. arXiv preprint arXiv:2412.01928.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. 2023. Language model self-improvement by reinforcement learning contemplation. arXiv preprint arXiv:2305.14483.

Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. 2024. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is really a reward model. Advances in Neural Information Processing Systems, 36.

Matthew Renze and Erhan Guven. 2024. Self-reflection in llm agents: Effects on problem-solving performance. arXiv preprint arXiv:2405.06682.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502.

Xiaojuan Tang, Jiaqi Li, Yitao Liang, Muhan Zhang, and Zilong Zheng. 2024. Mars: Situated inductive reasoning in an open-world environment. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024) Track on Datasets and Benchmarks.

Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. 2023. Large language models are in-context semantic reasoners rather than symbolic reasoners. arXiv preprint arXiv: 2305.14825.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Zilong Zheng Tong Wu, Yanpeng Zhao. 2024. An efficient recipe for long context extension via middle-focused positional encoding. In Advances in Neural Information Processing Systems (NeurIPS), volume 37.

Johan Von Wright. 1992. Reflections on reflection. Learning and instruction, 2(1):59–68.

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. 2024. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8228–8238.

Tianduo Wang, Shichen Li, and Wei Lu. 2024a. Self-training with direct preference optimization improves chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11917–11928. Association for Computational Linguistics.

Yuxuan Wang, Alan Yuille, Zhuowan Li, and Zheng Zilong. 2024b. Exovip: Step-by-step verification and exploration with exoskeleton modules for compositional visual reasoning. In The first Conference on Language Modeling (CoLM).

Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithest Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. 2023. Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308.02151.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020.

Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488.

Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, et al. 2024. Mr-ben: A comprehensive meta-reasoning benchmark for large language models. arXiv preprint arXiv:2406.13975.

Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jackyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. 2024. Small language models need strong verifiers to self-correct reasoning. arXiv preprint arXiv:2404.17140.

Runlong Zhou, Simon S Du, and Beibin Li. 2024. Reflect-rl: Two-player online rl fine-tuning for lms. arXiv preprint arXiv:2402.12621.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">LogiQA</th>
    <th colspan="4">MATH</th>
    <th colspan="4">MBPP</th>
    <th colspan="4">BIG-bench</th>
  </tr>
  <tr>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
    <th>Acc@t1</th><th>Acc@t2</th><th>Δ(t1,t2)</th>
  </tr>
  <tr>
    <th colspan="13">Gemma-2-9B-it</th>
  </tr>
  <tr>
    <th>Prompt based</th>
    <td colspan="13"></td>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>47.6%</td><td>63.0%</td><td>+15.4%</td>
    <td>34.6%</td><td>40.4%</td><td>+5.8%</td>
    <td>54.4%</td><td>59.2%</td><td>+4.8%</td>
    <td>61.2%</td><td>75.6%</td><td>+14.4%</td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>60.0%</td><td>+12.4%</td><td>40.0%</td><td>+5.4%</td><td>59.6%</td><td>+5.2%</td><td>74.4%</td><td>+13.2%</td>
  </tr>
  <tr>
    <th>SFT based</th>
    <td colspan="13"></td>
  </tr>
  <tr>
    <td>→ w/ SFT qa pairs</td>
    <td>50.6%</td><td>-</td><td>18.6%</td><td>-</td><td>-</td><td>37.0%</td><td>-</td><td>-</td><td>74.6%</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <th>Self-training based (Ours)</th>
    <td colspan="13"></td>
  </tr>
  <tr>
    <td>→ one-stage w/ \( D^+ \)</td>
    <td>62.4%</td><td>+14.8%</td><td>40.0%</td><td>+5.4%</td><td>57.8%</td><td>+3.4%</td><td>78.4%</td><td>+17.2%</td>
  </tr>
  <tr>
    <td>→ two-stage w/ \( D^+ \)</td>
    <td>60.6%</td><td>+13.0%</td><td>35.0%</td><td>+0.4%</td><td>56.8%</td><td>+2.4%</td><td>67.0%</td><td>+5.8%</td>
  </tr>
  <tr>
    <td>→ w/ \( D^+ \)</td>
    <td>47.6%</td><td>62.6%</td><td>+15.0%</td><td>34.6%</td><td>40.0%</td><td>+5.4%</td><td>54.4%</td><td>58.6%</td><td>+4.2%</td><td>61.2%</td><td>74.8%</td><td>+13.6%</td>
  </tr>
  <tr>
    <td>→ w/ \( D^{ref} \)</td>
    <td>63.0%</td><td>+15.4%</td><td>40.0%</td><td>+5.4%</td><td>59.4%</td><td>+5.0%</td><td>75.0%</td><td>+13.8%</td>
  </tr>
  <tr>
    <th colspan="13">Gemma-2-27B-it</th>
  </tr>
  <tr>
    <td>→ w/o reflection</td>
    <td>52.0%</td><td>59.2%</td><td>+7.2%</td><td>38.4%</td><td>43.8%</td><td>+5.4%</td><td>69.4%</td><td>+4.0%</td><td>72.0%</td><td>+8.6%</td>
  </tr>
  <tr>
    <td>→ w/ reflection</td>
    <td>65.4%</td><td>+13.4%</td><td>45.0%</td><td>+6.6%</td><td>65.4%</td><td>+3.8%</td><td>63.4%</td><td>75.2%</td><td>+11.8%</td>
  </tr>
</table>

Table 9: Performance on Gemma-2 using ReflectEvo.

A  Further analysis and results

Tab. 9 indicates that BIG-bench gains more from reflection tuning with Acc@t2 = 78.4% with a substantial increase of +17.2% compared with other tasks and the baseline methods. However, our method on Gemma-2 shows marginal improvement compared with Llama-3 and Mistral probably due to its inherent strong reasoning capability (comparable performance on different models in sizes of 9B & 27B). It may either need selection of higher-quality reflection data or supervision from superior models and further optimization on the training methods for reflection enhancement. Due to the defect of SFT training without step-by-step reasoning process and the limited number of training data, the SFT performance of MATH and MBPP degrade due to the nature of fast thinking than those with reflection.

B  Implementation Details

B.1  Reflection generation

In this paper, we conduct the self-reflection once during the process of the two turns of reasoning and answering for both data generation and inference across most experiments. The generalization performance of multi-turn self-reflection can be found in Fig. 4.

The number of reject sampling \( k \) is 2. To validate the effectiveness of reflection tuning on various tasks, we incorporate 14 datasets selected from BIG-bench besides LogiQA, MATH and MBPP. Those datasets are delicately selected to focus more on the comprehension and reasoning abilities across diverse domains, comprising a comprehensive collection of dataset. The datasets includes: Commonsense Reasoning (RiddleSense, TimeDial, Known Unknowns), Social Reasoning (Social IQa, Implicit Interpersonal Relations), Reading Comprehension (VitaminC, SQuADShifts), Logical Reasoning (StrategyQA, Analytic Entailment), Contextual QA (CoQA Conversational Question Answering), Context Free QA (Truthful QA), Causal Reasoning (Causal Judgment, Cause and Effect), and Physics Reasoning (Physical Intuition). For datasets with more than 1000 samples, we randomly select 1000 QA pairs; for datasets with fewer than 1000 samples, we retain the entire original set.

Each reflection instruction consists of the three stages introduced in step 1 in Sec. 2 and different variants of prompts used in each stage can be seen Appendix C.1. The combination of them forms a diverse, comprehensive instruction pool with 32 (2*8*2) instructions used in step 2. For each dataset, we random select 5 or 6 of the instructions (\( M \)) to generate the reflections in ReflectEvo-460k considering the data generation efficiency.

For each task, we use corresponding subset for training without using the whole ReflectEvo-460k. For example, we use the training set of LogiQA for reflection generation and learning for LLaMA-3.1-8B and evaluate the same model on the test set of LogiQA in the experiments.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">LogiQA</th>
    <th colspan="3">MATH</th>
    <th colspan="3">BIG-bench</th>
    <th colspan="3">MBPP</th>
  </tr>
  <tr>
    <th>D<sup>+</sup></th>
    <th>D<sup>±</sup></th>
    <th>D<sup>pref</sup></th>
    <th>D<sup>+</sup></th>
    <th>D<sup>±</sup></th>
    <th>D<sup>pref</sup></th>
    <th>D<sup>+</sup></th>
    <th>D<sup>±</sup></th>
    <th>D<sup>pref</sup></th>
    <th>D<sup>+</sup></th>
    <th>D<sup>±</sup></th>
    <th>D<sup>pref</sup></th>
  </tr>
  <tr>
    <th>Training</th>
    <td>25371</td>
    <td>152475</td>
    <td>59870</td>
    <td>13796</td>
    <td>50946</td>
    <td>28225</td>
    <td>20410</td>
    <td>70672</td>
    <td>30909</td>
    <td>1151</td>
    <td>5365</td>
    <td>2609</td>
  </tr>
  <tr>
    <th>Testing</th>
    <td>500</td>
    <td></td>
    <td></td>
    <td>500</td>
    <td></td>
    <td></td>
    <td>500</td>
    <td></td>
    <td></td>
    <td>500</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 10: Data statistics for training and testing samples in the experiments.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>one-stage w/ \( D^+ \)</th>
    <th>two-stage w/ \( D^+ \)</th>
    <th>w/ \( D^\pm \)</th>
    <th>w/ \( D^{pref} \)</th>
  </tr>
  <tr>
    <td>learning rate</td>
    <td>1e-3</td>
    <td>1e-5,1e-3</td>
    <td>5e-7,7e-7</td>
    <td>5e-7,7e-7</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td>0</td>
    <td>0-0.01</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>max grad norm</td>
    <td>1.0</td>
    <td>1.0</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>\( \beta_1 \) for SFT</td>
    <td>0.9</td>
    <td>0.99</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>\( \beta_2 \) for SFT</td>
    <td>0.999</td>
    <td>0.9</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>\( \beta \) for DPO</td>
    <td>-</td>
    <td>-</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>\( \epsilon \)</td>
    <td>1e-8</td>
    <td>1e-08</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>max new tokens</td>
    <td>512</td>
    <td>512</td>
    <td>248</td>
    <td>248</td>
  </tr>
</table>

Table 11: The hyperparameters for reflection tuning.

B.2 Training

All the experiments for reflection tuning can be conducted on two Nvidia A100 80GB GPU, 32GB memory, 128 Core AMD CPU. The resource costs are mainly dependent on the tuning methods (full-parameter fine-tuning, parameter-efficient fine-tuning (PEFT) and DPO), the sizes of the models, and the sizes of the datasets. The main hyperparameters used for different settings of reflection tuning are shown in Tab. 11. The learning rate varies based on different models and tasks.

For one stage training with w/ \( D^+ \), we use LoRA-based PEFT in this setting with 4-bit quantization via BitsAndBytes. We set LoRA rank \( r = 8 \), scaling factor \( \alpha = 32 \), and a dropout rate of 0.1. The per-device batch size is set to 16 for LogiQA and 8 for others. For two stage training with w/ \( D^+ \), we use the full-parameter SFT in this setting with bfloat16 data precision. The per-device batch size is set to 16 with gradient accumulation of 4. For DPO training with both \( D^\pm \) and \( D^{pref} \), the per-device training batch size is set to 2, and gradient accumulation is set to 32.
Figure 7: Task performance (Acc@t2) versus the correlation between reflection and second-turn thoughts for Llama-3-8B with self-training reflection (blue dots and curve) and prompt-based reflection (red dots and curve). The ideal correlation (green dashed curve) denotes a standard linear tendency for comparison purposes, and the black dashed line represents Llama-3-8B without reflection. Note that the y-values of the spots denote the average performance (axis-y), where an array of test data points is located in a specific interval of the correlation coefficient (axis-x), and the correlation coefficient of these spots is also averaged by the values in the same interval.

<table>
  <tr>
    <th></th>
    <th>Logical Reasoning</th>
    <th>Mathematics</th>
    <th>Coding</th>
    <th>Contextual QA</th>
    <th>Reading Comprehension</th>
    <th>Commonsense Reasoning</th>
    <th>Social Reasoning</th>
  </tr>
  <tr>
    <th>Logical Reasoning</th>
    <td>1.00</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Mathematics</th>
    <td>0.53</td>
    <td>1.00</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Coding</th>
    <td>0.61</td>
    <td>0.83</td>
    <td>1.00</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Contextual QA</th>
    <td>0.39</td>
    <td>0.18</td>
    <td>0.38</td>
    <td>1.00</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Reading Comprehension</th>
    <td>0.56</td>
    <td>0.22</td>
    <td>0.42</td>
    <td>0.16</td>
    <td>1.00</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th>Commonsense Reasoning</th>
    <td>0.59</td>
    <td>0.24</td>
    <td>0.45</td>
    <td>0.68</td>
    <td>0.24</td>
    <td>1.00</td>
    <td></td>
  </tr>
  <tr>
    <th>Social Reasoning</th>
    <td>0.48</td>
    <td>0.32</td>
    <td>0.90</td>
    <td>0.14</td>
    <td>0.31</td>
    <td>0.14</td>
    <td>1.00</td>
  </tr>
</table>

Figure 8: Correlation of reflection between each pair of tasks. We obtain the semantic representation for all reflections via the Nv-Embed-v2 model (Lee et al., 2025) and calculate the Spearman correlation between each pair of tasks. The results are as follows: 1) Logical Reasoning has a moderate correlation with all tasks, which indicates that logic is a fundamental ability that supports other tasks; 2) Coding and Math have a high correlation, implying that similar thinking patterns are required for handling math and coding problems; and 3) Commonsense Reasoning and Social Reasoning show low correlation (0.14), suggesting that these abilities might require different cognitive skills.
C  Prompts

C.1  Reflection generation for ReflectEvo-460k

Instruction: Given the question and relevant context, you were unsuccessful in answering the question. As an advanced reasoning agent, you are required to enhance your incorrect solution and correctly answer the question based on self-reflection.

Question: {Question}

Previous trial and your incorrect solution: {Scratchpad}

Based on this information, please provide the following:
# Stage 1: Verify the failed solution
    1-1. Analyze the failed solution by tracing and examining its execution with step-by-step verification.
    1-2. Quickly go through the failed solution without step-by-step verification.

# Stage 2: Locate errors and diagnose potential reasons
    Identify specific steps where the errors occur and diagnose potential reasons.
    2-1. Review your calculation process to ensure that all the operations are accurate.
    2-2. Review your algorithm logic to ensure all steps follow the correct order.
    2-3. Review your solution to ensure it maintains logical coherence.
    2-4. Review your solution to check statements and conclusions for internal consistency.
    2-5. Review the context and requirements presented in the question to confirm that the response aligns with them.
    2-6. Review your solution to ensure that it is relevant to the question and addresses each aspect of the question.
    2-7. Review your solution to ensure it conforms to the required format and guidelines in a well-organized structure.
    2-8. Review your solution to ensure all provided facts are accurate and up to date.

# Stage 3: Outline strategies and plans on error correction and mitigation
    3-1. Outline a high-level plan explaining how these changes will mitigate similar issues.
    3-2. Outline a low-level plan proposing specific strategies or corrections to address these issues.

Please follow the instructions without any additional introductory or concluding statements. Do not provide the answer directly. You will be punished to output more than 100 words.

C.2  Self-reflection for Reflector

Instruction: You are an advanced reasoning agent that can improve based on self-reflection. You will be given a previous reasoning trial in which you were given a question to answer. You were unsuccessful in answering the question. In a few sentences, diagnose a possible reason for failure and devise a new, concise, high-level plan that aims to mitigate the same failure. Use complete sentences.

Question: {Question}
Previous trial and your incorrect solution: {Scratchpad}
C.3  Reasoning for Generator

Instruction: In this task, you are required to solve a question with interleaving Thought, Action, and Observation steps. Thought allows you to reason and analyze the current situation. Action calls the ‘Finish’ function and fill in the answer in [ ] to finish the task after Thought. The observations will be provided to you automatically after you action.

You can think step-by-step to reach the answer. Here are some examples:
{Examples}
(END OF EXAMPLES)

You are solving the following question: {Question}

Below is your progress so far:
(BEGIN)
{Scratchpad}
(END)

Please complete the current step.

C.4  Self-correct for Reflector

Instruction: In this task, you are required to solve a question with interleaving Thought, Action, and Observation steps. Thought allows you to reason and analyze the current situation. Action calls the ‘Finish’ function and fill in the answer in [] to finish the task after Thought. The observations will be provided to you automatically after you action.

You can think step-by-step to reach the answer. Here are some examples:
{Examples}
(END OF EXAMPLES)

You are solving the following question: {Question}

Below is your previous reflection that helps to revise the incorrect solutions and correctly answer the question. It localizes the errors, summarizes the potential reasons for your failure and outlines a plan to mitigate the same failure:
{Reflections}

Below is your progress so far:
(BEGIN)
{Scratchpad}
(END)

Please complete the current step.
C.5  Self-reflection and correct in one stage

Instruction: You are an advanced reasoning agent that can improve based on self-reflection. You will be given a previous reasoning trial in which you were given a question to answer. You were unsuccessful in answering the question. In a few sentences, diagnose a possible reason for failure and devise a new, concise, high-level plan that aims to mitigate the same failure. Use complete sentences.

Question: {Question}

Previous trial and your incorrect solution: {Scratchpad}

Based on your self-reflection, you can think step-by-step to generate a new answer to the question. Call the ‘Finish’ function and fill in the answer in [ ] to finish the task.

C.6  Reasoning for direct QA for SFT

Instruction: In this task, you are required to solve a question by generating the final answer directly. Call the ‘Finish’ function and fill in the answer in [ ] to finish the task.
Here are some examples:
{Examples}
(END OF EXAMPLES)

You are solving the following question: {Question}

C.7  Reflection preference annotation by GPT

Instruction: You are a helpful assistant in evaluating the quality of reflections on an unsuccessful attempt to answer a question.

You will be provided with:

Question: {Question}
Groundtruth to the question: {Answer}
Initial student’s chain of thought and answer: {Scratchpad}
Student A’s reflection: {Reflections 1}
Student B’s reflection: {Reflections 2}

Student A and Student B have both reflected on the initial student’s unsuccessful attempt to answer the question. Above are their reflections that diagnose possible reasons for failure or devise a better plan to mitigate the same failure. Please determine which student’s reflection is better.

Your response should be either "Student A" or "Student B" without providing any explanation or other words for your choice.

Do NOT say both/neither are good.
C.8 Heuristic Error Constituent Annotation for Reflection

Instruction: You are a professional data annotator specializing in reasoning and chain-of-thought rationale analysis. Your task is to analyze the thought process provided and identify any fine-grained labels based on the given reflection and error taxonomy.

# Definitions
- Thought: The reasoning steps taken by a human or model to arrive at an answer.
- Reflection: The self-reflection of the human or model on the reasoning thought process.

# Error Taxonomy
1. Mathematical Errors
   • 1-1. Calculation Error
   • 1-2. Algorithm Error

2. Logic and Reasoning Errors
   • 2-1. Flawed Rationale Error
   • 2-2. Internal Inconsistency

3. Instruction Violation
   • 3-1. Context Misinterpretation
   • 3-2. Incomplete or Irrelevant Response
   • 3-3. Format Discrepancy

4. Factual Errors
   • 4-1. Factual Errors

5. No Errors
   • 5-1. No Errors Detected

# Input
- Question: {question}
- Thought: {thought}
- Reflection: {reflection}

# Output
- Labels: [Error Type(s) Assigned]
- Rationale: [Explanation for label assignment, with specific examples]

Reviewer paper summaries: The paper presents a novel pipeline ReflectEvo and construct ReflectEvo-460k, a large self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. The authors demonstrate the potential of meta introspection of LLMs that are asked to explicitly generate reflection as an intermediate step-by-step process supervision rather  than directly mapping an initial solution to a revised solution.

This paper proposes a self-training pipeline --- ReflectEvo --- that teaches smaller language models to improve via iterative self-reflection and self-correction. It creates a large-scale self-generated dataset (ReflectEvo-460k), spanning various tasks from math to commonsense reasoning. The authors show that fine-tuning on these reflections significantly boosts reasoning performance, sometimes surpassing larger models. They also present thorough analyses of error patterns and highlight the pipeline’s versatility for long-term iterative improvement.
Summary of strengths: - The paper is very well written and easy to follow.
- The methodology is detailed and comprehensive, the experiments show impressive improvements on a variety of datasets and tasks.
- The paper also presents a dataset that could be further used for the experiments
- The detailed breakdown of error types and their impact on model performance contributes to a deeper understanding of reflection learning.

1. Diverse domains, such as coding, math, etc.
2. A useful self-reflection dataset: ReflectEvo-460k.
Summary of weaknesses: The main weaknesses of the paper are as follows:

* There are no baselines presented at all, to compara your approach with. Although I highly appreciate the method, the idea in general, for the more accurate picture it would be better to add other self-learning baselines like Star, Score, Self-correction, etc. Moreover, I would also add non-self-learning baselines in order to show the performance of this method in comparison to others. (I realise that te scores might be not really higher then thte current SotA, as I work on the similar topic, however, it is still important for the broader audience and would not affect my score, if it is not the SotA)

* The related work seciton could be improved and extended. The authors only cite numberous papers in a row, however, they do not specify the concrete algorithms and do not explain how similar/difficult their approach is. (e.g. the Star algorithm also makes use of the correct/incorrect answers during training). 

I lean towards accepting the paper and confirm that my feedback/feeling is positive. The weaknesses could be improved by the final version and/or during the discussion period  (baselines)

1. Baseline is not fair. There are several self-improve methods with and without self-reflection. The paper cites many frameworks, but it does not compare ReflectEvo directly with any known methods.

2. Why there is no coding in eval? You have coding in training data but in main results, this paper does not report any evaluation on coding benchmarks.

3. No OOD eval. ReflectEvo-460k contains all the datasets you used in the evaluation. So what is the OOD task performance?
Comments, suggestions and typos: 1. It is not very clear, what doess {Scratchpad} in Appendix C refer to.

2. line 249: please, specify in the first sentence that there are two methods: SFT and DPO, lines 249-250 are confusing like there is only one methods but there is the second one introduced later.

3. lines 298-299: does two turns mean R-C-R-C or R-C only? Figure 4 shows 6 turns. 

4. Small LLMs -> SLMs?

5. Figure 3: how do you explain the small performance gain for maths and code subsets? Adding some examples to lines 338-346 or explanations would help

6. Star algorithm (Zelikmann  et al.,2022) sited by the authors also employs .... Could you explain the main differene between them given that they both employ ....

7. Line 427, 448 - Figures 7 and 8 are in Appendix, please, mention that. Normally, the paper is self-consistent, so the results shjould be put to the main part of the paper.

8. Case studies in 4.2 do not containnovel or useful info. At least specify how many case studies were performed

9. Just in case: maybe you could be interested in other papers on the topic at the ICLR workshop: Scaling Self-Improving Foundation Models without Human Supervision, e.g. https://arxiv.org/abs/2408.08072 or https://arxiv.org/pdf/2503.08681

10. In the SFT case the authors use the quantised version of LLaMA and fine-tune it with Lora (Appendix, line 809). This means that the quality of the model with first "cut" by quantisation and then was "restored" by fine-tuning Lora. However, the authors do not report the initial score of the non-quantised model, so we could see how big the drop is. Other approaches are trained with full models that also does not guarantee the fair comparison.

1. OOD evaluation on more datasets.
2. Coding Evaluation.
3. Add more state-of-art baselines.
Overall assessment scores: 3.5

3