Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst
Authors: Hongru WANG, Deng Cai, Wanjun Zhong, Shijue Huang, Jeff Z. Pan, Zeming Liu, Kam-Fai Wong
Accepted at: acl25

Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalyst

Hongru Wang*, Deng Cai*, Wanjun Zhong*, Shijue Huang*, Jeff Z. Pan*, Zeming Liu*,†, Kam-Fai Wong*,‡,§
*The Chinese University of Hong Kong, †ByteDance
‡The University of Edinburgh, §Beihang University
†MoE Key Laboratory of High Confidence Software Technologies
zmliu@buaa.edu.cn, {hrwang, kfwong}@se.cuhk.edu.hk

Abstract

Inference-time scaling has attracted much attention which significantly enhance the performance of Large Language Models (LLMs) in complex reasoning tasks by increasing the length of Chain-of-Thought. These longer intermediate reasoning rationales embody various meta-reasoning skills in human cognition, such as reflection and decomposition, being difficult to create and acquire. In this work, we introduce Self-Reasoning Language Model (SRLM), where the model itself can synthesize longer CoT data and iteratively improve performance through self-training. By incorporating a few demonstration examples (i.e., 1,000 samples) on how to unfold hidden reasoning chains from existing responses, which act as a reasoning catalyst, we demonstrate that SRLM not only enhances the model’s initial performance but also ensures more stable and consistent improvements in subsequent iterations. Our proposed SRLM achieves an average absolute improvement of more than +2.5 points across five reasoning tasks: MMLU, GSM8K, ARC-C, Hellaswag, and BBH on two backbone models. Moreover, it brings more improvements with more times of sampling during inference, such as absolute +7.89 average improvement with 64 sampling times, revealing the in-depth, diverse and creative reasoning paths in SRLM against the strong baseline.

1 Introduction

Recent studies have demonstrated that inference-time scaling (OpenAI, 2024) effectively improves performance of Large Language Models (LLMs) in various reasoning tasks, such as mathematics and complex question answering, by extending the length of Chain-of-Thought (CoT) (Wei et al., 2022) reasoning rationales (Qwen, 2024; DeepSeek-AI et al., 2025). With longer CoTs during inference, these LLMs could explore more creative and diverse reasoning rationales while assembling various meta-reasoning skills observed in human cognition (Gigerenzer, 1991; Johnson-Laird, 1988; Dye, 2011; Wang et al., 2024b), such as reflection and decomposition (Wang et al., 2024a, 2025a,b), resulting in better performance. However, the scarcity of these longer CoT data remains a significant obstacle to advancing inference-time scaling, especially for instructions without verifiable answers in the general instruction-tuning data.

Several prior studies have explored various approaches to get better CoT tuning datasets, where most of them utilize either the LLM itself (Wang et al., 2023c; Huang et al., 2023; Wang et al., 2023a) or external reasoning models (Li et al., 2024a; Hu et al., 2024) to generate (or refine) new (or existing) responses in the instruction-tuning dataset, leading to great improvements at various downstream tasks. Despite the effectiveness of these proposed methods, they still face several limitations. On the one hand, most of them focus on questions with verifiable answer such as math and code (Huang et al., 2023; DeepSeek-AI et al., 2025), being infeasible for general instruction-tuning dataset. On the other hand, another line of work typically assumes access to more powerful models to refine each sample iteratively (Xu et al., 2023; Li et al., 2024a). Such methods suffer from performance plateaus or even degradation (Ding et al., 2024) and are inherently constrained by the capability ceiling of the powerful model.

![A diagram comparing a language model and a self-reasoning language model, showing the process of meta-reasoning prompting and unfolding reasoning chains.](page_1012_370_324_246.png)

Figure 1: The comparison between (a) language model; and (b) self-reasoning language model where the later is capable to generate more longer reasoning chains alongside the final answer.
To this end, we present Self-Reasoning Language Models (SRLM), which is capable to self-unfolding its own reasoning rationales and iteratively optimize itself, leading to enhanced overall capability. Specifically, we first create only few reasoning catalyst data that compose the demonstrations of how to enrich shorter CoT rationales into more longer and comprehensive CoT with the augmentation of various meta-reasoning skills. After incorporating the reasoning catalyst data with the original instruction-tuning data, the tuned model not only inherit the basic reasoning capabilities from the instruction-tuning dataset, but also learn how to refine reasoning simultaneously, resulting in Self-Reasoning Language Models. Consequently, the SRLM can refine its own reasoning rationales at each iteration with the processing of reasoning expansion and selection. During this process, the model generates enriched reasoning rationale candidates for the same instructions in the original instruction-tuning dataset. These rationale pairs are then filtered and selected using three proposed selectors without any prior assumption about the instruction and answer. Finally, the newly selected instruction-tuning dataset is combined with the reasoning catalyst data to create the training data for the next iteration of SRLM, which is initialized from the same base model. To conclude, our contributions can be summarized as follows:
• We introduce Self-Reasoning Language Models (SRLM), which is capable to act as both: 1) a response generation model (i.e., learn to reason); and 2) reasoning models to refine its own reasoning rationales (i.e., how to reason).
• We present reasoning catalyst data, function as demonstrations to guide the LLM to unfold hidden reasoning chains with the augmentation of various meta-reasoning skills such as reflection and decomposition.
• Our experiments demonstrate that incorporating a small portion of reasoning catalyst data (less than 0.02%) enables small-size SRLMs to generate higher-quality instruction-tuning datasets compared to those generated by GPT-4o, while also achieving better performance and more stable improvements across iterations on various reasoning benchmarks.

2 Related Work

Synthetic SFT Data. Previous research on instruction tuning has primarily utilized human-annotated data to create large, high-quality datasets (Wang et al., 2022), being both time-consuming and labor-intensive. Therefore, recent studies turn to leverage more advanced models to automatically generate instruction-tuning datasets (Wang et al., 2023c; Peng et al., 2023; Li et al., 2024a). For instance, Magpie (Xu et al., 2024) leverage open-source models to generate more diverse user queries and responses by only feeding the left-side templates up to the position reserved for user messages as input. Moreover, there is another line of works focused on refining existing instruction-tuning data using powerful models, such as WizardLM (Xu et al., 2023) and Reflection-tuning (Li et al., 2024a) and NILE (Hu et al., 2024). In detail, Li et al. (2024a) utilize GPT-3.5 to reflect on existing instructions and responses while Hu et al. (2024) use GPT4 to capture the knowledge gap between internal parametric knowledge in LLMs and external world knowledge in instruction-tuning data.

In addition to constructing datasets for SFT purpose, there are also efforts in constructing CoT related datasets for evaluation purposes, such as for multi-modal LLMs (Zhou et al., 2025) and for attributed question answering (Hu et al., 2025).

Self-Improved LLMs. Recent studies have demonstrated that LLMs can refine themselves using self-generated data, leading to enhanced performance and efficiency without human intervention, resulting in self-improving LLMs (Huang et al., 2023). It is crucial to select high-quality samples from the vast amounts of generated data. There are different ways to acquire the quality signals of data samples: 1) LLMs (Yuan et al., 2024); 2) ranking or reward models (Dong et al., 2023; Lu et al., 2024); and 3) some automatic methods such as execution feedback (Haluptzok et al., 2022) or self-consistency in multiple reasoning paths (Wang et al., 2023b). One distinct aspect of our work is that we are working on general instruction-following data optimization iteratively, where the answers are not verifiable like math and coding problems, making most of previous studies infeasible and unaffordable (Huang et al., 2023; Li et al., 2024a).

On the other hand, there have been studies that make use of external knowledge (Pan et al., 2023) to improve the reasoning and planning abilities of LLMs, such as using graph patterns within knowledge graphs (Pan et al., 2017b,a) for CoT learn-
ing (Wang et al., 2024c), using knowledge graph triples extracted from passages for improving multi-hop QA (Shen et al., 2025), and using external PDDL planner to improve both planning performance and planning explainability for LLMs (Vyas et al., 2025).

3 Method

3.1 Task Formulation

Given a large language model \( \mathcal{M} \) and the original instruction-tuning dataset \( D^0 = \{(x_i, r_i^0, y_i)\} \) at time step 0, where \( x_i \) is the instruction, \( r_i \) is the chain-of-thought rationale (Wei et al., 2022) ¹ and \( y_i \) represents the final answer in the \( i \)th sample. The goal of self-improved LLMs is to iteratively refine the instruction-tuning dataset from the time step 0, leading to better dataset \( D^t = \{(x_i, r_i^t, y_i)\} \) at time step \( t \), ultimately resulting in enhanced performance at downstream tasks.

3.2 Framework

As shown in Figure 2, there are two main phases in our method. In phase 1, we leverage existing LLMs to acquire seed data of reasoning augmentation, and function as reasoning catalyst to enlighten the models to enrich existing reasoning rationales (\S3.2.1). In phase 2, the model then is fine-tuned using both original instruction-tuning data and reasoning catalyst data, and iteratively refined using self-generated new rationales, resulting in self-improved models (\S3.2.2).

3.2.1 Reasoning Catalyst Acquisition

There is a Chinese proverb: “It’s better to teach a man to fish than to give him fish.” Similarly, in the context of LLMs, rather than directly copying reasoning capabilities from existing instruction-tuning datasets, it is crucial to teach LLMs how to reason and refine their reasoning based on specific problems and existing reasoning rationales. To achieve this goal, we first carefully design meta-reasoning prompt ², which is composed of different meta-reasoning skills such as reflection, detail, and alternative-thinking, inspired by meta-reasoning theory in cognitive science (Zhou et al., 2024). Then we leverage these meta-reasoning skills to unfold hidden reasoning chains based on given instruction and previously reasoning rationales \( r_j^0 \), resulting in enriched reasoning rationales \( r_j^* \) as follows:

\[
r_j^* = \mathcal{M}_{meta}(x_j, r_j^0)
\]

Here the \( \mathcal{M}_{meta} \) could be human experts or any off-the-shelf models, \( r_j^* \) starts with <thoughts> and ends with </thoughts>. All other meta-reasoning skills used in the thoughts also be clearly indicated in a similar manner for better understanding and analysis. Therefore, it can effectively fill in missing details within existing rationales while preserving their core semantic meaning. Moreover, it is desirable to introduce new rationales (i.e., using alternative-thinking), correct errors in previous rationales (i.e., using reflection), and other hidden reasoning chains with different skills. Thus, this process provides exceptional flexibility and robustness to address errors, misleading reasoning paths, and unclear solutions in the previous reasoning rationale. Since we aim to teach the model how to reason instead of directly telling them the correct reasoning paths for all instructions. Therefore, we only incorporate a few samples (only 0.02% in the main experiment) as reasoning catalyst data to enforce the models’ generalization to other unseen instructions.

3.2.2 Self-Improved LLMs

Given the original instruction-tuning dataset \( D^t = \{(x_i, r_i^t, y_i)\} \) (\( t = 0 \) at the beginning) and fixed few reasoning catalyst data \( R = \{(x_j, r_j^t, r_j^*)\} \), the original base LLM \( \mathcal{M}^3 \) is then tuned using both types of data to learn reasoning and how to reason simultaneously, resulting in new LLM \( \mathcal{M}_t \). Then we enrich the reasoning rationales in original instruction-tuning datasets (i.e., iterative reasoning expansion) and select better rationales to form the instruction-tuning dataset in the next step \( t + 1 \) (i.e., iterative reasoning selection), which completes the whole iteration.

Iterative Reasoning Expansion. We directly prompt the \( \mathcal{M}_t \) model to generate enriched reasoning rationales given previous reasoning \( r_j^t \) and \( x_j \) for each sample in the instruction-tuning dataset, distinguishing from reasoning catalyst acquisition.

\[
r_{i,m}^{t+1} = \mathcal{M}_t(x_i, r_i^t), \quad \forall m \in \{1, 2, \ldots, N\}.
\]

¹Note this is optional, take it as \( y_i \) if it does not exist.
²The prompt can be found in Table 6.
³We note we always finetune the model start from base models instead of continual tuning.
Figure 2: The framework of our proposed Self-Reasoning Language Models, which consists of two phrases.

\[
r^* = \arg\max_{j \in \{t, t+1\}} P(y_i|x_i, r_j^i)
\] (3)

• On-policy Selector. Similar with off-policy selector, but we use the Self-Reasoning Language Model at last iteration to calculate the probability.

Afterwards, we can get the new generated instruction-tuning dataset \( D^{t+1} = \{(x_i, r_i^{t+1}, y_i)\} \) for next iteration, which completes the whole self-improved processing.

4 Experiments

4.1 Set Up

Models. We mainly use two different base models: Llama3.1-SB (Grattafiori et al., 2024) and Mistral-7B-V0.3 (Jiang et al., 2023) since both of them are open-sourced and support longer context.

Data. There are two different parts of data used in our experiments: 1) instruction-tuning data: we first randomly sample 50k instances from Magpie-reasoning-150k (Xu et al., 2024) and then adopt reflection-tuning (Li et al., 2024a) to reflect its instruction and the responses using GPT-4o model; 2) reasoning catalyst data: we use the same GPT-4o model\footnote{We use this model since our initial instruction-tuning data also is generated by this model.} to generate only additional 1k samples (only 0.02% of whole training dataset) to enrich the original rationales with various meta-reasoning skills. We compare the performance of various models fine-tuned using our method against two above instruction-tuning datasets 50k Magpie data and 50k Reflection-tuning data (i.e., Magpie and Reflect-tuning).

where \( N \) represents the number of sampling attempts, designed to enhance diversity and produce higher-quality output.

Iterative Reasoning Selection. It is important to determine which rationale is better for current instruction given the pair of \((x_i, r_i^t, y_i)\) and \((x_i, r_i^{t+1}, y_i)\). Thus we carefully design three different selectors to investigate corresponding effects as shown in Figure 3:

• Length Selector. We consider a longer rationale to be better because it provides more detailed insights into the previous reasoning. Even if it contains errors or irrelevant information, it can be refined and corrected in subsequent iterations.

• Off-policy Selector. We leverage the original base model to calculate the the probability of the answer \( y_i \) given the instruction \( x_i \) and different rationales, and then choose the rationale that has a higher chance of leading to the final answer.

Figure 3: Three different types of reasoning selectors.
Figure 4: The performance of Self-Reasoning Language Models (SRLM) across five different benchmarks, along with the average performance (Avg.). We report the best performance during the five iterations for each selector, and the detailed experimental logs can be referred to at Table 7. We also run significant test which showcases our method significantly outperforms the reflection-tuning with \( p < 0.05 \).

Evaluation and Metrics. We choose several existing benchmarks to validate the effectiveness and generalization of our proposed method. Specifically, we mainly focus on tasks necessitate complex reasoning such as general reasoning benchmarks: MMLU (Hendrycks et al., 2021), mathematical reasoning: GSM8K (Cobbe et al., 2021), commonsense reasoning: ARC-C (Clark et al., 2018), HellaSwag (Zellers et al., 2019) and other challenging tasks: BBH (Suzgun et al., 2022). We choose Accuracy as the evaluation metric and use the zero-shot prompting method to evaluate the performance of fine-tuned models, following lots of previous works (Grattafiori et al., 2024).

Implementation Details. We train 1,000 steps on 16 A100 80G GPUs for each method to ensure a fair comparison, considering the slightly different sizes of the dataset. During the training, we set the learning rate as 1e-05, batch size as 4, and gradient accumulation steps as 2 following Zheng et al. (2024). After obtaining the SRLM, we set the sample times \( N \) in Eq 2 as 5, and the maximum iteration times as 5 for all selectors. During the inference, we adopt the same setting with temperature as 0.2, top p as 0.9, and max tokens as 8192. We denote the iterations of SRLM as \( \mathcal{M}_{0,1,...,t} \).

4.2 Main Results

The main results are shown in Figure 4. Based on that, we have the following key observations.

Incorporating additional reasoning expansion samples into the instruction-tuning dataset acts as a catalyst, leading to improved performance. As we can find in the Avg. and most benchmarks, SRLM (\( \mathcal{M}_0 \)) achieves better performance than Reflection-tuning and Magpie in both models. The only two exceptions are GSM8K and HellaSwag, due to the relatively smaller number of instances of mathematical and commonsense reasoning in the reasoning expansion data.

Small Self-Reasoning Language Models can generate high-quality rationales iteratively. It is evident that SRLM with different selectors at various iterations all leads to improvement compared with the initial SRLM (\( \mathcal{M}_0 \)). Since the only difference between these SRLMs is the used instruction-tuning dataset, it strongly demonstrates that the SRLM with a small size (7B or 8B) can generate better instruction-tuning samples than GPT-4o (i.e., the instruction-tuning data at the \( 0_{th} \) step). It is encouraging to find that synthesized data by small language models still can provide stronger supervision signals than the latest powerful models. In addition, we find that Llama3.1-8B achieves more consistent and higher improvement than Mistral-7B since it generates better samples due to more strong base capabilities.

All types of selectors outperform SRLM (\( \mathcal{M}_0 \)), with the on-policy selector excelling in MMLU, while the length selector performs better on HellaSwag and BBH. Generally, Length selector achieves the best performance at Avg. for both models, revealing the excellence of this simple strategy in SRLM. In detail, we observe a seesaw
phenomenon across different datasets and models. For example, with the same Mistral-7B model, the on-policy selector performs better on ARC-C but worse on HellaSwag. However, the on-policy selector on the Llama3.1-8B model performs worse compared to the other selectors on ARC-C. This can be attributed to the interaction effects between the models and datasets, highlighting the challenges in evaluating self-improving LLMs.

5 Analysis

5.1 Effects of Sampling Times.
In the main experiments, we mainly consider the best-of-1 performance of different methods. Alternatively, the best-of-N performance has been proved to be an effective way to showcase the depth and creativity of the model’s problem-solving potential in terms of reasoning (Chow et al., 2024). In this way, we conduct the multiple sampling ranging from 1 to 64, aiming to investigate the performance under the best-of-N setting. Figure 5 shows the performance of our proposed SRLM (\( \mathcal{M}_3 \) with length selector) and the baseline. The full experimental results can be found in Table 12 and fitting functions can be found in Appendix. First of all, it is worth noting that SRLM achieves much better performance with more sampling times on all benchmarks, as the gains get bigger and bigger. This is further supported by the larger coefficient of the logarithmic term in the fitted function of SRLM. Secondly, we can observe that MMLU achieves the highest growth rate and then followed by ARC-C and HellaSwag. As shown in Table 12, SRLM achieves 94.24% on the Avg. performance with 64 sampling times with MMLU (96.15% (+11.35)), ARC-C (98.72% (+9.56)), HellaSwag (97.62% (+8.38)). Based on above observations, it is believed that our proposed SRLM can explore more depth, diverse and creative reasoning paths toward the final correct solution, revealing its promising potential.

5.2 Effects of Different Number of Iterations
Figure 6 presents the Avg. Performance under different iterations with three types of selectors. There are several key findings. On the one hand, on-policy methods typically perform poorly at the beginning of the iteration process. This can be attributed to the limited number of samples - only a few hundred - selected during the initial iteration. Moreover, as the process progresses from the first to the third iteration, the number of selected samples increases with the upward trend of the performance observed for both models in the figure. After the third iteration, performance diverges: the number of selected samples decreases for Mistral-7B, leading to a performance decline, while it remains constant for Llama3.1-8B, resulting in stable and saturated performance.

On the other hand, the length selector mostly leads to the best performance of the three selectors, and we also observe model degradation no matter which selector is chosen. After checking the number of selected numbers at each iteration, we find that the off-policy selector selects the most samples in the first iteration, nearly matching the original dataset size, but drops sharply to around 10k and continues decreasing. In contrast, the length-based selector also decreases, but at a slower and steadier rate. The last thing we want to emphasize is that it is evident that SRLM is capable of correcting previous rationales considering that there may be errors in the enriched rationales in previous steps, as we can find that length and off-policy selector achieve the best performance at \( 5_{th} \) iteration with the worst performance at \( 4_{th} \).

5.3 Effects of Reasoning Catalyst Data
We also investigate several factors of reasoning catalyst data that may have serious effects on the final performance of SRLM, including the ablation study, different sizes, and sources of the data.

Ablation Study. We first directly remove the catalyst data from each iteration and train the model independently following the same setting in the main experiments. Figure 7 presents the final results at different iterations. Two obvious observations can be drawn from the results. First of all, the performance at later iterations mostly outperforms the performance at the initial iteration (iteration at \( t = 0 \)) no matter incorporates a reasoning catalyst or not. This provides additional direct evidence that the newly generated instruction-tuning datasets at iterations (\( t > 1 \)) are better than the GPT-4o generated dataset at iteration \( t = 0 \). Secondly, when incorporating reasoning catalyst data, the improvements become more stable and consistent. We can find that IT + Catalyst mostly achieves better performance than only instruction-tuning data, and it brings consistent improvement while the performance using instruction-tuning data only may fluctuate.
Figure 5: The performance of the baseline and our proposed SRLM (\( \mathcal{M}_3 \) with length selector) different sampling times ranging from 1 to 64 (i.e., 1, 2, 4, 8, 16, 32, 64) on five various benchmarks and the Avg. performance.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>57.08</td>
    <td>66.49</td>
    <td>67.32</td>
    <td>50.34</td>
    <td>30.09</td>
    <td>54.26</td>
  </tr>
  <tr>
    <td>SRLM - \( \mathcal{M}_0 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 1k</td>
    <td>57.60</td>
    <td>64.59</td>
    <td><b>70.14</b></td>
    <td><b>49.23</b></td>
    <td>31.46</td>
    <td><b>54.66</b></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>57.46</td>
    <td>66.26</td>
    <td>67.15</td>
    <td>47.87</td>
    <td>25.23</td>
    <td>52.79</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td><b>58.81</b></td>
    <td><b>67.63</b></td>
    <td>67.41</td>
    <td>46.30</td>
    <td><b>32.29</b></td>
    <td>54.49</td>
  </tr>
  <tr>
    <td>SRLM - \( \mathcal{M}_1 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 1k</td>
    <td>58.31</td>
    <td>65.73</td>
    <td>72.57</td>
    <td><b>48.05</b></td>
    <td><b>34.08</b></td>
    <td><b>55.31</b></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>58.03</td>
    <td><b>67.85</b></td>
    <td><b>74.23</b></td>
    <td>46.54</td>
    <td>32.51</td>
    <td><b>55.83</b></td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td><b>58.55</b></td>
    <td>67.63</td>
    <td>73.29</td>
    <td>45.27</td>
    <td>31.78</td>
    <td>55.30</td>
  </tr>
  <tr>
    <td>SRLM - \( \mathcal{M}_2 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 1k</td>
    <td>58.27</td>
    <td>69.22</td>
    <td>72.70</td>
    <td><b>51.99</b></td>
    <td>32.03</td>
    <td><b>56.84</b></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>56.96</td>
    <td>67.17</td>
    <td>71.16</td>
    <td>47.11</td>
    <td>28.58</td>
    <td>54.20</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td><b>58.34</b></td>
    <td>67.40</td>
    <td><b>73.38</b></td>
    <td>51.06</td>
    <td>33.18</td>
    <td>56.67</td>
  </tr>
  <tr>
    <td>SRLM - \( \mathcal{M}_3 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 1k</td>
    <td>57.91</td>
    <td>67.25</td>
    <td>72.76</td>
    <td><b>52.88</b></td>
    <td><b>34.06</b></td>
    <td><b>56.36</b></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>57.30</td>
    <td>65.73</td>
    <td>70.73</td>
    <td>46.97</td>
    <td>30.61</td>
    <td>54.27</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td><b>58.24</b></td>
    <td>67.17</td>
    <td>70.56</td>
    <td>50.87</td>
    <td>32.83</td>
    <td>55.93</td>
  </tr>
</table>

Table 1: The results using different size of reasoning catalyst data based on Llama3.1-8B model with 1,000 steps optimization on three iterations (\( \mathcal{M}_{0,1,2,3} \)). We bold the best performance at each iteration.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>57.08</td>
    <td>66.49</td>
    <td>67.32</td>
    <td>50.34</td>
    <td>30.09</td>
    <td>54.26</td>
  </tr>
  <tr>
    <td>T1 + New Cat</td>
    <td>57.67</td>
    <td>68.21</td>
    <td>70.56</td>
    <td>49.52</td>
    <td>31.52</td>
    <td>55.30</td>
  </tr>
  <tr>
    <td>T1 + New Cat</td>
    <td>56.77</td>
    <td>68.31</td>
    <td>69.28</td>
    <td>45.53</td>
    <td>33.40</td>
    <td>54.66</td>
  </tr>
  <tr>
    <td>T1 + New Cat</td>
    <td>58.17</td>
    <td>67.40</td>
    <td>70.31</td>
    <td>48.10</td>
    <td>31.28</td>
    <td>55.05</td>
  </tr>
  <tr>
    <td>T1 + New Cat</td>
    <td>56.45</td>
    <td>66.49</td>
    <td>69.62</td>
    <td>46.47</td>
    <td>30.27</td>
    <td>53.86</td>
  </tr>
</table>

Table 2: The performance of SRLM with length selector on Llama3.1-8B using New 1k reasoning catalyst data with 1000 steps optimization.

Figure 6: The effects of different number of iterations in Self-Reasoning Language Models.

Figure 7: The ablation study by removing the reasoning catalyst data at each iteration. We report the Avg. performance here and the detailed results can be found in Table 9.

Size of Catalyst. We additionally random sample 5,000 and 10,000 from the original instruction-tuning dataset and build corresponding catalyst data. We also train the SRLM with 1,000 optimization steps to make a fair comparison following the same setting in the main experiments. Table 1
<table>
  <tr>
    <th>Method</th>
    <th>Arena-Hard</th>
    <th>Alpaca Eval V2</th>
  </tr>
  <tr>
    <td>Alpaca-7B</td>
    <td>-</td>
    <td>2.59</td>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>6.40</td>
    <td>5.92</td>
  </tr>
  <tr>
    <td>SRLM (Ours)</td>
    <td>8.60</td>
    <td>9.21</td>
  </tr>
</table>

Table 3: The win rate of SRLM and Reflection-tuning

shows the final results. Generally, adding more reasoning catalyst data does not significantly improve average performance. However, it can lead to noticeable gains on specific datasets.

In detail, SRLM (\( M_0 \)) with 10,000 reasoning catalyst data achieves best performance at MMLU, GSM8K and BBH. Moreover, it continues to show superior results on MMLU with additional training iterations. This evidence suggests that incorporating more catalyst reasoning data benefits general reasoning tasks that require comprehensive evaluation, rather than domain-specific reasoning tasks. Furthermore, we can find that it offers limited benefits for commonsense reasoning tasks like ARC-C and HellaSwag, as the SRLM with just 1,000 data points consistently achieves peak performance across all iterations. We attribute this to relatively smaller ratio of commonsense reasoning samples in the sampled dataset.

New Catalyst. We also investigate the effects of applying out-of-distribution reasoning catalysts. To achieve this, we construct the catalyst using a separate randomly sampled 1,000 instruction-tuning dataset that has no overlap with the existing datasets. Table 2 shows the performance at each iteration. It is shown that incorporating additional out-of-distribution reasoning catalyst data also brings improvement compared with the only 4o-generated instruction-tuning data. However, consistent improvement in later iterations is not guaranteed as we observe only partial gains in certain benchmarks rather than a significant increase in overall average performance. We consider this due to difficulty in refining the rationale for existing instructions given out-of-distribution demonstrations.

Update Catalyst. Until now, we only update the instruction-tuning data set as shown in Figure 2. Alternatively, we can also update reasoning catalyst data by utilizing the newly generated rationales \( r_j^t \). In detail, we can update the \( r_j^t \) in Figure 2 only with \( r_j^{t+1} \) or update \( r_j^0 \) and \( r_j^* \) with \( r_j^t \) and \( r_j^{t+1} \) at the same time with simple length selector. However, we find that both of these two updating mechanisms can not bring significant improvement and even downgrade the performance at the later iterations as shown in Table 11. We consider the primary reason may be that LLMs tend to become constrained by their own outputs, limiting exploration of diverse solutions when more self-generated rationales are included. Moreover, the reasoning catalyst data introduced at the initial stage may be nearly optimal, thus being difficult to beat with only length selector. More analysis can be found in Appendix.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <th colspan="7">Reflection-tuning</th>
  </tr>
  <tr>
    <td>SRLM (Ours)</td>
    <td>51.44</td>
    <td>48.07</td>
    <td>66.64</td>
    <td>37.24</td>
    <td>31.71</td>
    <td>47.02</td>
  </tr>
  <tr>
    <td>T1 + Cat.</td>
    <td><b>55.34</b></td>
    <td>49.51</td>
    <td><b>69.28</b></td>
    <td>42.86</td>
    <td>30.57</td>
    <td>49.58</td>
  </tr>
  <tr>
    <td>T2 + Cat.</td>
    <td>53.85</td>
    <td><b>51.78</b></td>
    <td>67.58</td>
    <td>44.92</td>
    <td><b>34.07</b></td>
    <td>50.44</td>
  </tr>
  <tr>
    <td>T3 + Cat.</td>
    <td>52.81</td>
    <td>49.81</td>
    <td>68.17</td>
    <td><b>47.35</b></td>
    <td>31.21</td>
    <td>49.87</td>
  </tr>
  <tr>
    <td>T4 + Cat.</td>
    <td>52.31</td>
    <td>47.08</td>
    <td>66.47</td>
    <td>45.03</td>
    <td>28.53</td>
    <td>47.89</td>
  </tr>
</table>

Table 4: The results of Llama3.1-8B on instruction-tuning dataset (collected by length selector) of each iteration for Alpaca (Taori et al., 2023) dataset with 2,000 optimization steps.

LLM-as-a-Judge Evaluation. Besides the exact match evaluation of five various reasoning benchmarks, we also conduct LLM-as-a-judge evaluation on two popular benchmarks: Arena-Hard (Li et al., 2024b) and Alpaca Eval V2 (Li et al., 2023). Following the default setting, we utilize gpt-4-1106-preview to compare the responses generated by different methods (i.e., Reflection-tuning and SRLM - \( \mathcal{M}_3 \) with length selector) against the response generated by the reference model (i.e., gpt-4-0314 in Arena-Hard, gpt-4-turbo in Alpaca Eval). Table 3 shows the win rate of response pairs. It is clear that our proposed SRLM achieves higher win rates at both datasets, leading to relatively 34% and 56% improvement on Arena-Hard and Alpaca Eval V2 respectively.

Effects of Instruction-tuning Data. To further validate the robustness of our proposed SRLM, we also conduct our experiments on another popular instruction-tuning dataset: Alpaca (Taori et al., 2023). Following the same procedure, we finetuned our models for all datasets in 2,000 steps with batch size at 2 due to more longer responses in original Alpaca dataset. Table 4 shows the final results. It is clear that our method still outperforms the competitive 4o-generated data by reflection-tuning, and achieves best performance at the first iteration with the absolute +3.42 improvement on average performance of five different benchmarks. This result
Figure 8: The distribution of various meta-reasoning skills employed at the training and inference of SRLM.

evidences the robustness and generalization of our proposed method on different datasets.

Meta-Reasoning Skills Distribution. We also study the distribution of various meta-reasoning skills employed during training and inference stage of SRLM. Figure 8 shows the results. On the one hand, it is obvious that detail is the dominant reasoning skills used to unfold hidden reasoning rationales, and then followed by decomposition, reflection, no matter in training and inference stage. On the other hand, the striking similarity in reasoning skill distribution patterns between training and inference phases strongly suggests that instruction-tuning data construction critically influences model behavior. The distribution of various reasoning skills across different benchmarks during inferences also shows similar trends (as shown in Figure 9).

7 Conclusion

In this paper, we present self-reasoning language models (SRLMs), which are capable of generating responses and unfolding hidden reasoning chains at the same time, leading to self-evolved models that enhance performance across various reasoning benchmarks. While there are many avenues left unexplored, we believe this is exciting and promising since the SRLMs are better able to discover diverse and creative reasoning paths in future iterations for improving instruction following – a kind of virtuous circle.

Limitations

Despite SRLM demonstrates effectiveness across various models and benchmarks, due to the limitation of computation resources, we do not test SRLM on more larger LLMs such as 70B or even larger models. In addition, SRLM can be applied to datasets or benchmarks with verifiable answers and trained continuously, as it is independent of the data used and training methods. We leave these aspects for future work, as they are beyond the scope of this paper.

Ethical Considerations

In conducting our research, we have thoroughly reviewed and ensured compliance with ethical standards. Our study utilizes existing datasets, which have been publicly available and previously vetted for ethical use. These datasets have been carefully selected to avoid any form of offensive or biased content. Therefore, we consider that our research does not present any ethical issues.

Acknowledgments

This work was partially supported by Hong Kong RGC GRF No. 14206324, CUHK direct grant No. 4055209, and CUHK Knowledge Transfer Project Fund No. KPF23GWP20.

References

Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust. 2024. Inference-aware fine-tuning for best-of-n sampling in large language models. Preprint, arXiv:2412.15287.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichihiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.
Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qushu Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqiong Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zhihu Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. 2025. Deeppeek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Yiwen Ding, Zhiheng He, Wei He, Zhuyouan Li, Yitao Zhai, Xiaowei Shi, Xunliang Cai, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Mitigating tail narrowing in llm self-improvement via socratic-guided sampling. Preprint, arXiv:2411.00750.

Hanzhe Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767.

Vanessa Dye. 2011. Reflection, reflection, reflection. i’m thinking all the time, why do i need a theory or model of reflection?. Developing Reflective Practice: A guide for beginning teachers. Maidenhead: McGraw-Hill Education, pages 217–234.

Gerd Gigerenzer. 1991. From tools to theories: A heuristic of discovery in cognitive psychology. Psychological review, 98(2):254.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, and Abhishek Kadian et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Patrick Halupczok, Matthew Bowers, and Adam Tauman Kalai. 2022. Language models can teach themselves to program better. arXiv preprint arXiv:2207.14502.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.

Minda Hu, Qiyuan Zhang, Yufei Wang, Bowei He, Hongru Wang, Jingyan Zhou, Liangyou Li, Yasheng Wang, Chen Ma, and Irwin King. 2024. Nile: Internal consistency alignment in large language models. Preprint, arXiv:2412.16686.

Nan Hu, Jiaoyan Chen, Yike Wu, Guilin Qi, Hongru WANG, Sheng Bi, Yongrui Chen, Tongtong Wu, and Jeff Z. Pan. 2025. Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphs. In In the Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).

Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Philip Nicholas Johnson-Laird. 1988. The computer and the mind: An introduction to cognitive science. Harvard University Press.

Ming Li, Lichang Chen, Jiuhai Chen, Shuai He, Jiuxiang Gu, and Tianyi Zhou. 2024a. Selective reflection-tuning: Student-selected data recycling for LLM instruction-tuning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 16189–16211, Bangkok, Thailand. Association for Computational Linguistics.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. Preprint, arXiv:2406.11939.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.
Jianqiao Lu, Zhiyang Dou, WANG Hongru, Zeyu Cao, Jianbo Dai, Yunlong Feng, and Zhijiang Guo. 2024. Autopsy: Automated process-supervised verifier. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

OpenAI. 2024. Learning to reason with llms.

Jeff Z. Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omelianenko, Wen Zhang, Matteo Lissandrini, Russa Biswas, Gerard de Melo, Angela Bonifati, Edilira Vakaj, Mauro Dragoni, and Damien Graux. 2023. Large Language Models and Knowledge Graphs: Opportunities and Challenges. Transactions on Graph Data and Knowledge, pages 1–38.

J.Z. Pan, D. Calvanese, T. Eiter, I. Horrocks, M. Kifer, F. Lin, and Y. Zhao, editors. 2017a. Reasoning Web: Logical Foundation of Knowledge Graph Construction and Querying Answering. Springer.

J.Z. Pan, G. Vetere, J.M. Gomez-Perez, and H. Wu, editors. 2017b. Exploiting Linked Data and Knowledge Graphs for Large Organisations. Springer.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. Preprint, arXiv:2304.03277.

Qwen. 2024. Qwq: Reflect deeply on the boundaries of the unknown. Accessed: 2025-02-01.

Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Paschal Merita, Shriram Pirmamayanagam, Enting Chen, Damien Graux, Andre Melo, Ruofei Lai, Zeren Jiang, Zhongyang Li, YE QI, Yang Ren, Dandan Tu, and Jeff Z. Pan. 2025. GeAR: Graph-enhanced Agent for Retrieval-augmented Generation. In In the Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025).

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Kaustubh Vyas, Damien Graux, Yijun Yang, Sebastien Montella, Chenxin Diao, Wendi Zhou, Pavlos Vougiouklis, Ruofei Lai, Yang Ren, Keshuang Li, and Jeff Z. Pan. 2025. From an LLM Swarm to a PDLL-empowered Hive: Planning Self-executed Instructions in a Multi-modal Jungle. In In the Proceedings of the International Conference on Learning Representations (ICLR 2025).

Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. 2024a. Empowering large language models: Tool learning for real-world interaction. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2983–2986, New York, NY, USA. Association for Computing Machinery.

Hongru Wang, Huimin Wang, Lingzhi Wang, Minda Hu, Rui Wang, Boyang Xue, Yongfeng Huang, and Kam-Fai Wong. 2024b. Tpe: Towards better compositional reasoning over cognitive tools via multi-persona collaboration. In Natural Language Processing and Chinese Computing: 13th National CCF Conference, NLPCC 2024, Hangzhou, China, November 1–3, 2024, Proceedings, Part II, page 281–294, Berlin, Heidelberg. Springer-Verlag.

Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zehong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. 2023a. Cue-CoT: Chain-of-thought prompting for responding to in-depth dialogue questions with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12047–12064, Singapore. Association for Computational Linguistics.

Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guanhua Chen, and Kam-Fai Wong. 2025a. Self-DC: When to reason and when to act? self divide-and-conquer for compositional unknown questions. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6510–6525, Albuquerque, New Mexico. Association for Computational Linguistics.

Junjie Wang, Mingyang Chen, Binbin Hu, Dan Yang, Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff Z. Pan, Wen Zhang, and Hua-jun Chen. 2024c. Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs. In In the Findings of the Association for Computational Linguistics: EMNLP 2024 (EMNLP 2024 Findings), pages 7813–7835.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. Preprint, arXiv:2203.11171.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023c. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormalabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak,
Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025b. Thoughts are all over the place: On the underthinking of o1-like llms. Preprint, arXiv:2501.18585.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. Preprint, arXiv:2304.12244.

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. Preprint, arXiv:2406.08464.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. Preprint, arXiv:2401.10020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyuan Luo. 2024. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 400–410, Bangkok, Thailand. Association for Computational Linguistics.

Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. 2024. Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620.

Xiongtao Zhou, Jie He, Lanyu Chen, Jingyu Li, Haojing Chen, Victor Gutiérrez-Basulto, Jeff Z. Pan, and Hanjie Chen. 2025. MiCEval: Unveiling Multimodal Chain of Thought’s Quality via Image Description and Reasoning Steps. In In the Proceedings of 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL 2025).
A Appendix

A.1 Prompt Details
Table 5 and Table 6 show the instruction and system prompt respectively used to instruct LLMs to unfold the hidden reasoning chains with the augmentation of various meta-reasoning skills.

Here is the given question: {ins}
Here is the original reasoning: {ans}

Table 5: The instruction used to prompt LLM to generate the enriched rationales.

A.2 Experimental Logs
Reflection-tuning. MMLU: \( y = 6.4535 * log(x) + 60.5418 \); GSM8K: \( y = 6.0944 * log(x) + 69.6957 \); ARC-C: \( y = 4.8088 * log(x) + 71.8375 \); HellaSwag: \( y = 9.2765 * log(x) + 54.6557 \); BBH: \( y = 11.1345 * log(x) + 34.2507 \); Avg.: \( y = 7.5551 * log(x) + 58.1925 \)

SRLM. MMLU: \( y = 9.0256 * log(x) + 62.8932 \) GSM8K: \( y = 6.5905 * log(x) + 71.4682 \) ARC-C: \( y = 5.9295 * log(x) + 78.0043 \) HellaSwag: \( y = 10.9284 * log(x) + 57.7607 \) BBH: \( y = 11.9599 * log(x) + 38.7086 \) Avg.: \( y = 8.8870 * log(x) + 61.7671 \)

A.3 Performance on Qwen-14B
We also conduct experiments on Qwen-14B models following the same setting in main experiments. Table 8 shows the final results. It is observed that SRLM initially exhibits higher performance at the 0th iteration but then gradually degrades. We attribute this to the ineffective of reasoning catalyst data to guide the reasoning processing since Qwen2.5-14B already achieves comparable performance against the used 4o models to generate reasoning catalyst data.

A.4 Case Study
Table 13 shows the non-cherry pick test examples of MATH. It is obvious that our proposed SRLM can take advantages of different meta-reasoning skills such as decomposition, detail, check, reflection and summary to get the correct answer.
Figure 9: The distribution of meta-reasoning skills on five various benchmarks and the global distribution during inference.
You are an expert at meta reasoning theory from cognitive science. Given the question, corresponding summarized reasoning and answer, you can always uncover hidden or unspoken reasoning, even when it isn’t explicitly stated.
You need to add any missing reasoning thoughts that you think it is helpful or may occurs to understand and solve this question based on given summarized reasoning result. Your new reasoning should be more comprehensive, detailed and clear. Your answer should follow the format like <thoughts> your new reasoning here </thoughts>
Inside <thoughts> </thoughts>, you need to explicitly indicate which meta reasoning skill is used, some of them are shown below, but note you can use anything else if you think it is helpful or it naturally occurs when you solve this question.

<decomposition>: breaking down a complex problem into smaller, more manageable parts. Making sure that you also provide answers for all decomposed problems in this section. You can decompose iteratively but should not contain same problem or exceed the max iteration depth which is three.

<backward>: starting with the desired observations at any previous reasoning step and working backward to identify the new reasoning directions.

<detail>: any details including but not limited to logic and reasons for your reasoning in this way, you are encouraged to add this at every unclear or unnatural reasoning step.

<summary>: summarize your reasoning to help future thinking.

<alternatives>: directly thinking in other ways, try to explore different solutions as much as possible to solve given problem.

<reflection>: you are encouraged to regularly reflect on your past reasoning in current response at various levels of detail, from sentence down to individual word. This will help you better understand and think through problems. It’s okay to make mistakes; use them as opportunities to learn and improve.

<analogy>: you are encouraged to regularly consider other analogous problems with the problem you’ve encountered at various reasoning steps, along with their solutions. Reference existing theories or methods that guided your approach to solving these problems. These similar problems can be at various levels of detail - from larger overarching issues down to smaller sub-problems you encountered along the way. The key is to demonstrate a diverse range of problems and solutions, to show how you have approached and resolved challenges that are analogous to the current situation.

<check>: consider different edge cases or test cases carefully.

<other>: other meta reasoning skills you think is helpful or worthy to try to solve the task.

Notice:

1. All tags must be properly invoked and closed, using the format like <reflection> and </reflection>.

2. You should always use first-person perspective.

3. You can add any new meta reasoning skill at any positions except <reflection>. Note <reflection> can not be invoked without any reasoning in current response and it can be invoked at any positions when you already have some reasoning results.

4. You cannot change the original reasoning. However, if you identify any errors or improvements in the reasoning, you can add new reasoning steps using above meta-reasoning skills afterwards to correct or clarify the path, ensuring a better understanding and solution.

5. You can apply the same reasoning skills multiple times or use different skills simultaneously.

6. Your answer should start with <thoughts>, and end with </thoughts>.

Table 6: The system prompt used to initialize LLM to unfold all hidden reasoning chains.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="1">MMLU</th>
    <th colspan="1">GSM8K</th>
    <th colspan="1">ARC-C</th>
    <th colspan="1">HellaSwag</th>
    <th colspan="1">BBH</th>
    <th colspan="1">Avg.</th>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Magpie</td>
    <td>44.35</td>
    <td>41.02</td>
    <td>55.72</td>
    <td>37.61</td>
    <td>27.13</td>
    <td>41.17</td>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>49.85</td>
    <td>56.56</td>
    <td>61.69</td>
    <td>45.15</td>
    <td>27.29</td>
    <td>48.11</td>
  </tr>
  <tr>
    <td rowspan="6">Mistral-7B</td>
    <td>IT<sub>0</sub> + Cat. (\( \mathcal{M}_0 \))</td>
    <td>51.63</td>
    <td>56.03</td>
    <td>63.05</td>
    <td>40.82</td>
    <td>29.97</td>
    <td>48.30</td>
  </tr>
  <tr>
    <th colspan="7"><i>Length Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>49.50</td>
    <td>58.91</td>
    <td>62.37</td>
    <td>45.91</td>
    <td>31.33</td>
    <td>49.60</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>50.01</td>
    <td>59.59</td>
    <td>64.85</td>
    <td>42.16</td>
    <td>29.82</td>
    <td>49.29</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>50.56</td>
    <td>56.41</td>
    <td>64.59</td>
    <td>46.88</td>
    <td>32.11</td>
    <td>50.11</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>49.07</td>
    <td>58.45</td>
    <td>60.41</td>
    <td>42.03</td>
    <td>29.35</td>
    <td>47.86</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>50.66</td>
    <td>56.79</td>
    <td>66.13</td>
    <td>47.30</td>
    <td>31.87</td>
    <td>50.55</td>
  </tr>
  <tr>
    <th colspan="7"><i>Off-policy Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>49.55</td>
    <td>58.07</td>
    <td>65.02</td>
    <td>41.02</td>
    <td>32.82</td>
    <td>49.30</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>50.63</td>
    <td>57.92</td>
    <td>62.12</td>
    <td>41.98</td>
    <td>32.49</td>
    <td>49.03</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>49.74</td>
    <td>55.95</td>
    <td>65.02</td>
    <td>42.28</td>
    <td>32.12</td>
    <td>49.02</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>50.19</td>
    <td>56.56</td>
    <td>60.24</td>
    <td>42.55</td>
    <td>31.48</td>
    <td>48.20</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>50.79</td>
    <td>57.77</td>
    <td>64.25</td>
    <td>45.66</td>
    <td>32.00</td>
    <td>50.09</td>
  </tr>
  <tr>
    <th colspan="7"><i>On-policy Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>50.00</td>
    <td>58.68</td>
    <td>64.59</td>
    <td>41.27</td>
    <td>29.64</td>
    <td>48.84</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>51.69</td>
    <td>57.47</td>
    <td>66.30</td>
    <td>42.29</td>
    <td>30.77</td>
    <td>49.70</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>50.28</td>
    <td>56.33</td>
    <td>64.59</td>
    <td>46.44</td>
    <td>30.36</td>
    <td>49.60</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>51.66</td>
    <td>55.34</td>
    <td>65.36</td>
    <td>44.55</td>
    <td>29.07</td>
    <td>49.20</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>49.54</td>
    <td>55.57</td>
    <td>62.63</td>
    <td>43.00</td>
    <td>32.00</td>
    <td>48.55</td>
  </tr>
  <tr>
    <td rowspan="2">Magpie</td>
    <td>54.34</td>
    <td>44.81</td>
    <td>68.77</td>
    <td>44.05</td>
    <td>25.10</td>
    <td>47.41</td>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>57.08</td>
    <td>66.49</td>
    <td>67.32</td>
    <td>50.34</td>
    <td>30.09</td>
    <td>54.26</td>
  </tr>
  <tr>
    <td rowspan="6">Llama3.1-8B</td>
    <td>IT<sub>0</sub> + Cat. (\( \mathcal{M}_0 \))</td>
    <td>57.60</td>
    <td>64.59</td>
    <td>70.14</td>
    <td>49.23</td>
    <td>31.46</td>
    <td>54.60</td>
  </tr>
  <tr>
    <th colspan="7"><i>Length Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>58.31</td>
    <td>65.73</td>
    <td>72.87</td>
    <td>48.05</td>
    <td>34.08</td>
    <td>55.81</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>58.27</td>
    <td>69.22</td>
    <td>72.70</td>
    <td>51.99</td>
    <td>32.03</td>
    <td>56.84</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>57.91</td>
    <td>67.25</td>
    <td>72.70</td>
    <td>52.88</td>
    <td>34.06</td>
    <td>56.96</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>57.27</td>
    <td>67.17</td>
    <td>72.87</td>
    <td>49.83</td>
    <td>33.44</td>
    <td>56.12</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>56.17</td>
    <td>65.73</td>
    <td>69.45</td>
    <td>50.27</td>
    <td>33.75</td>
    <td>55.07</td>
  </tr>
  <tr>
    <th colspan="7"><i>Off-policy Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>58.26</td>
    <td>67.70</td>
    <td>74.23</td>
    <td>46.11</td>
    <td>32.14</td>
    <td>55.69</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>58.50</td>
    <td>67.25</td>
    <td>73.46</td>
    <td>48.03</td>
    <td>31.28</td>
    <td>55.70</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>58.30</td>
    <td>65.43</td>
    <td>73.04</td>
    <td>46.93</td>
    <td>30.37</td>
    <td>54.81</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>58.57</td>
    <td>67.40</td>
    <td>74.15</td>
    <td>46.93</td>
    <td>30.03</td>
    <td>55.42</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>57.72</td>
    <td>66.79</td>
    <td>75.51</td>
    <td>47.40</td>
    <td>31.12</td>
    <td>55.71</td>
  </tr>
  <tr>
    <th colspan="7"><i>On-policy Selector</i></th>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \hat{\mathcal{M}}_1 \))</td>
    <td>57.96</td>
    <td>65.20</td>
    <td>70.05</td>
    <td>45.07</td>
    <td>29.69</td>
    <td>53.59</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>58.53</td>
    <td>64.67</td>
    <td>69.20</td>
    <td>48.41</td>
    <td>31.32</td>
    <td>54.43</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>58.26</td>
    <td>66.41</td>
    <td>72.35</td>
    <td>50.43</td>
    <td>32.89</td>
    <td>56.07</td>
  </tr>
  <tr>
    <td>IT<sub>4</sub> + Cat. (\( \mathcal{M}_4 \))</td>
    <td>58.45</td>
    <td>68.76</td>
    <td>70.14</td>
    <td>46.54</td>
    <td>33.37</td>
    <td>55.45</td>
  </tr>
  <tr>
    <td>IT<sub>5</sub> + Cat. (\( \mathcal{M}_5 \))</td>
    <td>59.17</td>
    <td>66.19</td>
    <td>69.88</td>
    <td>50.76</td>
    <td>33.01</td>
    <td>55.80</td>
  </tr>
</table>

Table 7: The detailed experimental results of each iteration.
<table>
  <tr>
    <th>Models</th>
    <th>Methods</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="4">Qwen2.5-14B</td>
    <td>Reflection-tuning</td>
    <td>78.72</td>
    <td>92.41</td>
    <td><b>93.51</b></td>
    <td>71.86</td>
    <td>57.56</td>
    <td>78.81</td>
  </tr>
  <tr>
    <td>IT<sub>0</sub> + Cat. (\( \mathcal{M}_0 \))</td>
    <td>77.89</td>
    <td>92.64</td>
    <td>93.17</td>
    <td><b>74.24</b></td>
    <td><b>59.69</b></td>
    <td><b>79.53</b></td>
  </tr>
  <tr>
    <td colspan="7"><i>Length Selector</i></td>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Cat. (\( \mathcal{M}_1 \))</td>
    <td>77.30</td>
    <td><b>92.66</b></td>
    <td>91.72</td>
    <td>68.44</td>
    <td>58.47</td>
    <td>77.72</td>
  </tr>
  <tr>
    <td></td>
    <td>IT<sub>2</sub> + Cat. (\( \mathcal{M}_2 \))</td>
    <td>76.97</td>
    <td>92.34</td>
    <td>92.32</td>
    <td>67.65</td>
    <td>57.08</td>
    <td>77.65</td>
  </tr>
  <tr>
    <td></td>
    <td>IT<sub>3</sub> + Cat. (\( \mathcal{M}_3 \))</td>
    <td>76.28</td>
    <td>92.04</td>
    <td>90.44</td>
    <td>65.70</td>
    <td>58.13</td>
    <td>76.52</td>
  </tr>
</table>

Table 8: The detailed experimental results of each iteration on Qwen-14B with length selector.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Reflection-tuning</td>
    <td>57.08</td>
    <td>66.49</td>
    <td>67.32</td>
    <td>50.34</td>
    <td>30.09</td>
    <td>54.26</td>
  </tr>
  <tr>
    <td>IT<sub>1</sub></td>
    <td>57.74</td>
    <td>67.93</td>
    <td>74.06</td>
    <td>49.22</td>
    <td>32.94</td>
    <td>56.38</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub></td>
    <td>57.87</td>
    <td>66.87</td>
    <td>72.53</td>
    <td>48.63</td>
    <td>30.46</td>
    <td>55.27</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub></td>
    <td>57.51</td>
    <td>68.46</td>
    <td>73.21</td>
    <td>51.59</td>
    <td>31.61</td>
    <td>56.48</td>
  </tr>
</table>

Table 9: The results of Llama3.1-8B on instruction-tuning dataset (collected by length selector) of each iteration.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>SRLM – \( \mathcal{M}_0 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>58.47</td>
    <td>67.10</td>
    <td>69.54</td>
    <td>47.13</td>
    <td>27.98</td>
    <td>54.04</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td>59.22</td>
    <td>66.26</td>
    <td>67.58</td>
    <td>46.68</td>
    <td>30.78</td>
    <td>54.10</td>
  </tr>
  <tr>
    <td>SRLM – \( \mathcal{M}_1 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>58.04</td>
    <td>66.41</td>
    <td>75.17</td>
    <td>45.71</td>
    <td>31.47</td>
    <td>55.36</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td>58.56</td>
    <td>66.72</td>
    <td>75.00</td>
    <td>48.00</td>
    <td>30.02</td>
    <td>55.66</td>
  </tr>
  <tr>
    <td>SRLM – \( \mathcal{M}_2 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>58.05</td>
    <td>68.46</td>
    <td>72.70</td>
    <td>44.76</td>
    <td>29.28</td>
    <td>54.65</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td>57.84</td>
    <td>67.70</td>
    <td>73.04</td>
    <td>50.79</td>
    <td>28.70</td>
    <td>55.61</td>
  </tr>
  <tr>
    <td>SRLM – \( \mathcal{M}_3 \)</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Cat. = 5k</td>
    <td>57.35</td>
    <td>65.96</td>
    <td>73.04</td>
    <td>45.52</td>
    <td>30.73</td>
    <td>54.52</td>
  </tr>
  <tr>
    <td>Cat. = 10k</td>
    <td>57.29</td>
    <td>67.32</td>
    <td>73.38</td>
    <td>48.65</td>
    <td>29.23</td>
    <td>55.17</td>
  </tr>
</table>

Table 10: The results using different size of reasoning catalyst data based on Llama3.1-8B model with 3 epochs optimization on three iterations (\( \mathcal{M}_{0,1,2,3} \)). We bold the best performance at each iteration.

<table>
  <tr>
    <th>Data</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>IT<sub>0</sub> + Cat.</td>
    <td>57.60</td>
    <td>64.59</td>
    <td>70.14</td>
    <td>49.23</td>
    <td>31.46</td>
    <td>54.60</td>
  </tr>
  <tr>
    <td colspan="7"><i>Update \( r_j^* \) with \( r_{j'}^{t+1} \)</i></td>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Up. Cat.<sub>1</sub></td>
    <td>59.27</td>
    <td>66.03</td>
    <td>74.15</td>
    <td>49.84</td>
    <td>31.29</td>
    <td>56.12</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Up. Cat.<sub>2</sub></td>
    <td>57.42</td>
    <td>65.81</td>
    <td>71.08</td>
    <td>47.57</td>
    <td>29.45</td>
    <td>54.27</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Up. Cat.<sub>3</sub></td>
    <td>55.52</td>
    <td>65.13</td>
    <td>69.28</td>
    <td>49.25</td>
    <td>30.92</td>
    <td>54.02</td>
  </tr>
  <tr>
    <td colspan="7"><i>Update both \( r_j^0, r_j^* \) with \( r_{j'}^{t+1}, r_{j'}^0 \)</i></td>
  </tr>
  <tr>
    <td>IT<sub>1</sub> + Up. Cat.<sub>1</sub></td>
    <td>56.99</td>
    <td>64.75</td>
    <td>72.87</td>
    <td>47.94</td>
    <td>33.83</td>
    <td>55.28</td>
  </tr>
  <tr>
    <td>IT<sub>2</sub> + Up. Cat.<sub>2</sub></td>
    <td>57.18</td>
    <td>63.99</td>
    <td>67.92</td>
    <td>49.78</td>
    <td>33.11</td>
    <td>54.40</td>
  </tr>
  <tr>
    <td>IT<sub>3</sub> + Up. Cat.<sub>3</sub></td>
    <td>55.09</td>
    <td>65.13</td>
    <td>68.17</td>
    <td>49.42</td>
    <td>32.57</td>
    <td>54.08</td>
  </tr>
</table>

Table 11: The performance of SRLM with length selector on Llama3.1-8B using <i>Update 1k</i> reasoning catalyst data with 1000 steps optimization.
<table>
  <tr>
    <th># Sampling</th>
    <th>Methods</th>
    <th>MMLU</th>
    <th>GSM8K</th>
    <th>ARC-C</th>
    <th>HellaSwag</th>
    <th>BBH</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="2">1</td>
    <td>Reflection-tuning</td>
    <td>57.08</td>
    <td>66.49</td>
    <td>67.32</td>
    <td>50.34</td>
    <td>30.09</td>
    <td>54.26</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>57.91</td>
    <td>67.25</td>
    <td>72.70</td>
    <td>52.88</td>
    <td>34.06</td>
    <td>56.96</td>
  </tr>
  <tr>
    <td rowspan="2">2</td>
    <td>Reflection-tuning</td>
    <td>65.50</td>
    <td>74.07</td>
    <td>76.02</td>
    <td>60.88</td>
    <td>41.55</td>
    <td>63.60</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>69.56</td>
    <td>75.89</td>
    <td>83.02</td>
    <td>64.25</td>
    <td>46.61</td>
    <td>67.87</td>
  </tr>
  <tr>
    <td rowspan="2">4</td>
    <td>Reflection-tuning</td>
    <td>71.66</td>
    <td>80.52</td>
    <td>82.00</td>
    <td>70.33</td>
    <td>52.48</td>
    <td>71.40</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>78.44</td>
    <td>83.85</td>
    <td>89.51</td>
    <td>76.11</td>
    <td>57.74</td>
    <td>77.13</td>
  </tr>
  <tr>
    <td rowspan="2">8</td>
    <td>Reflection-tuning</td>
    <td>76.42</td>
    <td>84.38</td>
    <td>84.56</td>
    <td>77.81</td>
    <td>61.09</td>
    <td>76.85</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>85.41</td>
    <td>88.32</td>
    <td>93.94</td>
    <td>85.46</td>
    <td>68.23</td>
    <td>84.27</td>
  </tr>
  <tr>
    <td rowspan="2">16</td>
    <td>Reflection-tuning</td>
    <td>79.79</td>
    <td>88.10</td>
    <td>85.75</td>
    <td>82.61</td>
    <td>67.56</td>
    <td>80.76</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>90.31</td>
    <td>91.89</td>
    <td>96.33</td>
    <td>91.67</td>
    <td>74.87</td>
    <td>89.01</td>
  </tr>
  <tr>
    <td rowspan="2">32</td>
    <td>Reflection-tuning</td>
    <td>82.48</td>
    <td>90.75</td>
    <td>88.05</td>
    <td>86.41</td>
    <td>72.79</td>
    <td>84.10</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>93.85</td>
    <td>93.63</td>
    <td>98.12</td>
    <td>95.41</td>
    <td>80.23</td>
    <td>92.25</td>
  </tr>
  <tr>
    <td rowspan="2">64</td>
    <td>Reflection-tuning</td>
    <td>84.80</td>
    <td>92.27</td>
    <td>89.16</td>
    <td>89.24</td>
    <td>76.27</td>
    <td>86.35</td>
  </tr>
  <tr>
    <td>SRLM</td>
    <td>96.15</td>
    <td>95.38</td>
    <td>98.72</td>
    <td>97.62</td>
    <td>83.31</td>
    <td>94.24</td>
  </tr>
</table>

Table 12: The detailed experimental results with multiple sampling for baseline and SRLM.
Question: Suppose \(a\) and \(b\) are positive integers, neither of which is a multiple of 3. Find the least possible remainder when \(a^2 + b^2\) is divided by 3.

Reflection-tuning Answer: To solve this problem, we need to find the remainder of \(a^2 + b^2\) when divided by 3, given that \(a\) and \(b\) are integers not divisible by 3.
First, consider the properties of numbers modulo 3. Any integer \(n\) can be expressed as \(n = 3k + r\) where \(r\) is the remainder when \(n\) is divided by 3. Since \(a\) and \(b\) are not multiples of 3, \(rq0\).
Now, let's express \(a\) and \(b\) in terms of their remainders when divided by 3: \(a = 3m + r_1\) and \(b = 3n + r_2\), where \(r_1, r_2 \neq 0\).
Substitute these into \(a^2 + b^2\): \(a^2 + b^2 = (3m + r_1)^2 + (3n + r_2)^2 = 9m^2 + 6mr_1 + r_1^2 + 9n^2 + 6nr_2 + r_2^2\).
Notice that all terms containing \(m\) and \(n\) are multiples of 3, and thus their sum is also a multiple of 3. The only terms not divisible by 3 are \(r_1^2\) and \(r_2^2\).
Since \(r_1\) and \(r_2\) are not zero, \(r_1^2\) and \(r_2^2\) can each be 1 or 2 (since they are squares of non-zero remainders when divided by 3). The smallest possible value for \(r_1^2 + r_2^2\) is 1 (when both \(r_1\) and \(r_2\) are 1 or both are 2) and the largest possible value is 4 (when one is 1 and the other is 2). Thus, the remainder of \(a^2 + b^2\) when divided by 3 can only be 1 or 2.
Therefore, the possible remainders when \(a^2 + b^2\) is divided by 3 are [1, 2]

SRLM: <thoughts>
<decomposition> Let's break down the problem into smaller parts to ensure clarity and accuracy in our reasoning. First, we need to understand the behavior of squares modulo 3. Next, we'll examine the possible values of \(a\) and \(b\) under the constraint that they're not multiples of 3. Finally, we'll compute (\(a^2 + b^2 \mod 3\)) for these values and determine the remainder. </decomposition>
<detail> Understanding the behavior of numbers under modulo 3 involves recognizing that any integer \(n\) can be expressed as \(n = 3q + r\), where \(q\) is the quotient and \(r\) is the remainder. For \(n^2\), we have \(n^2 = (3q + r)^2 = 9q^2 + 6qr + r^2\). Simplifying gives us \(n^2 \equiv r^2 \pmod{3}\). Thus, the possible remainders for \(n^2\) are 0, 1, or 2 depending on \(r\). </detail>
<detail> Since neither \(a\) nor \(b\) is a multiple of 3, they can take values of 1 or 2 modulo 3. This means \(a \equiv 1\) or \(2 \pmod{3}\) and similarly for \(b\). We need to evaluate \(a^2 + b^2\) for these possibilities. </detail>
<check> Let's verify each case:- Case 1: \(a \equiv 1 \pmod{3}, b \equiv 1 \pmod{3}\) - Then \(a^2 \equiv 1^2 \equiv 1 \pmod{3}\) - \(b^2 \equiv 1^2 \equiv 1 \pmod{3}\) - So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\)
- Case 2: \(a \equiv 1 \pmod{3}, b \equiv 2 \pmod{3}\) - Then \(a^2 \equiv 1^2 \equiv 1 \pmod{3}\) - \(b^2 \equiv 2^2 \equiv 4 \equiv 1 \pmod{3}\) - So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\)
- Case 3: \(a \equiv 2 \pmod{3}, b \equiv 1 \pmod{3}\) - Then \(a^2 \equiv 2^2 \equiv 4 \equiv 1 \pmod{3}\) - \(b^2 \equiv 1^2 \equiv 1 \pmod{3}\) - So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\)
- Case 4: \(a \equiv 2 \pmod{3}, b \equiv 2 \pmod{3}\) - Then \(a^2 \equiv 2^2 \equiv 4 \equiv 1 \pmod{3}\) - \(b^2 \equiv 2^2 \equiv 4 \equiv 1 \pmod{3}\) - So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\)</check>
<summary> In all cases, the remainder when \(a^2 + b^2\) is divided by 3 is consistently 2. There were no overlooked scenarios or computational errors, confirming the correctness of the result. </summary>
<reflection> By systematically evaluating each case and cross-verifying calculations, I ensured that the solution aligns with the conditions provided. The consistency in outcomes confirms the robustness of the method applied. </reflection>
Therefore, the least possible remainder when \(a^2 + b^2\) is divided by 3 is [2]</thoughts>

Based on the above processing, the key reasoning steps can be summarized clearly, followed by the concise answer.
To solve this problem, we need to understand the behavior of squares modulo 3. Any integer \(n\) can be expressed as \(n = 3q + r\), where \(q\) is the quotient and \(r\) is the remainder when \(n\) is divided by 3. Squaring both sides gives \(n^2 = (3q + r)^2 = 9q^2 + 6qr + r^2\). This simplifies to \(n^2 \equiv r^2 \pmod{3}\). Therefore, the possible remainders for \(n^2\) are 0, 1, or 2 depending on \(r\).
Since neither \(a\) nor \(b\) is a multiple of 3, they can be either 1 or 2 modulo 3. We need to calculate \(a^2 + b^2\) for these cases:
1. If \(a \equiv 1 \pmod{3}\) and \(b \equiv 1 \pmod{3}\), then \(a^2 \equiv 1^2 \equiv 1 \pmod{3}\) and \(b^2 \equiv 1^2 \equiv 1 \pmod{3}\). So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\).
2. If \(a \equiv 1 \pmod{3}\) and \(b \equiv 2 \pmod{3}\), then \(a^2 \equiv 1^2 \equiv 1 \pmod{3}\) and \(b^2 \equiv 2^2 \equiv 1 \pmod{3}\). So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\).
3. If \(a \equiv 2 \pmod{3}\) and \(b \equiv 1 \pmod{3}\), then \(a^2 \equiv 2^2 \equiv 1 \pmod{3}\) and \(b^2 \equiv 1^2 \equiv 1 \pmod{3}\). So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\).
4. If \(a \equiv 2 \pmod{3}\) and \(b \equiv 2 \pmod{3}\), then \(a^2 \equiv 2^2 \equiv 1 \pmod{3}\) and \(b^2 \equiv 2^2 \equiv 1 \pmod{3}\). So, \(a^2 + b^2 \equiv 1 + 1 \equiv 2 \pmod{3}\).
In all cases, the remainder when \(a^2 + b^2\) is divided by 3 is 2. Therefore, the least possible remainder is [2].

Table 13: The (non-cherry-pick) test instance from MATH.

Reviewer paper summaries: This paper introduces Self-Reasoning Language Models (SRLMs), a novel approach to improve the reasoning capabilities of LLMs by enabling them to iteratively refine their own reasoning chains (Chain-of-Thought, CoT).

This paper attempts to improve chain-of-thought reasoning skills in language models through instruction fine-tuning on a large dataset where a portion (0.02% / 2000 samples) have their reasoning chains generated from a much larger model (gpt-4o) and contain curated skills (like reflection).  From those 2000, the authors show that LMs start to incorporate these higher level skills of reasoning, and are able to reannotate the large SFT dataset with new reasoning capabilities for an iterative fine-tuning loop where the LM regenerates the training data (with increasingly more complex reasoning in the generated traces).  They also compare different ways to select which questions to regenerate or include in the next iteration of training including things like the length of the response and probability of correct answers.

The authors show performance increases in reasoning datasets like GSM8k and MMLU as the LM is trained, then regenerates data, and trains again.  They show that these gains hold for instruction following datasets as well, i.e. Alpaca Eval v2.  These main results are paired with fairly extensive ablations showing how different sizes of annotated reasoning traces from GPT-4o, different numbers of iterations of training, and different models can affect the performance. 

Ultimately this work is providing a foundation for a similar loop to that of STaR, where a language model iteratively generates its training data that is filtered by some criteria and then is trained on it.  Unique to this paper, however, is the focus on instilling specific reasoning capabilities, inspired by Long CoT / R1 type of inference techniques.
Summary of strengths: - The idea of teaching the model how to reason, rather than just what to reason, is a valuable contribution.
- The paper includes several analyses, such as the effect of sampling times, the number of iterations, and different selectors, which provide valuable insights into the behavior of SRLMs.
- The logic of the overall framework and algorithms is very clear.

1. The results seem promising. I have a minor note with them, but the results overall are positive and beat the baselines, indicating that this type of fine-tuning yields some promise! Especially Figure 4, where Best-of-N shows better accuracy vs sample rate efficiency.

2. The idea of focusing on specific meta-reasoning skills (section 3.2.1) seems novel and exciting! However, I don't think this was expanded upon later on in the paper (this could've been a very neat source of ablations!)

3. Minor, but I appreciated the inclusion of non-math related datasets in the evaluation showing that the method outlined in the paper works beyond MCQA / Arithmetic but also work on instruction following / llm-as-a-judge evaluation datasets.
Summary of weaknesses: - Lack of comparison with other methods.
- The justification for the three selectors (length, off-policy, on-policy) could be strengthened. While the intuition is reasonable, it's not theoretically grounded. Why is longer always better? 
- In section 4.2, the statement "small language models still can provide stronger supervision signals than the latest powerful models" is a strong claim and should be supported with more evidence.
- Table 6 (the system prompt) is helpful, but connecting it more directly to the examples in the reasoning catalyst data would improve understanding. 
- The paper acknowledges that many existing self-improvement methods focus on tasks with verifiable answers.  While the paper addresses this, the chosen benchmarks still largely fall into this category.  Including more tasks that require subjective judgment or open-ended reasoning (e.g., creative writing, argumentation) would strengthen the claim of generalizability.

1. The paper is missing an iterative fine-tuning baseline: The baseline consistently compared against in the paper is "Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning" Li et. al. 2024a, which feels like a useful baseline but not maybe the most applicable?  STaR or other variants of iterative fine-tuning (especially with self-generated training data) may be useful to see the efficacy of this "reasoning catalyst" data.  

2. No error bars / significance testing on the results found per iteration.  The performance of each benchmark, for the most part, fluctuates iteration to iteration sometimes dipping below the baseline but then in the next does extremely well.  This makes me question the stability of the methods here. Are we sure the gains we see from iteration to iteration are not influenced by noise? Hellaswag for example, goes from 49.32 to 45.53 and then back to 48.1, I would've liked more explanation into these kinds of results.

3. (minor) Novelty: There is talk in section 3.2.1 about how to build the initial set of questions to annotate with GPT-4o (and manually curate the reasoning skills with it), however, I don't see any ablations on different types of reasoning skills, how often those reasoning skills are present in the catalyst data, etc.  Because of this, the method feels very close to something like STaR with the exception of how you select the traces to be in the next iteration (length vs "is the chain correct"). 

---

Overall, there really needs to be an iterative training baseline. Finetuning with GPT-4o annotating the whole dataset or fine-tuning on t=0 for the same number of steps as each iteration would also be useful to understand if the self-generation part of the pipeline is useful.  Finally, the paper, I think, would have tremendously more novelty if more ablations on different reasoning capabilities were included in the catalyst data (hand made vs gpt-4o vs R1 or something) which would help distinguish this paper.
Comments, suggestions and typos: See above.

Look at weaknesses.

Figure 4. projections should end at 100% (I saw some lines going above 100% accuracy that threw me off for a second).

Some minor spelling / grammar stuff throughout, the main stuff that impacted my understanding of the work were:
- 207-209: "All other meta-reasoning skills used in the thoughts [should?] also be [...]"
- 417-420: "After checking the number of selected numbers at each iteration" (I think this means, checking how large the SFT dataset size is after each iterations filtering process?)
- 443-445: "[...] the performance at the initial iteration (iteration at t=0) [regardless of incorporating] a reasoning catalyst or not."
Overall assessment scores: 3

3
Metareview score: 3
Metareview: This paper proposes a framework to enhance LLM reasoning through iterative self-training, guided by a small set of curated *reasoning catalyst* examples. With only a few such samples, the model learns to enrich its own rationales, outperforming both reflection-tuning and an iterative fine-tuning baseline across multiple reasoning and instruction-following benchmarks. The authors present strong empirical results, comprehensive ablations, and directly address reviewer concerns—including baseline comparisons and the role of meta-reasoning—with additional experiments and clear explanations. Overall, the approach is simple, well-justified, and consistently effective, making it a strong fit for sessions on reasoning, instruction tuning, or LLM training methodologies.
Summary of reasons to publish: This paper presents a simple and effective framework that enables LLMs to iteratively refine their own rationales using only 1,000 reasoning catalyst examples. It outperforms reflection-tuning and iterative baselines across multiple reasoning tasks. The key innovation lies in teaching models how to reason—via meta-reasoning skills—by refining existing rationales rather than generating new answers. The authors provide thorough ablations and introduce new baselines to isolate the effect of catalysts, while addressing reviewer concerns with clear experiments and analysis. Overall, the method is data-efficient, well-supported, and offers a meaningful contribution to LLM reasoning.