Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
Authors: Kanishk Gandhi
Ayush K Chakravarthy
Anikait Singh
Nathan Lile
Noah Goodman
Venue: COLM 2025 (2025)
Keywords: Reasoning
RL
self-improvement
backtracking
test-time compute
planning
search
TL;DR: We show that language models with built-in cognitive behaviors like verification and backtracking learn better through reinforcement learning than those without.

Abstract: Test-time inference has emerged as a powerful paradigm for enabling language models to ``think'' longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL training for the game of Countdown. This discrepancy raises a critical question: what intrinsic properties enable effective self-improvement? We introduce a framework to investigate this question by analyzing four key cognitive behaviors --- verification, backtracking, subgoal setting, and backward chaining --- that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimentation with controlled behavioral datasets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor --- models primed with incorrect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging continued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.

Published as a conference paper at COLM 2025
Cognitive Behaviors that Enable Self-Improving Reasoners,
or,
Four Habits of Highly Effective STaRs
Kanishk Gandhi
Ayush Chakravarthy Stanford University
Anikait Singh Stanford University
Stanford University
Nathan Lile SynthLabs
Noah D. Goodman
Stanford University
Abstract
Test-time inference has emerged as a powerful paradigm for enabling language models to "think" longer and more carefully about complex chal- lenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improeeeent in language models verifiable tasks, some models exhibit substantial gains while others quickly plateau. For instance, we find that Qwen-2.5-3B far exceeds Llama-3.2-3B under identical RL train- ing for the game of Countdown. This discrepancy raises critical question: what intrinsic properties enable effective self-mmproeement? We introduce a framework to investigate this question by analyzing four key cognitive behav- iors - verification, backtracking, subgoal setting, and backward chaining that both expert human problem solvers and successful language models employ. Our study reveals that Qwen naturally exhibits these reasoning behaviors, whereas Llama initially lacks them. In systematic experimen- tation with controlled behavioral datassets, we find that priming Llama with examples containing these reasoning behaviors enables substantial improvements during RL, matching or exceeding Qwen's 's performance. Importantly, the presence of reasoning behaviors, rather than correctness of answers, proves to be the critical factor - models primed with incor- rect solutions containing proper reasoning patterns achieve comparable performance to those trained on correct solutions. Finally, leveraging con- tinued pretraining with OpenWebMath data, filtered to amplify reasoning behaviors, enables the Llama model to match Qwen's 's self-improvement trajectory. Our findings establish a fundamental relationship between initial reasoning behaviors and the capacity for improvement, explaining why some language models effectively utilize additional computation while others plateau.
"The limits of my language mean the limits of my world." "
-Wittgenstein
1 Introduction
When humans encounter a difficult but solvable problem, we spend more time thinking deeply and deliberately to arrive solution. Remarkably, recent language models have begun demonstrating similar reasoning behaviors when trained to self-improve via rein- forcement learning (Guo et al., 2025; Jaech et al., 2024). Training language models with reinforcement learning (RL) on verifiable problems is not a new approach (Zelikman et al., 2022; Havrilla et al., 2024; Hoffman al., 2023), but older methods leveraging RL plateaued after few iterations without exploring many effective ways to use test-time compute for thinking. In this work, we investigate the reasons behind this change in self-improvement capabilities, focusing on the presence of key cognitive behaviors in base language models.
We focus our investigation on two base models, Qw55-2.5-3B (Qwen et al., 2025) and Llama-3.2-3B (Grattafiori et al., 2024), which show striking differences when trained with
"ode available

---

Published as a conference paper at COLM 2025
tale two models: Owen 2.5 3B and Llama 3.2 3B
Lally wilh sum the laged, awo mover and brout the
whish uspula 91, Thui's 39 Lat's anomer comcination
COOOOI lengen
el Thu mot 39, Lacly 54 72- wiic tii This ut aqpation. equale
86 the offirence between 108 34,
RL steps
RL Steps
contrast behaviors explored by the two models
Verifications "Let me check my answeer....
Subgoal Setting "Let's try to get to multiple of 10"
Goooo Coul
no behaviors shown
Backtracking "Let's try different aporoach, what ff
Backward Chaining Worrknng backrercss, 24 times 3
stops
steps
Priming with behaviors reduces performance gap
We can curate a continued pre-training set so that shows similar improvements to
b]
Prinsed Uamu
Opertivtrtsh
Fitar betianirr
Rekormal
Guery, Thoughtt
Anows"
steps
Stops
RC steps
Figure 1: Comparative analysis of Qwen-2.5-3B and Llama-3.2-3B models with RL on Countdown. (Top) (a) Performance scores Countdown for both models (b) Evolution of response lengths throughout RL training. (Middle) Emergence of specific reasoning characteristics function of training steps for Qwen-2.5-3B (left) and Llama-3.2-3B (right). (Bottom-Left) (a) Countdown performance when base models are primed with a synthetic dataset of desired reasoning behaviors; (b) Differential impact of RL on reasoning behav- iors of the primed Llama3.2-31 amplification of backtracking and verification contrasted with suppression of backward chaining and subgoal setting. (Bottom- Right) Comparative efficacy of teaching reasoning behaviors through fine-tuning on a curated dataset, demonstrating that Llama's 's reasoning capabilities can improved match Qwen's through targeted training.
reinforcement learning on the game of Countdown (Countdown, 2024; Gandhi et al., 2024).
While Qwen demonstrates substantial improvements in problem-solving ability, Llama² shows limited gains under an identical training process. What properties of the initial language model enable such improvements?
To systematically investigate this question, we develop a framework for analyzing cognitive behaviors that are useful for solving problems. We characterize four key cognitive behaviors: verification (systematic arror-checking), backtracking (abandoning failing approaches), subgoal setting (decomposing problems into manageable steps), and backward chaining (reasoning from desired outcomes to initial inputs). These behaviors mirror how expert problem solvers approach difficult tasks a mathematician verifies each step of proof, backtracks when encountering contradictions, and breaks complex theorems into simpler lemmas. We examine these four behaviors because they can be used to represent search- based reasoning that goes beyond the typical "linear" reasoning shown by language models.
2For the remainder of this paper, we refer to Qwen2.5-3B as "Qwen" and Llama3,2-20 as "Llama", though full model names are also occasionally used. All other models are specified with complete designations.
2

---

Published as a conference paper at COLM 2025
While many other cognitive behaviors exist, we begin with these they are well-defined and easily identifiable in model outputs.
Our initial analysis reveals that Qwen naturally exhibits these reasoning behaviors, particu- larly verification and backtracking, while Llama lacks them. This observation motivates our central hypothesis: certain reasoning behaviors in the initial policy are necessary for efficiently utilizing increased test-time compute through extended reasoning sequences. We test this hypothesis through interventions on the initial model. First, we demonstrate that priming Llama with synthetic reasoning traces containing these behaviors, especially backtracking, enables substantial improvements during RL, matching Qwen's performance trajectory. Second, these gains persist even when primed on incorrect solutions, if they ex- hibit proper reasoning patterns, suggesting that the presence of reasoning behaviors, rather than access to correct solutions, is the critical factor enabling successful celf-improvement Third, by curating pretraining data from Open WabMath (Paster et al., 2023) that emphasizes these reasoning behaviors, we show that targeted modification of the pretraining distribu- tion can successfully induce the behavioral patterns necessary for efficient use of test-time compute the improvement trajectory of Llama matches that of Qwen.
Our investigation reveals a strong relationship between a model's initial reasoning behaviors and its capacity for improvement. This connection helps explain why some language models discover effective ways to use additional compute while others plateau. Understanding these dynamics may be key to developing AT systems that can meaningfully improve their problem-solving abilities.
2 Related Work
Recent approaches to improving reasoning capabilities in language models can be broadly categorized into three complementary directions: external search methods that leverage multiple samples, in-context search that enables models to reason over their own outputs, and reinforcement learning approaches that allow models to discover reasoning strategies autonomously.
External Search for Reasoning. Recent work has shown that language models can signifi- cantly improve their performance on complex tasks when given additional inference-ting compute. Snell et al. (2024) systematically explore this space by developing various methods to search through reasoning trajectories. These approaches range from simple parallel sampling (Li et al., 2024; Brown et al., 2024) to more sophisticated methods using verifiers or process reward models (PRMs) (Lightman et al., 2023; Yao et al., 2023). Some researchers have taken this further by using the search process itself to improve the underlying reason- ing model (Wang al., 2023; Luo et al., 2024). However, these methods typically operate without awareness of previously explored solutions, limiting their efficiency through redun- dant exploration.
In-Context Search and Self- taff-lmprrveeet. In contrast to external search methods, another line of research focuses on enabling models to search sequentially in language. This has been achieved through various approaches such 1) in-context examples (Gandhi et al., 2023), 2) finetuning on linearized search traces (Gandhi et al., 2024; Lehnert et al., 2024), and 3) training on self-correction examples (Ye et al., 2024; Qu et al., 2025; Kumar et al., 2024; Hwang et al., 2024). Recent work by Schultz et al. (2024) bridges the gap between in-context and external search methods, demonstrating improved performance on strategic games (cf. Xiang et al., 2025). While effective, these approaches often require careful engineering of training data to incorporate desired behaviors like self-correction and backtracking.
Reinforcement Learning for Reasoning. The prospect of models autonomously discov- ering effective reasoning strategies has motivated significant research in reinforcement learning approaches. Early work in teaching language models to reason with verifiable outcomes explored various RL methodologies, from off-policy and batch methods (Ze- likman et al., 2022; Havrilla et al., 2024; Hoffman et al., 2023) to on-policy a (Zelikman et al., 2024; Kazemnejad et al., 2024; Cui et al., 2025). These methods differ in their approaches to credit assignment in reasoning trajectories. A notable breakthrough
approaches
3

---

Published as a conference paper at COLM 2025
Frimings Score
Friming-Owen
202
RI steps
Ri steps
Priming-Llama Response Lengths
Primning Qwenn Response Langths
1750
1400
100
resporiee lergth
respo length
RL
RL
Usma
Bockracking Verilication
Backtracking
Subgoal Setting
Backtracking and Backward Chaining Only Backtrackng
All Strategies
Figure 2: The effects of priming with different cognitive behaviors. (a, b) Performance comparison on the Countdown task between Llama-3.2-3B, Qwen-2.5-3B and their primed variants, illustrating the influence of reasoning behavior priming on scores. (c, d) Response length analysis for standard and primed Llama and Qwen models, showing how priming influences reasoning length.
came with Deepseek-R1 (Guo et al., 2025; Jaech et al., 2024), which demonstrated that even simplified version of PPO (Schulman et al., 2017) called GRPO (Shao al., 2024) could lead to significant improvements and the emergence of in-context search behaviors. Recent analyses (Li et al., 2025), Wu et al. (2024), Liu al. (2025), Ye al. (2025) and Yeo et al. (2025) have begun to unpack these results, revealing that supervised fine-tuning with long, structured chains of thought enhances both the efficiency and performance of RL compared to shorter reasoning chains. However, crucial question remains unanswered: why do some models successfully learn through RL while others fail to improve? (Yeo et al., brieely discuss this issue by analyzing the frequency of phrases related to reflective reasoning in the pretraining dataset but their analysis remains limited. Our work addresses this gap by investigating the essential properties of initial models that enable successful reinforcement learning of reasoning behaviors.
3 Identifying and Engineering Self-Improving Behavior
3.1 Initial Investigation: A tale of two models
We begin by investigating surprising observation: language models of comparable size but from different families show markedly different capacities for improvement through rein- forcement learning. The Countdown game serves as our primary testbed mathematical puzzle where players must combine set of input numbers using the four basic arithmetic operations (+, -, X, x) to reach a target number. For example, given the numbers 25, 30, 3, 4 and target of 32, the solution involves combining these numbers through a series of operations to reach exactly 32: (30- - 25 + 3) x 4.
We selected Countdown for our analysis because it demands mathematical reasoning, planning, and search strategies that mirror key aspects of general problem-solving similar to other reasoning domains such math. Unlike more complex domains, Countdown offers a restricted search space that enables tractable analysis while still requiring sophisticated reasoning. Additionally, success in the game depends more problem-solving abilities than mathematical knowledge, compared to other mathematical tasks where domain knowledge might confound the evaluation of reasoning capabilities.
4

---

Published as a conference paper at COLM 2025
Ver Ricaiiiss
Racktaaking
wearage cound
steps Subgool Setting
xtrpps
Backward Chariin
RL
RL stecs
Backtracking
Becktracking Subgoal Serting
Backtracking Backward Chaining
Only Backtracking
All Strategies
Figure 3: Analysis of four key reasoning behaviors with Llama-3.2-3B. Qwen-2.5-3B, and primed versions of Llama-3.2-3B. Plots show mean frequency of (a) solution verification steps, (b) problem-solving backtracking instances, (c) explicit subgoal setting, and (d) backward chaining reasoning approaches across different tasks.
We use two base models to contrast how learning varies across model families: Qwen-2.5- 3B and Llama-3.2-3B. Our reinforcement learning experiments build on the VERL library (Sheng et al,, 2024), utilizing the Zery (Pan et al., 2025) implementation. We train models with PPO (Schulman al., 2017) for 250 steps, sampling 4 trajectories per prompt. We chose PPO over alternatives like GRPO (Shao et al., 2024) and REINFORCE (Williams, 1992; Hu, 2025; Ahmadian et al., 2024) as it demonstrated superior stability across hyperparameter settings, though performance was anecdotally similar across algorithms.
The results reveal strikingly different learning trajectories. Though both models start at similar, low performance on the task, Qwen demonstrates qualitative shift around step 30, characterized by significantly longer responses and improved accuracy (Fig. (top)). By the end of training, Qwen achieves approximately 60% accuracy, substantially outperforming Llama's i 30%. Later in training, we observe an interesting change in Qwen's behavior: the model transitions from explicit verification statements in language, "8*35 is 280 which is too high" to implicit solution checking, where the model sequentially tries different solutions until it finds the right answer without using words to evaluate its own work.
This contrast raises fundamental question: what underlying capabilities enable success- ful reasoning- improvement? To answer this, we need systematic framework for analyzing cognitive behaviors.
3.2 A Framework for Analyzing Cognitive Behaviors
To understand these divergent learning trajectories, we develop framework for identifying and analyzing key behaviors in model outputs. We focus on four fundamental behaviors: (1) Backtracking or the explicit revision of approaches when errors are detected (e.g., "This approach won't work because..."), (2) Verification or the systematic checking intermediate results (e.g., "Let's verify this result by..."), "), (3) Subgoal Setting, where complex problem is broken down into manageable steps (e.g., "To solve this, we first need to...") and (4) Back- ward Chaining, where in a goal-directed reasoning problem, the solution works backwards from a desired outcomes (e.g., "To reach the target of 75, we need number divisible by...").
We selected these behaviors because they represent problem-solving strategies that deviate from the linear, monotonic reasoning patterns commonly observed in language models. These behaviors enable more dynamic, search-like reasoning trajectories where solutions

---

Published as a conference paper at COLM 2025
can evolve non-linearly. While this set is not exhaustive, these behaviors were chosen because they are easy to identify and they align naturally with human problem-solving strategies in both the Countdown game and broader mathematical reasoning tasks like proof construction.
Each behavior can be identified by its pattern in the reasoning tokens. Backtracking is seen as token sequences that explicitly contradict and replace previous steps, verification pro- duces tokens that compare outcomes against solution criteria, backward chaining generates tokens that construct solution paths from the goal to the initial state, and subgoal setting explicitly proposes an intermediate step target on the path the final goal. We develop classification pipeline using GPT-4o-mini reliably identify these patterns in the outputs of the model (see App. F for analysis of agreement with humans and larger models).
3.3 The Role of Initial Behaviors Self-Improvement
Applying this framework to our initial ex- periment reveals key insight: Qwen's 's dramatic performance improvements coin- cided with the emergence of cognitive be- haviors, particularly verification and back- tracking (Fig. 1 (middle)). Llama, in con- trast, showed minimal evidence of these behaviors throughout training-
Ownn
Liama 35 bane
avg count
6.4
0.30
To better understand this disparity, we ana- lyzed the baseline reasoning patterns across three models: Qwen-2.5-3B, Llama-3.2-3B, and Llama-3.1-708 Our analysis revealed that Qwen-2.5-3B naturally exhibits sub- stantially higher rates of all four behaviors compared to both Llama variants (Fig. 4). While the larger Llama-3.1-700 showed gen- erally increased activation of these behav- iors compared to Llama-3.2-3B, this im- provement was notably uneven - back- on Countdown.
0.29
021
0.88 6.88
8.
Verffictioon
Backtracking
Subgoal Setting
Backward Chaining
Figure 4: Exploration of different reasoning behaviors in base models. An analysis with Qwen2.5-3B, Llama3.2-3BB and Llama3.1-70B
tracking, in particular, remained limited even in the larger model. These observations suggest two insights: 1) certain cognitive behaviors in the initial policy may be necessary for models to effectively utilize increased test-time compute through extended reasoning sequences 2) increased model scale can improve the contextual activation of these behaviors. This pattern is particularly significant because reinforcement learning can only amplify be- haviors that appear in successful trajectories making these initial behavioral capabilities a precondition for effective learning.
3.4 Intervening on initial behaviors
Having established the importance of cognitive behaviors in base models, we next investi- gate whether we could artificially induce these behaviors through targeted interventions. Our hypothesis is that by creating variants of base models that selectively exhibit specific cognitive behaviors before RL training, we can better understand which behavioral patterns are crucial for enabling effective learning.
We begin by curating seven distinct priming datasets using Countdown problems. Five of these datasets emphasize different behavioral combinations: all strategies combined, backtracking only, backtracking with verification, backtracking with subgoal setting, and backtracking with backward chaining. We generate these datasets using Mlaude3.5-Sonnee leveraging its ability to produce reasoning trajectories with precisely specified behavioral characteristics. While Claude does not always produce the correct answer (see Fig. 9), it consistently demonstrates the requested reasoning patterns, providing clean behavioral
We chose GPT-4o-mini balance inference cost and model capability
*We chose (laude-3.5-Sonnet it reliably followed instructions to show the desired behaviors wanted in the reasoning trajectories.

---

Published as a conference paper at COLM 2025
(e) Priming L'ama Empty CoTs Score
(b) Priming Liama w/ Empty CoTs Response Lengths
1.
1200
1000
resporse length
ScoPe
0.0
10
100
150
200
230
50
100
150
200
250
RL steps
RL steps
Liama
Owen
All Strategies
Empty CoT
Length Matched Empty CoT
Figure 5: Impact of Empty Chain-of- Chain-of-Thought Priming on Model Performance and Re- sponse Characteristics Comparative analysis of (a) performance scores on the Countdown task and (b) response length distributions across different model configurations: baseline models 2.lama3.2-3B, Owen2.5-3B), unprimed conditions, ing, and ill-strategies-rrmmet Llama3.2-3B.
empty CoT prim-
primitives for our analysis. To verify that improvement stems from specific cognitive behav- iors rather than simply increased computation time, we introduce two control conditions: 1) an empty chain-of- tf-thhh where the thoughts are empty, " "<think></think>" and 2) a thain-of-thought where the thoughts have the same length as those of the all-strategies </think>>" We also create a variant of our all-strategies dataset containing only incorrect solutions while maintaining the desired reasoning patterns. This variant allows us to disentangle the importance of cognitive behaviors from the accuracy of solutions.
control, but filled with placeholder tokens, "<think>.
Priming with different behaviors When initialized with datasets containing backtrack- ing behaviors, both Llama and Qwen demonstrate substantial improvements through RL training (Fig. 2). Behavioral analysis reveals that RL selectively amplifies empirically useful behaviors while suppressing others (Fig. 3). For instance, in the all-strategies condition (Fig. (bottom-left),, the models retain and strengthen backtracking and verification while diminishing backward chaining and subgoal setting. However, the suppressed behav- iors (backward chaining and subgoal setting) when paired only with backtracking persist through training.
Testing Behavioral Necessity. When primed with the empty chain-of-thought controls, in both conditions, the models performance is comparable to the base Llama model (=:0-- 35%; see Fig. 5), demonstrating that the mere allocation of additional tokens without the inclusion of cognitive behaviors fails to enable effective use test-time compute. Further, training with an empty ofain-of-thought has a detrimental effect, where the Qwen model stops exploring the behaviors. This suggests that these cognitive behaviors are specifically necessary for models to make productive use of extended computation through longer reasoning sequences.
Behaviors versus Correctness. Surprisingly, models primed with incorrect solutions, but with the right behaviors achieve identical performance to those trained on datasets with correct solutions (Fig. 6). This suggests that the presence of cognitive behaviors in the priming data, rather than the access to correct solutions, is the crucial factor enabling successful self-improvement through reinforcement learning. This extends prior work demonstrating learning from corrupted reasoning trajectories (Li et al., 2025) in significant way - we show that reasoning patterns from weaker models can effectively bootstrap the learning process to build more capable ones, suggesting that the presence of cognitive behaviors matter more than just the correctness of the outcomes.
The above results show that certain cognitive behaviors are necessary for self-improvement However, our priming method for inducing behaviors in the initial model was domain- specific, relying on the Countdown game. This may adversely impact the generalization of
7

---

Published as a conference paper at COLM 2025
(a) Priming Liama Incorrect CoTs Score
(b) Priming L'ama w/ Incorrect CoTs Response Lengths
14
1202
response length
0.4
002
02
200
RL steps
Llama
Owen
All Strategies
All Stratogies w/ Incorrect Answers
Figure 6: Effects of Incorrect Mhain-of-Thoughe Priming Model Performance and Output Characteristics Evaluation of (a) Countdown task performance scores and (b) response length distributions comparing four conditions: Llama3.2-30 and Qwen2.5-3B base models, Llama3.2-3B with correct all-strategy priming, and Llama3.2-3B primed with incorrect reasoning examples.
the resulting reasoning. Can we instead enable self-improvement by modifying model's pretraining distribution to increase the frequency of beneficial reasoning behaviors?
Behavioral Frequencies in Pretraining Data. We first analyze the natural frequency of cogmitive behaviors in pre-training data, focusing on OpenWebMMath (Paster al., 2023), and FineMath (Allal et al., 2025), which are constructed specifically for mathematical reasoning, Using Qwen-2.5-32B as a classifier', we analyze 200,000 randomly sampled documents for the presence of our target behaviors. Even in this math-focused corpus, cognitive behav- iors such backtracking and verifications appear infrequently, suggesting that standard pretraining provides limited exposure to these crucial patterns (see Fig. 7).
3.5 Selectively amplifying behaviors in pretraining data
Behavioral Augmentation of the Pretrain- ing Data. To test whether artificially in- creasing exposure to cognitive behaviors en- hances the potential for self-improvement, we develop a targeted continued pretrain- ing dataset from WebnWabatth First, ing Qwen-2.5-32B as classifier, we ana- lyze mathematical documents from the pre- training corpus for the presence of our tar- get reasoning behaviors. This allows us to create two contrasting sets: one with the cognitive behaviors and control set that shows minimal evidence of these behav- iors. We then use Qwen-2.5-32B to rewrite each document in the set into structured format, preserv- ing the natural presence or absence of cogni- tive behaviors from the source documents. The final pretraining datasets each contain a total of 8.3 million tokens. This approach allows us to isolate the impact of reasoning behaviors while controlling for the format and amount of mathematical content during
Aaraggeooo count per document
0.00
1000000
Verifications
Backtracking
Backward
Subgoal Setting
Chaining
creestin-ttoughtt-ee
Figure 7: Behaviors present in Math Pretrain- ing Datasets. An analysis of the behaviors present in 200,000 randomly sampled docu- ments of Open OpenWebMaa WebMath and Fine We measure the average count of the behaviors in each document.
pre-training.
5e Owen-22.-32BB to balance throughput, capability and cost.
8

---

Published as a conference paper at COLM 2025
Curated Protraining Score
1bl Curated Pretraining Response Longths
(c) Behaviors the Curated Protraining Model
100
One
Becosding
riiedd Caroud
resppox lengthh
Mein Comr
Drinng
RI 156
205
204
RC steps
110
23
RL staps
Figure 8: Impact of curated pretraining model performance and behavior. (a) Com- parison of performance scores across base models (Llama-3.2-3B, Qwen-2.5-3B) and Llama variants with curated pretraining versus behavior-minimized control. (b) Evolution of response lengths during training for each model configuration. (c) Emergence and develop-
ment of specific cognitive behaviors in the curated pretraining model over training steps.
After pre-training Llama-3.2-24 on these datasets and applying reinforcement learning, we observe that: 1) the behavior-enriched model achieves performance comparable to Qwen, while the control model shows limited improvement (Fig. 8a) and 2) behavioral analysis of the trained models reveals that the behavior-enrichee of reasoning behaviors throughout training, while the control shows behaviors similar to the base Llama model (Fig. 8c). These results demonstrate that targeted modification of pre-training data can successfully induce the cognitive behaviors necessary for effective self-improvement through reinforcement learning.
variant maintains high activation
4 Discussion
We have found that model's initial exploration of cognitive behaviors rehaviors-particulaalyy its tendency toward verification, backtracking, subgoal setting, and backward chaining- -plays crucial role in enabling self-improvement. Models that naturally exhibit these reason- ing behaviors (such as Qwen-2.5-3B) show dramatically better improvement through RL compared to models lacking these behaviors (such Llama-3.2-3B). Priming models with cogmitive behaviors, by small amount of finetuning, enabled significant performance gains even in models that initially lack these capabilities. Remarkably, this holds even when primed with incorrect solutions that exhibit the target behavioral patterns, suggesting that cognitive behaviors matter more than solution accuracy. Together these results indicate that the presence of cognitive behaviors is causal factor enabling self-improvement through RL. Our initial priming experiments used data based on the Countdown game for train- ing, possibly limiting generality. We thus developed more diverse behavior-enriched training dataset derived from OpenWebMath. Training Llama on this set then led to self- improvement comparable to Qwen, demonstrating that the capacity for improvement can be engineered through careful curation of pretraining data.
When humans try to solve problems that are difficult but not unsolvable for them, they exhibit certain behaviors that support the problem-solving processes, structuring search over the space of possible solutions to a problem. These cognitive behaviors are usually sequential, deliberate and dependent on the problem space (Simon & Newell, 1971). Corre- spondingly, the cognitive behaviors that are amplified or suppressed during RL training are likely to be highly dependent on the tasks and environments being optimized for. In our studies using Countdown, backtracking and verification were the most critical. This raises important questions about the patterns that enable self-improvement in tasks such as coding, game play or creative writing. We believe that the principle described here will extend to other domains, but future work should explore how task-specific constraints interact with cognitive behaviors. Further, the cognitive behaviors specified in this work are not exhaustive; other behaviors are worth exploring, such as making analogies (Mitchell, 2021) and identifying one's 's existing state of knowledge (Metcalfe, 1986).
In conclusion, findings show how cognitive behaviors enable language models to show relf-improveement - effectively using increased test-time compute to solve increasingly more challenging problems. Humanity has rich inheritance of cognitive behaviors that
9

---

Published as a conference paper at COLM 2025
enable effective reasoning. Future artificial intelligence may go beyond learning to use these existing behaviors it may discover new ones, potentially revealing entirely new approaches to reasoning and computation.
Acknowledgments
We would like to thank Charlie Snell, Eric Zelikman, Archit Sharma, Rafael Rafailov, Alex Havrilla, Ulyana Piterburg, Chase Blagden, Dimitris Papailiopoulos, and Jan-Philipp Fränken for their discussions and support. KG was supported by an HAI-SAP Grant and an NSF Expeditions grant; AS was supported by the NSF GRFP Fellowship. Compute for this work was provided by Synthlabs, Stanford Marlowe and the Toyota Research Institute (TRI).
References
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arviv preprint arXiv:2402.14740, 2024.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Four- rier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric URL https: org wtbs/2502.ce 02737..
training of a small language model, 2025.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https: / http///xxxvvv///////e///e/////// orw.orgg/////.. 2b//////.222222/// 21787.
Countdown. Countdown (game show). Online, 2024. URL https://en www.eewwiiiiiiaiiaeeee org/ wiki/Countdown (gamdddssw_aaa___soossoooo [Accessed 29-March-2024],
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. ar Xiv preprint ar arXit:2502.004456 2025.
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. arXiv preprint ar arXiv:2305.1916 2023.
Kanishk Gandhi, Denise HJ Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (sos): Learning to search in language. In First Conference on Language Modeling, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chem, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Qlano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar,
S
10

---

Published as a conference paper at COLM 2025
Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Praijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierr Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, A Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanaan Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal
11

---

Published as a conference paper at COLM 2025
Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekk, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Ilama 3 herd of models, 2024. URL https://arxxv..rgg/././s
Summer
21783.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXito preprint Xix::2200 100.0.....990 12948, 2025.
Zhang, Runxin Xu, Qihao Zhu,
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models reason with reinforcement learning. ar Xiv preprint ar Xrxie:2403.0464/ 2024.
Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-thought via latent-variable inference. In NeurlPS, 2023.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models, 2025. https:// //axivv
htttpp://aaxxi..or///////////// wrgg/bb/////0 03262.
Hyeonbin Hwang, Doyoung Kim, Seungone Kim, Seonghyeon Ye, and Minjoon Seo. Self- explore: Enhancing mathematical reasoning in language models with fine-grained re- wards. arXiv preprint ar arXio:2404.10346 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai of system card. arxiv preprint arXiv:2412.1672)) 2024.
Andy Jones. Scaling scaling laws with board games, arv preprint arXio:2104.03113 2021.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arxiv preprint arXiv:2409. 12977, 2024.
12

---

Published as a conference paper at COLM 2025
Woosuk Kwon, Zhuohan Li, Styuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposuum on Operating Systems Principles, 2023.
Lucas Lehnert, Sainbayar Sukhbaatar, Difia Su, Qinqing Zheng, Paul Mevay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping. Xixiv preprint 1xXi::2402114083 2024.
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. ar arxiv preprint arXiv:2403.04706 2024.
Dacheng Li, Shiyi Cao, Tyler Griggs, Shu Liu, Xiangxi Mo, Shishir G Patil, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Lims can easily learn to reason from demonstrations structure, not content, is what matters! arXiv preprint ar aXXiv:2502.07374, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. There may not be aha moment in rl-zero-like training a pilot study. oat-zero, 2025. Notion Blog.
notimo.ooooon...mmmmmmoomo.oo... site/
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.orrg hwtp://wwww//////////oooooos//// 11711.0501.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint ar Xiv:2406.06692, 2, 2024.
Janet Metcalfe. Feeling of knowing in memory and problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition, 12(2):288, 1986.
Melanie Mitchell. Abstraction and amalogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1))79-101, 2021.
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. 2025. Accessed: 2025-01-24.
https:/
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. Advances in Neural Information Processing Systems, 37:55249-55288 2025,
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https:/: //axxvv...iiv/////////// www.av.orrrg.eoo orgg/bss2411 15115.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference Language Modeling, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 17370
policy optimization algorithms, 2017. URL
13

---

Published as a conference paper at COLM 2025
John Schultz, Jakub Adamek, Matej Jusup, Marc Lanctot, Michael Kaisers, Sarah Perrin, Daniel Hennes, Jeremy Shar, Cannada Lewis, Anian Ruoss, et al. Mastering board games by external and internal planning with language models. ar preprint arXio:2412.12119 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: http://twaaaaaa/aaa//////// org/abs/ 2402, 03300.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. ariio preprint ar Xiv: 2409.19256, 2024.
Herbert A Simon and Allen Newell. Human problem solving: The state of the theory in 1970. American psychologist, 26(2):145, 1971.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. aiv preprint xxxh:2408.00314 2024.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations. Xiv preprint xrXiv:2312.08935, 2023.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4))229-256, May 1992. ISSN 0885-6125. doi: 10.1007 BBF0099260099 URL
Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, et al. A comparative study on reasoning patterns of openai's ol model. Xiv preprint ar arXiv:2410.13633 2024.
Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system reasoning llms: Learning how to think with meta chain-of-though. arXiv preprint 1rXiv:2501.04682 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems,, 36:11809-11822, 2023.
Guanghao Ye, Khiem Due Pham, Xinzhi Zhang, Sivakanth Gopi, Baolin Peng, Beibin Li, Janardhan Kulkarni, and Huseyin A Inan. On the emergence of thinking in 1lms i: Searching for the right intuition. arxiv preprint ar A612:22500, 2025.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2 how to learn from mistakes on grade-school math problems. arXio preprint arXiv:2408.16229 2024.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in lims, 2025. URL https://arxii wrw. htrrg/abs/2500 03373.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman. Quiet-star: Language models can teach themselves to think before speaking. In First Conference on Language Modeling, 2024.
14

---

Published as a conference paper at COLM 2025
1.0
0.8
score
0.44
0.42
044
0:38
0.2
0.19
All Strategies
Backtracking
Backtracking
Backtracking
Backtracking
Empty CoT
Backwward
Subgoe
+ Venfiictioon
Chaining
Setting
Figure 9: Scores of the priming datasets generated with Claude. Analyzing the average scores of Claude when instructed to solve Countdown employing different cognitive behav- iors.
A
Data Generation
For experimental setup, we implemented the Countdown task according to the method- ology outlined by Gandhi et al. (2024) and Pan et al. (2025). The task consists of an equal distribution (50-50 split) between 3-digit and 4-digit Countdown problems. Both the start- ing numbers and the target number were generated using random sampling to ensure variability across trials while maintaining consistent difficulty parameters. This random- ization approach allowed to assess mathematical problem-solving capabilities across representative range of Countdown games.
B Priming
To prepare our supervised fine-tuning (SFT) data, we use Claude 3.5 Sonnet (claude-3-5- 20nnet-20241022) generate reasoning trajectories. We developed five distinct SFT datasets, each designed to capture different cognitive behaviors:
1. Backtracking Only: This dataset focuses exclusively on the backtracking strategy, where the model explores solution paths and retreats when encountering dead ends.
2. Backtracking with Answer Verification: In addition to backtracking, this dataset incorporates answer verification, where the model checks its intermediate solutions with the target number.
3. Backtracking with Subgoal Setting: This dataset combines backtracking with explicit subgoal setting, where the model breaks down complex problems into manageable intermediate steps.
4. Backtracking with Backward Chaining: This dataset demonstrates backward chain- ing with backtracking, where the model works backward from the goal state to the initial state.
5. All Strategies: This comprehensive dataset incorporates all four reasoning strategies mentioned above.
To control specific reasoning behaviors across each dataset, we implemented customized system prompts that explicitly guide Claude toward the desired reasoning patterns. These
15

---

Published as a conference paper at COLM 2025
3.5
3.0
2.5
count
2.0
avg
023
1:00
0.20
Backtracking Backtracking Subgoal Setting
Backtracking Backtracking All Strategies Backward + Verification
Chaining
Figure 10: Behaviors present in priming datasets. Analyzing the average counts of behav- iors when Claude is instructed to only show specific behaviors in its thoughts.
system prompts were carefully crafted to elicit distinct problem-solving approaches while maintaining consistency across datasets (see Fig. 10 for analysis of behaviors present). To create datasets rich in only the targeted behaviors and determine causal efficacy, we instructed Claude to use exclusively the specified cognitive behavior while prohibiting all others. For example, in the "Backtracking Only" condition, we emphasized through the system prompt that Claude could only use backtracking and was explicitly not permitted to verify answers, set subgoals, or work backwards from the target. For each of the five datasets, we selected 1,200 unique Countdown games as our problem set. We prompted Claude to solve these games while adhering to the specified reasoning strategy for each dataset. We then evaluated the accuracy Claude's solution trajectories, with results presented in Fig. 9.
We provide the SFT hyperparameters Tab. 1.
Data & Model
Training Validation Dataset Size
1000 / 200
Context Window
2048
Training/ Validation Batch Size
64 64
Optimization
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Peak Learning Rate
1e-5
Warmup
Linear (5% of total steps) Cosine
Annealing
Total Epochs
5
Table 1: SFT Hyperparameters
C Reinforcement Learning
We provide the PPO training hyperparameters in Tab. 2.
AIl prompts available in our Github repository: behaviors.
cognitive-
16

---

Published as a conference paper at COLM 2025
Data & Model
Training/Validatn Validation Batch Size
256 / 1312
Context Window
256 prompt 1024 response tokens
Optimization
Actor Learning Rate
le-6
Critic Learning Rate
le-5
KL Coefficient
0.001
PPO Mini-batch Size
128
Number of Rollouts
4
Rollout Temperature
1.0
Reward Structure
Correct Answer
1.0
Incorrect Answer
0.0
Correct Format Bonus
0.1
Table PPO Training Hyperparameters
D Metrics
Next, we describe our behavioral evaluation metrics. The 4 behavioral metrics we track are:
1. Average Backtracking Count
2. Average Verification Count
3. Average Backward-Chaining Count
4. Average Subgoal-Setting Count
We generate samples from the trained model at using a temperature of 1.0, and a maximum of 1024 tokens. Each of these metrics track the average occurrence of each reasoning behavior. We develop classification pipeline using GPT 4o-mini (the classifier model).
Our classification pipeline asks the classifier model 4 questions per reasoning trajectory. In each question, we provide the classifier model examples of each behaviors, so for instance for answer verification in Countdown, which is not equal to 22" in the prompt.
add examples like "This sequence results in
For each reasoning behavior, ask the classifier model to count up and report the number of distinct occurrences. We sample 512 tokens from the classifier model with temperature for reproducibility.
E Pretraining Data Interventions.
Analyzing Frequency in Pretrained Data. To investigate the natural frequency distribu- tion of various reasoning behaviors in the OpenWebMath (Paster al., 2023) and FineMath (Allal et al., 2025) datasets, we used the Qwen2.5-32B model as the classifier, deployed via vLLM (Kwon et al., 2023) for efficient inference. The analysis encompassed 200,000 randomly sampled documents from both datasets. For each document, the classifier model was prompted with examples demonstrating specific cognitive behaviors, following the classification framework described in App. D. For instance, to identify instances of Subgoal Setting, we provided examples such as: "To solve this system of equations, let's first isolate x in the first equation, then substitute it into the second." To ensure reproducibility, all classification runs were conducted with a temperature setting of O. Each document was processed with generation length of 1024 tokens per cognitive behavior. This methodical approach enabled us to quantitatively assess the prevalence of different cognitive behaviors across the mathematical content in both datasets.
17

---

Published as a conference paper at COLM 2025
Verification
Backtracking
Subgoal Setting
Backward Chaining
cla.de
2/2
1:23
2.44
dauds
2714
2:28
837
2.80
dlude
64
3.2
1.44
dauds
8:18
224
2:28
human
1.4
3.43
1:22
Yumion
6..
882
2.17
human
6:14
5.84
4.23
human
4.2
0.14
6o-mii
doude human 1 human
6o-min drude human human
4o-ni'i clx.de human human
4o-iini dloude human hurmnn
Figure 11: Inter-rater reliability for counting and classifying behaviors. Intraclass Correla- tion Coefficient (ICC3) values for (a) Verification (b) Backtracking (c) Subgoal Setting (d) Backward chaining for labels from GPT-4o-mini, Claude, and two human raters.
Curating Pretraining Data. We write similar scripts using the same classification pipeline to perform two data preprocessing steps:
1. Isolating Reasoning Behaviors. We leverage our classification pipeline to sys- tematically identify and extract passages exhibiting specific cognitive behaviors from the source datasets. For each document, the classifier determines whether each of the cognitive behaviors is exhibited in that document. These binary evalua- tions, are then used to populate two datasets - the behavior curated training, and dechaviormmiimizee control training datasets.
2. Re-formatting for Training. We process the data into a structured format utilizing XML tags to delineate question, thinking process, and answer components while maintaining the integrity of the original question content. During the paraphrasing process, we implement first-person language specifically within the thinking sec- tions to accurately represent cognitive processes. For the behavior-curated training dataset, we incorporate explicit instructions to ensure the paraphrased thoughts contain the targeted cognitive behaviors under investigation. Conversely, for the behavior-minimized dataset, we omit these specific instructions to establish an appropriate control condition.
Interrater Reliability for Classifying Behaviors
To assess the reliability of classification and counting of behaviors from GPT-4o-mini, conducted an inter-rater reliability analysis using the Intraclass Correlation Coefficient (ICC3). We compared agreement of GPT-4o-mini, Claude, and two human annotators of four key problem-solving behaviors: verification, backtracking, subgoal setting, and back- ward chaining. We randomly sample 100 reasoning trajectories from different conditions and trained models to measure agreement.
GPT-4o-mini showed (see Fig. 11) strong agreement with other raters across most behav- iors, though with varying levels of consistency. For verification, GPT-4o-mini achieved substantial agreement with both Claude (ICC3 0.70) and human raters (ICC3 0.66 and 0.65). For backtracking, GPT-4o-mini maintained robust reliability with Claude (ICC3 0.74) and slightly lower but consistent agreement with human raters (ICC3 = 0.67). Subgoal setting showed the strongest consistency, with GPT-4o-mini achieving high agreement with Claude (ICC3 0.88) and human raters (ICC3 0.70 and 0.77). Backward chaining demonstrated lower agreement levels, with GPT-4o-mini showing stronger correlation with Claude (ICC3 = 0.78) than with human raters (ICC3 0.55 and 0.40). These results suggest that GPT-44-mini can reliably identify and classify most cognitive behaviors.
G Test-time Scaling.
We test how model performance scales with different token budget constraints at inference time. By varying the maximum allowed token length from 27 (128) to 200 (1024) tokens,
18

---

Published as a conference paper at COLM 2025
0.8
Scofe
0.6
0.4
0.
0.0
2 token budget
2'
2
Figure 12:
Test-time Scaling by Varying Token Budgets during Inference. The graph shows how different training checkpoints of the Llama model, fine-tuned with RL using the all-strategies primed model when tested with different maximum token lengths (128, 256, 512, and 1024 tokens). This analysis demonstrates how model performance scales with available computational resources at inference time.
analyzed the relationship between computational resources and model performance. We test training checkpoints ranging from 200 steps of PPO training for the all-strategies primed Llama model. Performance consistently improves with larger token budgets across all checkpoints, but with diminishing returns after 512 tokens (see Fig. 12). Later checkpoints (150-200 steps) show stronger performance even with restricted token budgets, achieving scores of =00.4 with just 128 tokens compared to near-zero performance for early checkpoints under the same constraints. This suggests that training helps the model learn to use limited token budgets more efficiently - more train-time interactions lead to more efficient test-time performance (Jones, 2021). The scaling behavior also reveals an interesting convergence pattern while early checkpoints (0-20 steps) show nearly log-linear scaling with token budget increases, later checkpoints exhibit more logarithmic scaling, suggesting they have learned to make better use of additional tokens. By 1024 tokens, the best checkpoints achieve scores around 0.7, with relatively small gaps between checkpoints after 100 training steps.
H Transfer of Behaviors to Other Domains.
To evaluate the generalization of learned cognitive behaviors, we investigated whether models trained to exhibit specific problem-solving strategies in mathematical reasoning could transfer these behaviors to other data distributions. First, we compared two variants: model trained with curated pretraining dataset designed to amplify cognitive behaviors in mathematical reasoning, and a control model trained on behavior-minimized data on 200 questions from GPQA (Google Proof Question Answering; Rein al. (2024)). The results (see Fig. 13) demonstrate significant transfer of learned behaviors, with the curated pretraining model exhibiting substantially higher frequencies of all four cognitive strategies (verification, backtracking, subgoal setting, and backward chaining) when solving general knowledge questions. Most notably, subgoal setting showed the strongest transfer effect, with an average of 6.5 instances per question in the curated model compared to 0.7 in the control. This suggests that cognitive behaviors amplified in mathematical reasoning can generalize effectively to broader question-answering contexts. It should be noted that there is no significant performance difference between the two models (both score about 12%). This performance parity despite behavioral differences suggests is because the model is limited in its forward inference capabilities rather than its problem-solving strategy in other words, the model can learn to approach problems more systematically, but still struggles with the basic reasoning and knowledge needed to arrive at correct answers.
RL training on Countdown transfers to other domains. Next, we tested a Qwen model trained with RL on Countdown and the base Qwen model on 200 questions from GPQA
19

---

Published as a conference paper at COLM 2025
Upma Cunted Peraring
average count
Figure 13: Transfer of Behaviors Question Answering. Average frequency of four cognitive behaviors (verifications, backtracking, subgoal setting, and backward chaining) observed when solving GPQA questions. Comparison between Llama model trained with curated pretraining data (orange) that amplified these behaviors in mathematical reasoning versus behavior-minimizen control model (blue).
(s) GPOA
(b) MATH
Coued
Bachbaking
Subyoa Setleg
Buctnoding
Subyod Seting
Chaning
Figure 14: Transfer of Behaviors to Question Answering learned through RL. Average frequency of four cognitive behaviors (verifications, backtracking, subgoal setting, and back- ward chaining) observed when solving (a) GPQA and (b) MATH questions. Comparison between Qwen model RL finetuned on Countdown (green) that amplified these behaviors in mathematical reasoning versus the base Qwen model (purple).
and MATH each. The results (see Fig. 14) demonstrate similar accentuation of learned behaviors, with the RL finetuned model demonstrating higher frequencies of all four cognitive behaviors. Surprisingly, we find a performance increase from 38% to 50% with our prompt format and 4-shot 44% to zero-shot 50% with Qwen's 4-shot prompt format on the MATH dataset. Similar the previous finding, we observe substantial increases in backtracking (0.15 to 0.59) and verification (0.02 to 0.29) in MATH and on GPQA, all four behaviors show meaningful increases: verification (0.008 0.03), backtracking (0.019 to 0.114), subgoal setting (0.478 to 0.616), and backward chaining (0.128 to 0.313). Notably, subgoal setting and backward chaining were not demonstrated by the Countdown RL finetuned Qwen model on Countdown (see Fig. (Middle)) but despite this, these particular cognitive behaviors show accentuated activations post RL finetuning. These results taken together suggest that cognitive behaviors induced in the context of mathematical reasoning have a larger signature across other question-answering domains and beyond the training domain.
20

---

Published as a conference paper at COLM 2025
(a) Instruct Model Scores
(b) Instruct Model Response Lengths
1.5
2390
Uana
Ove
2020
8000
300
sto05
100
1.2
172
50
steps
120
150
175
[c] Owen Instruct Behaviors
(cl Liama Instruct Behaviors
12
10
Suleull lockig
Ascoaari Cla vig
Kurbur Chullgg
ove.ggh.com
01
stops
25 30 100 125
150
175
Figure 15: Analysis of the effect of instruction-tuning on the four cognitive behaviors. Plots compare (a) score and (b) response lengths between base models and their instruction tuned counterparts; and plotting the four key cognitive behaviors through the course of RL on Countdown with (c) Qwen2.5-3B-B --en.st-tBn and (d) -lama3.2-3B-Inan
I Results for Instruction-tuned Models
We investigated whether instruction tuning affects the emergence and amplification of cognitive behaviors during reinforcement learning (RL). Instruction tuning teaches models to reliably follow user instructions, but its impact on cognitive behaviors through RL is unclear. Toward this, we ran our behavioral probes on instruction-tuned variants of Llama- 3.2-3B and Qwen-2.5-3B models on Countdown through RL training, following the same methodology applied to their base model counterparts. Our results (see Fig. 15) show that instruction-tuned models from both families nearly saturate task score and use more of their token budgets than the base models. More significantly, we observed the emergence and progressive strengthening of key cognitive Sahaviors-particullae backtracking. These findings suggest that instruction-tuun tunnnngtnnnngggu datasets not only teach models to follow instructions but also contain examples of the cognitive behaviors that then surface during subsequent RL. finetuning.
verification and
J Behavioral Analysis for Larger Models
We conducted scaling experiments to determine whether our findings generalize across model sizes and to investigate the role of scale in cognitive behavior emergence. Our initial results revealed a striking difference between model families: Qwen models consistently de- veloped sophisticated cognitive behaviors during RL training, while Llama models showed no evidence of such behavioral emergence. This disparity raised the question about whether the observed differences were artifacts of the specific 3B parameter scale or represented fundamental data differences between the model families. To address this question sys- tematically, we extended our analysis to larger model variants, training Qwen2.5-14 and Llama3.2 3-BBBB using GRPO on Countdown. The results (see Fig. 16) provided compelling evidence that our earlier findings are robust across model scales. Qwen2.5-14B demon- strated clear and consistent emergence of three of the four cognitive behaviors we identified with patterns similar to those observed in Qwen2.5-3B, Conversely, Llama3. 3-BB failed to exhibit any meaningful cognitive behaviors throughout RL training, despite its substantial
21

---

Published as a conference paper at COLM 2025
(a) Owen 14B Behaviors
(b) Llama 8B Behaviors
Bortons Eadrrwlrg
RL stogs
RL 5tops
Figure 16: Analysis of the effect of scaling on the four cognitive behaviors. Plotting the four key cognitive behaviors through the course of RL training for (a) Qwen2.5-14B and (b) Llama3.1-80 on Countdown.
parameter increase over Llama3.2-3B. The results strongly indicate that model scale alone is insufficient for developing cognitive behaviors during RL training. Instead, the capacity for cognitive behavior emergence appears to be determined by whether the models are exposed to the cognitive behaviors through the course of their pretraining, and that these capacities cannot be reliably induced through scaling alone.
22


Reviews:
[{"review_id": "E2UhxMe9ii", "reviewer": "Reviewer_UQGC", "summary": "This paper dives into a curious difference observed when training language models to improve using reinforcement learning: some models get significantly better, while others just sort of hit a wall. The authors noticed this particularly when comparing Qwen-2.5-3B and Llama-3.2-3B on a math game called Countdown, where Qwen improved dramatically and Llama didn't, even with the same training. They wondered what intrinsic qualities in a model allow for this kind of self-improvement.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think one of the paper's main strengths is identifying and systematically studying these specific cognitive behaviors. It moves beyond just looking at final performance and provides a structured framework for analyzing how models attempt to solve problems. The experimental design, particularly the priming experiments and the finding that incorrect solutions with proper reasoning patterns still enable improvement, is quite insightful and provides strong evidence for their hypothesis about the importance of behaviors over just correctness. The connection back to pretraining data is also a valuable contribution, suggesting ways to engineer models with better self-improvement potential from the ground up.", "reasons_to_reject": "N/A", "questions_to_authors": "How would this be like if you run it on 70B+ or MoE model?", "extra_scores": {}}, {"review_id": "3OWjFBJn3s", "reviewer": "Reviewer_CcUW", "summary": "This paper investigates the fundamental mechanisms that enable language models to improve their reasoning capabilities, specifically learn self-improving, through reinforcement learning (RL). The authors focus on two 3B-parameter models, Qwen-2.5-3B and Llama-3.2-3B, for comparison when trained on the Countdown game.\nThe research identifies four key cognitive behaviors critical to effective problem-solving: verification (error-checking), backtracking (abandoning failing approaches), subgoal setting (breaking down complex problems), and backward chaining (reasoning from desired outcomes to initial inputs). Surprisingly, the study finds that Qwen naturally exhibits these behaviors, while Llama does not, leading to significant performance disparities during RL training. \nCrucially, when Llama was primed with datasets emphasizing these cognitive behaviors or pretrained on OpenWebMath dataset that amplifies such behaviors, its performance approached that of Qwen and presents desired test-time scaling behaviors. This suggests that a model's initial reasoning capabilities fundamentally determine its potential for self-improvement.", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper demystifies the long CoT training, performing a rigorous ablation study to present a reproducible recipe on how to train a o1-like model: priming LLMs with cognitive patterns as priors for RL.\n- The experimental setup is rigorous and the results are convincing, even though tasks and models are a bit limited.", "reasons_to_reject": "- How the authors identified four cognitive behaviors looks a bit arbitrary. \n- They only focus on Countdown, a task that may be easier to elicit self-reflection as there is an explicit verification signal (if the calculated result equals target), which is somehow limited.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DeY9DiGFRP", "reviewer": "Reviewer_ZG7V", "summary": "The paper investigates why some language models (LMs) improve substantially under reinforcement learning (RL), while others plateau. Focusing on two models—Qwen-2.5-3B and Llama-3.2-3B—the authors identify four cognitive behaviors (verification, backtracking, subgoal setting, and backward chaining) as critical for enabling self-improvement. Qwen naturally exhibits these behaviors, leading to superior performance, while Llama does not unless primed. Through controlled experiments, the study demonstrates that even incorrect solutions containing the right reasoning patterns can boost performance, suggesting that exhibiting these behaviors is more important than correctness. Further, the authors show that pretraining on behavior-rich data enables Llama to match Qwen’s RL-driven improvement, establishing a causal link between initial cognitive behaviors and RL efficacy.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a clear and testable framework for analyzing cognitive behaviors, offering a principled way to study emergent reasoning in LMs beyond task accuracy.\n\n- The use of carefully controlled priming and pretraining experiments, including behavioral classifiers and ablative baselines (e.g., empty CoTs), supports strong causal claims regarding the role of reasoning behaviors.", "reasons_to_reject": "- The primary testbed (the Countdown game) is narrow and highly structured. Although it's suitable for isolating reasoning behaviors, it remains unclear whether findings generalize to more diverse domains such as math reasoning, code synthesis and other reasoning domains.\n\n- While the authors focus on four cognitive behaviors—verification, backtracking, subgoal setting, and backward chaining—they do not provide a principled rationale (e.g., from cognitive science or computational learning theory) for selecting these particular behaviors. Further, while behaviors are examined individually and in combinations, the work doesn't deeply explore how these behaviors interact or conflict.\n\n- The paper’s comparisons are limited to Qwen-2.5-3B and Llama-3.2-3B.  Without broader coverage: It’s unclear whether the observed gaps are due to behavior differences or opaque model-specific idiosyncrasies. I understand that adding further experiments on larger scale models might be infeasible for an academic project, I would not deem this point a true reason to reject. If the authors could add more experiments on larger scale models (>7B, e.g, 14B or 32B), I'd consider raising my score.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffDGg6T7N4", "reviewer": "Reviewer_1iHv", "summary": "This paper investigates the behaviors of two models Qwen2.5-3B and Llama3.2-3B  when training with RL on the task of Countdown. It introduces an analysis framework based on four cognitive  behaviors. It is found that Qwen naturally exhibits these behaviors, while LLaMA requires behavior priming via finetuning on specifically designed data. Further experiments show that amplifying LLaMA's behaviors in the math domain can also align its behavior development curve with that of Qwen.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a timely and detailed analysis of cognitive behaviors in language models trained via RL on the Countdown task, offering insights into model-specific reasoning patterns.\n2. The paper is well written, clearly structured and easy to follow.\n3. The proposed framework of four cognitive behaviors provides a useful lens for understanding and analyzing models' self-improvement capabilities.", "reasons_to_reject": "1. All experiments are conducted on 3B models. It remains unclear whether the observed behaviors generalize to larger or smaller model sizes.\n2. The analysis is restricted to base models. Given that instruct models (such as Qwen2.5-3B-Instruct) are trained on more diverse datasets, their cognitive behaviors may differ.", "questions_to_authors": "1. If Qwen is further trained on the math data that explicitly contains cognitive behaviors, will this amplify its existing behaviors or potentially conflict with its natural behaviors?\n\n2. Appendix H shows that models trained on math data generalize to GPQA in terms of cognitive behaviors, but fail to improve performance. Can the cognitive behaviors developed during RL on the Countdown task transfer to other domains, such as math? If so, do they lead to any performance gains due to these behaviors?\n\n3. In Figure 8(c), there is a noticeable increase in verification behavior between steps 50–90. Could you please elaborate on why the model behaves this way? Have you checked the outputs during these steps to explain the spike?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper identifies that certain models are inherently more amenable to improved reasoning via RL finetuning due to the pretraining data or priming. The reviewers generally agree that the work is principled and systematic, providing a rigorous framework and axes on which to evaluate these reasoning capabilities that leads to the clear and convincing conclusions. The authors additionally addressed the concerns relating to two potential limitations (model sizes, categorization of reasoning), with only one (choice of task) which remains somewhat unaddressed. Nonetheless, the reviewers believe this study is already sufficiently well-structured and informative that I recommend acceptance.
Decision: Accept