Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
Authors: Xingwu Chen
Tianle Li
Difan Zou
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Language Models
Reasoning Patterns
Training Dynamics
Primary area: learning theory

Abstract: While reinforcement learning (RL) demonstrated remarkable success in enhancing the reasoning capabilities of language models, the training dynamics of RL in LLMs remain unclear. In this work, we provide an explanation of the RL training process through empirical analysis and rigorous theoretical modeling. First, through systematic reasoning-pattern-level and token-level analysis across the RL training process, we show that while different reasoning patterns exhibit relatively stable success rates during training, RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions to affect model performance. Building on these empirical insights, we develop a theoretical framework to understand the training dynamics of RL with two typical rewards: verifiable reward (RLVR) and model's internal feedback (RLIF). For RLVR, we analyze the training dynamics under two special cases: one where models readily converge to optimal reasoning strategies, and another where optimization becomes challenging, revealing that the base model's reasoning quality is crucial for determining convergence behavior. For RLIF, we examine how internal rewards initially improve model performance but can potentially lead to degradation with continued training. Extensive experiments validate our findings, advancing both theoretical understanding and practical applications of RL in language model enhancement.

Published as a conference paper at ICLR 2026
RESHAPING REASONING IN LLMs: A THEORETICAL ANALYSIS OF RL TRAINING DYNAMICS THROUGH PAT- TERN SELECTION
Xingwu Chen
Tianle Li*
School of Computing & Data Science The University of Hong Kong xingwul connect. .hu.h.. hk
Institute of Data Science The University Hong Kong t tianle el @cconecc. hku hk
Difan Zou
School of Computing & Data Science Institute of Data Science The University of Hong Kong dzouêcs. hku.hk
ABSTRACT
While reinforcement learning (RL) demonstrated remarkable success in enhane- ing the reasoning capabilities of language models, the training dynamics RL in LLMs remain unelear. In this work, provide explanation of the RL training process through empirical analysis and rigorous theoretical modeling. First, through systematic reasoning sertooinngpatten---eee pattern-level and token-level analysis across the RL training process, we show that while different reasoning patterns exhibit relatively stable success rates during training. RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions to affect model performance. Building on these empirical insights, we develop a theoretical framework to understand the training dynamics of RL with two typical rewards: verifiable reward (RLVR) and model's internal feedback (RLIF). For RLVR, we analyze the training dynamics under two special cases: one where models readily converge to optimal reasoning strategies, and another where optimization becomes challenging, revealing that the base model's reasoning quality is crucial for deter- mining convergence behavior. For RIIF,, we examine how internal rewards initially improve model performance but can potentially lead degradation with continued training. Extensive experiments validate our findings, advancing both theoretical understanding and practical applications of RL language model enhancement.
INTRODUCTION
Recently, state-of-the-art reasoning models such as Gemini2.5 (Comanici 2025). Qwen3
(Yang et al. 2025), and DeepSeek-R1 (Guo et al. 2025) demonstrate exceptional performance on complex logical tasks including mathematics (Shao al. 2024 Zeng al. 2025 Yu et al., 2025)
and programming (Zhu et al.) (0244] Yang et al.) [omaniic et al. 2025). Reinforcement learning (RL) serves as a key technique behind this success, demonstrating the potential to elevate model capabilities to a new level.
The success of RL has triggered research into underlying mechanisms for LLMSs. By comparing pass @k k performance, Yue et al. (2025a) show that models post-trained with RL struggle to surpass base models, suggesting that RL may not clicit fundamentally new reasoning patterns. From an entropy perspective, Cui al. (2025a): Zhang al. (2025b) theoretically prove that RL-based methods can reduce policy entropy, with Cui et (2025)) also empirically establishing a connection between model performance and policy entropy. |Wang al. (2025a): Huan al. (2025): Meng et al. (2026) further demonstrate that RL primarily optimizes a sparse subset of critical tokens. Despite
"Equal contribution.

---

Published as a conference paper at ICLR 2026
these various efforts understand RL mechanisms, the underlying dynamics of the RL training process remain incompletely understood both empirically and theoretically.
To understand the RL training process, first present systematic reasoning-pattee-llewe pattem-level and token-level analysis across RL training. Specifically, focusing on the training dynamics, we not only examine the ranking shifts across training, but also use LLM-based and rule-based methods to extract and classify reasoning patterns from models' responses, analyzing the corresponding success rates and distributions during training. Compared with previous works (Huan al., 2025 Yue et al., 2025a), our experiments provide clearer and more compelling evidence demonstrating that RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions affect model performance. Moreover, we find that the intrinsic success rate of individual patterns remains relatively stable. These experimental insights inspire us to develop mathematical framework to theoretically understand the RI. training process.
Based on our empirical findings, we further develop theoretical framework that conceptualizes reasoning as two-stage question-reason-anssee process g - a: (I) reasoning pattern selection based on the question, i.e., (rlgg), and (2) answer generation based on the chosen pattern, i.e., x (alrr, g). Using this framework, we theoretically analyze the training dynamics of RL in LLMs with two typical reward types: verifiable reward (RLVR) (Guo et al.] 2025 Shao et al. 2024) and the model's 's internal feedback (RLIF) (Zhao et al., 2025b Agarwal et al. 2025). For RLVR, we show that it can converge to the reasoning pattern r" with the highest success rate, which precisely matches our empirical observations. Moreover, we characterize two distinct convergence regimes: models with strong initial reasoning quality demonstrate rapid convergence to optimal patterns, while weaker models face entanglement-stage optimization challenges. For RLIF, we provide an explanation of why RL with internal rewards can improve model performance, and we also show that RLIF may ultimately converge to state with worse performance than the base model, aligning with our empirical findings. Additional experiments validate our theoretical analysis.
The main contributions of this paper are highlighted follows:
We conduct systematic reasoning-patten pattern-leeel and token-level analysis across the RL training process. Through examining next token prediction ranking shifts, combined with LLM-based and rule-based reasoning pattern analysis, we provide clearer and more compelling evidence for understanding RL training dynamics compared with previous works (Huan et al.) [225]] Yue et al., 2025a). Our experiments demonstrate that RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions to affect model performance, while the intrinsic success rate of individual patterns remains relatively stable during training.
We develop a formal two-stage mathematical framework that models reasoning as + r + a cetestioonreasoo-anssee and theoretically analyze training dynamics for two typical RL-based approaches. For RLVR, we prove convergence to the reasoning pattern with the highest success rate and characterize two distinct regimes: rapid convergence for strong base models versus optimization challenges during entanglement stages for weaker models. For RLIF, we provide theoretical justification for the performance improvements at early training stages, and explain why such methods may ultimately converge to worse performance than the base model.
We validate our theoretical analysis through additional case studies. Our framework provides practical insights for understanding and improving RL-based LLM post-training, bridging the gap between empirical observations and theoretical understanding of RL training dynamics in LLMs.
2 RELATED WORKS
Reinforcement Learning for LLMs Reinforcement learning has demonstrated remarkable success in enhancing large language models (L.MMs), particularly in aligning models with human preferences (Ouyang et al.) 2022, Zhu et al. 2023) and in solving complex mathematical and programming tasks (Shao et al. 2024 Jaech et al. 2024 Lambert et al. 2024). A central component of RL is the reward model, which traditionally relies on human-annotated datasets (Rafailov et al. 2023 Ouyang et 2022 Achiam et al., 2023) with extensive training. Recently, the paradigm has shifted toward leveraging more easily obtainable and verifiable rewards, such in reinforcement learning with verifiable rewards (RLVR) (Shao et al. 2024 Yang et al. 2025 Guo et al. 2025) and reinforcement learning with internal feedback (RLIF) (Zhao et al. 2025b Agarwal et al. 2025). Moreover, the
2

---

Published as a conference paper at ICLR 2026
success of RL has spurred the development of new RL algorithms, including GRPO (Sao et al. 2024). DAPO (Yu et al. 2025), (Chen et al. 2025). and VAPO (Yue et al. 2025b).
Theoretical Analyses and Mechanism Interpretation of RL for LLMs. While cxisting theoretical
analyses have provided valuable insights into the general mechanics large language models (Azar
et a 2024 Aminian al. 2025 Chen and Zou, 2024 Chen et al. 2024 Li al.
2025 Zhang
et a 2026 2025a). J the remarkable success of RL enhancing LLMs has triggered a dedicated wave
of research focused specifically its theoretical foundations and mechanism interpretation. Initial efforts in this area mainly concentrated on high-level RL dynamics, such as reward design [Scheid
all [2024); Huang et (2025); Xu al. (2024) and the development of novel training algorithms Xiong et al. (202); ; Das et al. (2024) : Ji al. (2024). As RL paradigms shift towards simpler reward
structures like RLVR and RLIF, researchers have sought to understand the fundamental reasons
behind its effectiveness from various perspectives. [Shao et al. (2025) revealed that spurious rewards enhance reasoning by unlocking latent skills from pretraining, while Agarwal et al.) (2025) explained
performance improvements through entropy minimization. Gandhi et al. (0255) identified specific cognitive behaviors, such as backtracking, that contribute to improved reasoning capabilities. Further insights came Zrom et al. (2025a), who demonstrated that RL fine-tuning amplifies pre-train pre-trained behaviors, leading to convergence towards dominant output formats, a phenomenon further analyzed by Wang al. (2025b); Cui al. (2025b) through the lens of entropy collapse. Unlike previous work, our study presents an analysis of RL training dynamics from the perspective of reasoning pattern selection, supported by a mathematical framework with theoretical analysis.
PRELIMINARIES
Reinforcement Learning for LLMs Let ma be language model with parameters 0, which serves as the policy to be optimized. Given an input question x = (æo, T1,-..,,,, ,.....,,,,,. the policy To generates an answer y (yo, V1 ....... Vm.) The optimization objective can be formulated as:
ØRL 0) = E ( [r (x. y)] 3L (T y x) Feee x)]
(3.1)
where Tref is the base reference policy, and /3 is hyperparameter that controls the KL divergence to prevent excessive deviation between To and Tref-
The reward function T ra (ar, y) can be implemented in various formats, such trained reward model Ouyang et (2022) or a rule-based scoring function (Shao et al. (2024). In this paper, we focus on verifiable reward (RLVR) (Guo et al., 20255 Shao et al. 2024 and the model's internal feedback (RLIF) (Zhao et al. 2025b Agarwal et al. 2025).
In RLVR, the reward function (a,, y) directly evaluates whether the answer y matches the correct answer to question æ. A typical reward function is defined as:
if y = the ground truth of x;
To (z, y)
= 0 otherwise.
(3.2)
Unlike RLVR, RLIF (Zaa et al. 2025b Agarwal et al. 2025 leverages rewards based solely on intrinsic
consider the RLIF reward as the negative average KI. divergence between a uniform distribution over the vocabulary V and the model's next-token distribution
signals. Following Zhao et al. (2025b)
V
(x. 4) = DKL (U = yyl
log([V] Te(-kæ, yee)). (3.3)
M
In this paper, we aim to understand the training dynamics of both RLIF and RLVR through empirical and theoretical analysis.
4 EXPERIMENTAL EXPLORATION FOR RL TRAINING
To understand the RL training process, conduct systematic experiments and analysis across RL training with different rewards. Our experiments begin with a high-level overview of the training procedure, revealing that RLVR yields steady improvements, whereas RI.F can be unstable and even
3

---

Published as a conference paper at ICLR 2026
100%
RLVR vs. RUF
10274
RIVR
RLIF
102%
RLVR
Katinn Pal
0.2%
40%
Accuracy
Proportion
60%
6.0%
40%
Accuraci
40%
6.%
40%
0%
20
40
50
80
100
294
0
20
40
50
80
100
40
60
50
100
0%
Step
Step
Step
(a) Training Accuracy
(b) Pattern Distribution Dynamics
(a) The training procedures using RLVR and RLIF, showing the performance the MATH dataset. A key finding is that RLVR provides stable gains, while RLIF causes model performance to initially increase and then decrease. We conduct multiple rounds of experiments for each training paradigm by setting the random seed. (b) The
Figure I:
level comparison. The solid line represents the proportion of a certain pattern in the responses among all patterns (left vertical axis). The dotted line represents the accuracy corresponding to the pattern (right vertical axis). During RLVR, the model gradually adopts patterns with higher accuracy and reduces the use of patterns with lower accuracy, while RLIF exhibits unstable training dynamics. For ease of observation, we sort and name the different patterns from high low according to their corresponding accuracy.
degrade performance. To dissect these dynamics, we perform reasoning pattern level analysis, which shows that RLVR's 's success stems from its tendency to adopt reasoning patterns with higher success rates, while RLIF fails to specialize in these patterns. Finally, token-level analysis investigates the underlying mechanism for these reasoning pattern dynamics, revealing that they are driven by changes in the probability ranks of surprisingly small fraction of tokens. Compared with previous works (Huan et al.] 2025 (Yue et al.] (025a)) our experiments provide clearer and more compelling evidence of these RL training dynamics, demonstrating that RL primarily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distributions to affect model performance.
4.1 TRAINING PROCEDURE OVERVIEW FOR RLVR AND RLIF
We first analyze the complete training procedures for both RLVR and RLIF. For controlled compari- son, we select Qwen2.5-3B (Yang et al. (2244) as the base model and train it on the MATH dataset (Hendrycks al. 2021 keeping all other settings identical for both algorithms.
As illustrated in Figure Ta the two methods exhibit markedly different performance trajectories. RLVR leads to stable training process, where model performance continuously improves and gradually converges throughout training. In contrast, while RLIF initially improves performance in the early stages, continued training could lead a performance drop, sometimes resulting in a model that is worse than the original base model. These results motivate us to conduct more systematic, fine-grained empirical analysis to understand the underlying RL training dynamics.
4.2 REASONING- LEVEL ANALYSIS
To understand the performance disparities observed between RLVR and RLIF, we conduct a fine- grained analysis the reasoning- responses from the base model and employ GPT-4o (Hurst et al., 2024) to group them into distinct categories based keywords and logical structure, also generating description for each pattern. We then use these classifications to analyze the reasoning pattern distribution and their corresponding success rates throughout the RL training procedure.
cansoning-pattemn pattern level. To define and extract these patterns, first collect
Specifically, we analyze the dynamics of reasoning patterns and their accuracy during both RLVR and RLIF training. Figure [b illustrates our analysis on responses from Number Theory tasks from the MATH dataset (Hendrycks et al., 202T), with additional results for varying models and datasets provided in Appendix Our analysis reveals three key findings:
RLVR- trained models consistently shift towards adopting reasoning patterns with higher success rates, explaining why the model's 's overall accuracy steadily improves with RLVR.
4

---

Published as a conference paper at ICLR 2026
The reasoning pattern distribution for RLIF exhibits unstable training dynamics and fails to specialize in more effective patterns.
The success rate of any individual reasoning pattern remains stable throughout the training process for both methods.
This analysis indicates that the performance difference between RLVR and RLIF stems from their distinct reasoning pattern dynamics. We therefore conduct further experiments to understand the underlying mechanisms driving these changes during RI. training.
4.3 TOKEN-LEVEL ANALYSIS
To understand the mechanisms behind the reasoning pattern dynamics observed during RL training, inspired by Huan et al. (2025); Yue et al. (2025a), we further conduct a token-level analysis. We first sample responses from the base model for questions from the GSM8K (Cobbe al. 2021). MATH (Hendrycks et al. 2021). and AIME (Codeforces) datasets. Then, for each token position these responses, examine the corresponding token ranks in both the base and the RL-enhanced models. Here, a token's is rank refers to the position of its probability among all vocabulary tokens, given the preceding context. Our analysis, summarized in Table T reveals that the ranks at individual token positions remain largely stable, with fewer than 10% of the positions in each response experiencing rank shift in most cases. This finding indicates that RL selectively modifies the probabilities at sparse set of critical decision points while leaving the majority of the reasoning process unchanged.
Our experiments provide clean and compelling evidence for RL training dynamics: RL primar- ily optimizes a sparse subset of critical tokens, thereby reshaping reasoning pattern distribu- tions to affect model performance. Moreover the intrinsic success rate of individual patterns remains relatively stable during training.
Table 1: The Ranking Change Ratio after RL.
Task
Method
Step20
Step40
Step60
Step80
Step100
REVR RLIF
5.2%
6.0%
7.1%
7.3%
GSMBK
6.8%
8.0%
8.9%
9.5%
10.1%
6.1%
6.5%
6.6%
These empirical findings not only offer deeper understanding of RL training dynamics but also provide insights for building mathematical framework to conduct further theoretical analy- sis.
MATH
RLIF
6.1%
7.0%
7.7%
8.3%
8.8%
RLVR
5.1%
5.7%
5.9%
6.1%
6.3%
ATMF24
RLIF
5.6%
6.5%
7.4%
8.2%
8.6%
5 THEORETICAL CHARACTERIZATION OF RL TRAINING DYNAMICS
Beyond empirical observations, we take a further step toward theoretically understanding the RL training process. In this section, first build theoretical framework based experimental insights. We abstract the model's reasoning procedure as a two-step process: first selecting a reasoning pattern, then performing answer deduction, where RL specializes optimizing the first part. Based on this framework, we provide a theoretical analysis for our observations of RL with different rewards. For RLVR, analyze its training dynamics and examine two special cases with distinct optimization behaviors. For RLIF, we analyze why RL with internal rewards can improve model performance while also explaining why such methods may ultimately result in models with worse performance than the base model.
5.1 A THEORETICAL FRAMEWORK FOR REASONING MODELS
A bstrrct Reasoning Process. We formalize the reasoning process as follows: given a question
q, the model (1) selects reasoning pattern from candidate patterns R (r1,72,...) and (2) generates a final answera E A = {a, a..... accordingly. The model selects reasoning pattern r; with probability p(r:lg) and each reasoning pattern has distinct success rate. Within this framework,
5

---

Published as a conference paper at ICLR 2026
can reformulate the RLIF optimization objective (Eq.
3.3 follows
log (IA) mo(aylu:r)) )
rel re((r,a).g): ==
log ([R) (T(r):))))
(5.1)
R
A
PIR
asEA
Policy Parameterization. Let v {q) URU.U denote the vocabulary set and y = (r, a),, represent the set of output sequences. Given question q, the language model produces distribution over output sequences y E y autoregressively:
by II
(general policy)
Tee (y x:)
To (0lll
softmax(Je y<1)), 1=1
(5.2)
where fe - - R is a function parameterized by 0, and the model predicts the I-th to- ken based on the previous context y<t- The next token follows the distribution softmax(z),, exp(z.)/ Evey exp(z.)) for x fo(æ, 9<1).
Due to the complexity of practical models, establishing optimization guarantees for understanding RL training dynamics has proven very challenging (Agarwal et al. 2021 Li et al., 2021). Following
previous works (Razin al. 2023 Mei et al. 2020; Cui al. 2025b Zhang et al. 2025b). we
consider a tabular policy parameterization, which can be viewed as special case of Eq. 5.2 where
each output is assigned its own trainable logit for the corresponding last token, i.e., for 0 E R x
(tabular policy)
( x. y=1) = Te (3r 11-1) softmax (0.....))))
(5.3)
where 0 an E RIV is the column of 0 corresponding to y:-1-
Optimization for RL. For our analysis of training dynamics, consider the tabular policy from 5.3 with the optimization objective (+) ) in 3.1 We analyze the policy gradient in the small learning rate limit using gradient flow:
(() = (O()( 120
(5.4)
where 0(t) represents the parameters of the policy TT(e) at training time f, initialized with ( = 0.ef-
Our experiments revealed that success rates for individual reasoning patterns often remain stable during training. We attribute this to the model's 's architectural constraints: optimizing the mapping from questions to reasoning patterns is substantially easier than optimizing the path from reasoning patterns to final answers. To formalize this observation, we introduce the following assumption:
Assumption 5.1. The success rate for each reasoning pattern vs R. to provide the correct answer r" defined as Te (a" lq, r.), remains constant during training.
In the remaining part of this section, we adopt Assumption 5. and use p*(r) = Te(a" q, r) to denote the fixed success rate for the given pattern We first derive the optimal policy for the RL objective (.1 under the general autoregressive policy (Eq(.2)). We then analyze RLVR and RLIF training dynamics using the tabular policy (Eq. 5.3)
5.2 THE OPTIMAL POLICY FOR RL
Our empirical results demonstrate that RLVR improves model performance steadily through incen- tivizing reasoning patterns with higher success rate, while the RLIF demonstrates an unstable improvement for the model, here, we first provide a theoretical explanation for the optimal policy for RLVR and RLIF optimizing objective:
Proposition 5.2. Suppose maximize the RL objective 3.1) using general autoregressive policy (Equ 5.2) and Assumption holds. Then, the optimal policy satisfiex:
1 R(r) (
Topt (rlg)
exp
Te.e (rlq) for all r E R.
(5.5)
For simplicity, we omit the normalizing coefficient 1//p/ from Eq. framework.
3.3 which can be treated as constant
1/2

---

Published as a conference paper at ICLR 2026
where = E.ER exp ( R(r) To (rlq) the normalizing coefficient, and R(r) denotes the
reasoning path reward. Specifically:
For RLVR, R(r) equals the success rate of reasoning pattern v, i.e, PRIVR(F) - pr()..
for RLIF, R(r) equals to the confidence for the reasoning pattern v for final answer distribution, i.e., RRLIE (r) A CaeA (og((e((ag)) (a).
Proposition 5.2 characterizes the optimal solutions for RLVR and RLIE. Assuming Topt resides within the general autoregressive policy parameterized space (which always holds for both LLMs and the tabular policies discussed later), the RLVR- ELVR-optimized model increases the probability of a reasoning pattern r when the its probability to deduct the correct answer p" (r) is high enough such that exp( 3pp" (r))/Z >> 1. In contrast, for RLIF, the formulation of RRLIF (r) just consider the confidence of the final answer distribution, fails to distinguish between correct answer a" and incorrect alternatives. Consequently, while optimal policy under RLVR could consistently leading to better performance, the optimal policy for RLIF does not guarantee improved accuracy over the base
model.
Moreover, since 3 is typically small (e.g.. 0.001) in practice Rafailov et al. 2023): Bai (0022) Zeng et al. (2025). ). 3 + 0, we have:
Topt (rlg) lim 8-5 Z exp ( Te.. (rlg) (1 r= max, R(r)
(5.6)
otherwise
In this limit, the policy converges to deterministic strategy that always selects the reasoning pattern r with highest R(r), regardless of how the initialized reference model chooses reasoning patterns. However, due the non-convexity of the optimization landscape, the dynamics of how RLVR reliably finds high-reward patterns and why RLIF may exhibit early performance improvements remain unclear. To address this, we next analyze the training dynamics of both methods under tabular policy (Eq. 5.3) providing further insight into the RI. training process.
5.3 TRAINING DYNAMIC ANALYSIS FOR RLVR
Our experiments demonstrate that RLVR improvements heavily depend on the capacity of the base model. Here, we focus on the probability of the optimal reasoning patlern, i.e., the pattern r" with the highest success rate for reaching the correct answer a' r" arg max,, p" (r). We reveal two distinct regimes in the training dynamics of RLVR via gradient flow. In the first regime, the probability of the optimal reasoning pattern (r (q) steadily increases until convergence to 1. In the second regime, the model initially experiences
entanglement stage, where suboptimal reasoning pattern r' * r" hinder the optimizing process for the optimal reasoning pattern. After this entanglement stage, the model eventually transitions to the dynamics of the first regime and converges to the optimal reasoning pattern r.
Theorem 5.3 (Regime 1: Sufficient Condition for Efficient Convergence). Consider the RLVR (Eg
3.2 with = 0. optimizing objective Eq 3.1. using a tabular policy (.3 with Assumption 5.1 holds. Let r* be the optimal reasoning pattern, if the overall accuracy of the initialized model mo... defined as ACCo.. = ErER To (rlg)p" (r), satisfies:
(Regime I)
ACCo.. > >p" (r) for all T Rr
(5.7)
then for any 0, there exists Ti (()) such that for t >> Tr have 1 - ToCo <
In Theorem 5.3 we consider case where the base reference model sufficiently strong such that its overall accuracy exceeds the success rate of all non-optimal reasoning patterns (Eqq 5.7). In this case, RLVR can efficiently guide the model to select the optimal reasoning pattern rate (1/e), achieving high overall accuracy (close to the success rate of the optimal reasoning pattern).
However, practical experience shows that in some scenarios, RLVR optimization can be challenging Zeng al. (2025); Xie et al.] (2025), which typically occurs when the base reference model is less powerful. We then consider the second regime, where the model initially experiences an entanglement stage, and the suboptimal reasoning pattern with the second-hig success rate, defined as r' = arg max, max,, r,r p" (r), slows down the optimization process for the optimal reasoning pattern:
7

---

Published as a conference paper at ICLR 2026
Theorem 5.4 (Regime 2: Slow Convergence for optimal reasoning pattern). Consider the RLVR (Eq
3.2 with 3 == 0 for optimizing objective 3.1 using a tabular policy (Eq with Assumption 5.1
holds. re and r' be the optimal and second optimal reasoning patterns, if the overall accuracy of the initialized model satisfies:
(Regime 2)
p'(r') ,C V (r) all "e/(r",'',
(5.8)
then there exists:
To.. (rlq)
To 2 - 2 (r'lq) ( (Cin))
Mmiee) 200 where "irtet
(5.9)
rER/(r) To (r (g)
with constants C1, C2 depending on the success rates of reasoning patterns, such that for we can guarantee the model transform from regime 2 (Eq 5.8) to regime / (Eq 5.7 ie., ACCa()) > p" (r),, E R, + r" fort 2 To-
In Theorem 5.4 we consider special regime where only the success rates of the optimal and suboptimal reasoning patterns exceed the overall accuracy. We consider this case for ease of theoretical analysis and believe it can be extended to more general settings where at least two reasoning patterns are allowed to achieve higher success rates than the average, albeit with more complicated theoretical analysis.
Additionally, the critical insight of Theorem 5.4 is that RLVR may require To time steps to ensure that the overall success rate exceeds the success rate of the suboptimal reasoning pattern, i.c., reach the regime discussed in Theorem 5.3 where the selection probability of the optimal reasoning pattern is sufficiently large. While Theorem (5.3 shows that the convergence time Ti is polynomial in 1/c, the time step To in Theorem 5.4 may grow super-exponentially with respect to "Yn.. -te the ratio between the total success rate through R/ R/(r') and the success rate through r". Clearly, when the base model assigns a very small selection probability to the optimal reasoning pattern, we may have a very large Touer which leads to prohibitively large To- Consequently, it can take extremely long training period for the model to select the optimal reasoning pattern with a reasonably large probability, which we refer to as the entanglement stage.
We provide additional experiments and case studies to further illustrate these two regimes in See 6
5.4 THEORETICAL EXPLANATION FOR RLIF
In Proposition 5.2 we reveal that RLIF, which only considers the confidence of the final answer distribution, fails to distinguish between the correct answer a' and incorrect alternatives. This indicates that RLIF may eventually lead to performance degradation compared to the base model, which aligns well with our empirical observations. However, previous studies have demonstrated that RLIF can improve model performance without external rewards Agarwal et al. (2025); Zhao et al. (2025b). Here, we provide theoretical explanation for why RLIF can improve the performance of tral-trained LLM at the initial training stage.
First, we consider a well-trained LLM satisfying the following assumption: Assumption 5.5. For the base model, the correct answer a" has the highest probabiiity across all possible answers, i.e., arg max A max
ETER Tbase ( qq T Tbase (rlq) = a".
Since majority voting Wang al. (2022) has proven to be an effective method for improving model performance, and such technique tends to choose the answer with the highest probability /wu et al. (2025), such improvements indicate a well-trained LLMs tend assign higher probability to the correct answer a" than to incorrect answers, Assumption 5.5 is likely to hold for modern LLMss in many problems. Under this constraint, we consider the case that |A| 2, the success rate for each reasoning path follows a uniform distribution and the reasoning pattern selection is high-entropy In this case, we analyze the overall accuracy dynamics at O, yielding the following theorem:
Theorem 5.6 (RLIF Increases Overall Accuracy at Initialization). Consider the RLIF 5.1) with 3=0 for optimizing objective Eq 3. using labular policy 5.3) with Assumption 5./ [and 5.5
hold, Te... (r|g)= = [ for all r E R, |A| = 2 and success rate for each reasoning path follows uniform distribution p (r) U U0,, 1]. Then when [R] + +oc, the following holds:
2 Previous work (Wang al. 2025al reveals that RI. primarily optimizes high-entropy tokens. Here, we
consider the highest entropy case: To (rlg) 1/|R| for all R.
8

---

Published as a conference paper at ICLR 2026
Case Study for RLVR Regime
Case Study for RIVR Regime 2: Dynamics
ACCz))) 2 p p"(r)) (rt)
Patter Dynanics
p'(r) ACC p'(r)
20
0.
0.25
0.70
0.05
ryloptimat)
tyloptimee
0.70
G
30
12-10"
0.0
Steps
Steps
1ur Thakiig Se Nin
RLnTuiiing Srecleris brly 23?
maliitin engiciig best whti
5.0
0.2
ADC 22..022%
4:51.39%
5x10
15x10
0.60
10
16
Mteps
steps
[4]
Figure 2: Case studies for RLVR (upper) and training simulations for RLIF (bottom). The upper panels demonstrate two distinct training regimes RLVR: rapid convergence (left) and entanglement phase (right). The bottom panels show the probabilities of model convergence to different states (left), and the probability of initial performance gains (right) for RLIF across varying (R].
The accuracy derivative t = positive with probability i.e. P P(gACCom) t= With probability p 0.5, that arg max, RRLIF = arg min, p (r).
> 0) = 1.
The first result in Theorem 5.6 shows that overall accuracy increases at initialization, offering a theoret- ical explanation for RLIFs early performance gains. However, as training progresses, Proposition 5.2 implies that when / = O, the policy converges deterministically to the reasoning pattern r that maxi- mizes (R)I The second result in (.6 that, with probability 0.5, this maximizing pattern coincides with the one that minimizes the success rate: arg max, ARLIF (F) = arg min, p"(( (r). In other words, there is 50% chance the model converges to the least accurate reasoning path. Consequently, while RLIF initially improves model performance (as the derivative positive), continued training may cause the model to converge to states that favor reasoning paths with very low accuracy, ultimately resulting in performance worse than the base model.
accuracy is
CASE STUDIES AND NUMERICAL SIMULATIONS
To further validate and interpret our theoretical findings, we present case studies and training simula- tions to illustrate our theorems. For RLVR, we demonstrate in Theorems 5.3 and 5.4 as illustrated in Fig. 2 (upper). For RLIF, we conduct multiple training simulations where each reasoning path's 's success rate follows a uniform distribution, validating our analysis of RLIF's behavior during initial training steps and at convergence, as shown in Fig. (bottom).
distinct training regimes discussed
6.1 CASE STUDIES FOR RLVR
In Section we identified distinct regimes in RLVR's training dynamics. the first regime (Eq 5.7). the probability of selecting the optimal reasoning pattern To (r*(q) monotonically increases until convergence. In the second regime (Eq(588. the model undergoes an initial entanglement phase and requires a substantially longer training period before transitioning to the dynamics of the first regime and converging to the optimal reasoning pattern r*. We examine both regimes:
As demonstrated in Fig. 2 (upper left), when the reference model satisfies the conditions in 5.7 where the overall accuracy exceeds the success rates of all non-optimal reasoning patterns, the model converges rapidly to the optimal reasoning pattern.
the second case, corresponding to Theorem 5.4 when suboptimal reasoning pattern achieves higher success rate than the initial overall accuracy, the model first experiences entanglement phase. A transition period To (defined Theorem 5.4) must elapse before the model enters the rapid optimization phase characteristic of case 1. Fig. (upper right) illustrates scenario where

---

Published as a conference paper at ICLR 2026
Yow (defined in Eq 5.9) is large (Yr.u 6, (0.25 + 0.05)/0.05 6). In such cases, the transition time To, which scales with Mas becomes prohibitively long, significantly delaying convergence to the optimal reasoning pattern. The pattern dynamics in Fig. clearly demonstrate such entanglement stage, align well with our theoretical result.
6.2 NUMERICAL SIMULATIONS FOR RLIF
In Section 5.4, we demonstrated that under Assumption 5.3 and the base model distribution specified in Theorem 5.6 [R]] oo, the accuracy derivative at is positive with probability 1, while there exists a 50% probability of convergence to the least accurate reasoning path. Our simulations validate these findings. Fig. 2 (bottom left) presents RLIF training simulations across varying (R]. The results confirm that the probability of convergence to an improved state (ACC t) approximately equals the probability of convergence to a degraded state (ACC 4), validating the second result in Theorem 5.6 Additionally, our examination of initial training step performance, shown in Fig. 2 (bottom right), demonstrates that for large [R], the probability of initial performance improvement approaches 1, supporting first result in Theorem 5.6
7 CONCLUSIONS AND LIMITATIONS
This work analyzes reinforcement learning dynamics in LLMs through both empirical investigations and theoretical frameworks. We develop mathematical analyses for two representative reward mechanisms (RLVR and RLIF) and validate our findings through case studies and simulations. Our analysis has certain limitations that warrant further investigation: the interpretability of L.LM- identified reasoning patterns needs additional validation, our theoretical framework could be extended handle more complex real-world reasoning scenarios, and the current analysis could be generalized beyond specific base model assumptions.
8 ACKNOWIEDDGENTT
We would like thank the anonymous reviewers and area chairs for their helpful comments. We acknowledge the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A1515012444, and the central fund from HKU IDS.
REFERENCES
MAA. 2023, American mathematics competitions. https::/arttofprottt www.slllmmsoliingggoggooo com/ wiki/ index.nxx.. www.i.pwwpw.......mmmmpppppp hwtppphppmmaaaaaaa http///rrr...aaaaaaaeaaaaaaaee Competitions 2023. Online.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
Shivam Agarwal, Zimin Zhang. Lifan Yuan, Jiawei Han, and Hao Peng. The unrcasonable effective- ness of entropy minimization in I1m reasoning, arXiv preprint 25Xiv:2505.15134 2025.
Gholamali Aminian, Amir R Asadi, Idan Shenfeld, and Youssef Mroueh. Theoretical analysis of kl-regularized rlhf with multiple reference models. arXiv preprint 2502:2502.002000 2025.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo. Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 2022.
10

---

Published as a conference paper at ICLR 2026
Xingwu Chen and Difan Zou. What can transformer learn with varying depth? case studies sequence learning tasks. In International Conference on Machine Learning, pages 1002-0001. PMLR. 2024.
Xingwu Chen, Lei Zhao, and Difan Zou. How transformers utilize multi- attention in in-context learning? a case study on sparse linear regression. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Sitao Cheng, Tianle Li, Xuhan Huang, Xunjian Yin, and Difan Zou. Differentiable evolutionary reinforcement learning. arXiv preprint arXiv:2512.13399 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
MAA Codeforces, American invitational mathematics examination-aine 2024, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng. Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models, May 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025b.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
Amir Dembo. Large deviations techniques and applications. Springer, 2009.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self eef-viipo reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.013077 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint 2..I:::5011.244:: 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadhench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. /4008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint 1rXiv:2103.03874 2021.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning, July 2025.
Jiawei Huang, Bingcong Li, Christoph Dann, and Niao He. Can rlhf be more efficient with imperfect reward models? a policy coverage perspective. In ICLR 2025 Workshop on Bidirectional Human-Al Alignment, 2025.
11

---

Published as a conference paper at ICLR 2026
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries. arXiv preprint arXiv:2402.09401, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T,"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. 2024.
Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time converge. In Conference Learning Theory, pages 3107-3110. PMLR, 2021.
Tianle Li, Chenyang Zhang. Xingwu Chen, Yuan Cao, and Difan Zou. On the robustmess of transformers against context hijacking for linear classification. arXiv preprint arXiv:2502.15609 2025.
Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International conference on machine learning, pages 6820-6829. PMLR, 2020.
Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang. Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. Sparse but critical: A token-level analysis of distributional shifts in RLVR fine-tuning of LLMs. In The Fourteenth International Conference on Learning Representations, 2026.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023.
Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. arXiv preprint arXiv:2503.15477 2025.
Antoine Scheid, Etienne Boursier, Alain Durmus, Michael Jordan, Pierre Ménard, Eric Moulines, and Michal Valko. Optimal design for reward modeling in rlhf. arXiv preprint arXiv:2410. 17055, 2024.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolci Du Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. /0947, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. / 19256, 2024.
Qwen Team. Qwq-32b: 32b: Embracing the power of reinforcement learning, March 2025. URL https://gwent www.:/ewenm.........m.mm ghu.. ier..io/bb. to/blog////ooooooooooooooooooooo qwg-32b.
12

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng. Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang. and Junyang Lin. Beyond the 80/20 Rule: High- Eigh-Entropy Minority Tokens Drive Effective Reinforcement Learning for L.M Reasoning, June 2025a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effecctiee reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939 2025b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arxiv:2203.11177: 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optime inference for LLM problem-solving. In The Thirteenth International Conference on Learning Representations, 2025. URL JJHHn
Set//orun/IGGNNGKD
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning, arXiv preprint arXiv:2502.147000 2025.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint. In International Conference on Machine Learning, pages 54715-54754. PMLR, 2024.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. In International Conference on Machine Learning, pages 54983 54998.. PMLR, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15113 15115, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint Xii::2505.00388. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv: 13837, 2025a
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.0511
2025b.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Chenyang Zhang, Xuran Meng, and Yuan Cao. Transformer learns optimal variable selection in group- sparse classification. In The Thirteenth International Conference on Learning Representations, 2025a.
13

---

Published as a conference paper at ICLR 2026
Chenyang Zhang, Qingyue Zhao, Quanquan Gu, and Yuan Cao. Transformers trained via gradient descent can provably learn a class of teacher models. In The Fourteenth International Confer- ence Learning Representations, 2026. URL httpt: httpss:///ppeeeeee...ee.. net forenreereww.nne/ooruauaa formm?i 1k UkiRIdgoIF,
Yanzhi Zhang, Zhaoxi Zhang, Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No Free Lunch: Rethinking Internal Feedback for LLM Reasoning.
June 2025b.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: post-training amplifies behaviors learned pretraining. arXiv preprint arXiv:2504.07912. 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sengey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025b.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. In International Conference on Machine Learning. pages 43037 43067. PMLR, 2023.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder--V: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. /931, 2024.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In this paper, we utilized LLMs to perform grammatical corrections and to generate code for data visualization and model training.
B ADDITIONAL EXPERIMENTS
B. PATTERN DISTRIBUTION DYNAMICS ON RL-ENHANCED QWEN2.5-3B
In section 4.2
we sample Number Theory tasks from MATH to analyze the dynamics the reasoning
patterns. We also sample Geometry tasks for dynamic analysis (Figure 3) During RLVR, the model gradually adopts the reasoning patterns with higher accuracy, whereas the patterns selection for RLIF paper.
stable. This result is consistent with the empirical and theoretical results presented in the main
RLVR
RLIF
100%
100%
100%
100%
Pattern 1 Ratio
Patter Acc
Pattern Ratio
Fattem Acc
Pattern 2 Ratio
Fattem Acc
Pattern Ratio
Fattem Acc
Proportion
Accuracy
Proportion
Accuracy
60%
60%
60%
60%
40%
40%
40%
40%
20%
0%
0%
0%
0%
Step
Step
Figure 3: Dynamics of the patterns distribution and corresponding accuracy of RL-enhanced models on Geometry tasks. In RLVR, the model tends to choose the pattern with the highest accuracy (Pattern 1), but RLIF is not stable.
B.2 O
VS. 2.1...2.--B-SIIPLELL...
In order to verify whether our findings are applicable to different models, we conduct supplementary experiments validate the experimental insights and theoretical conclusions (Figure For these experiments, we use 2.5cn-2.5-B-Innttreee with Qwen-2.5-7B- 2.5-7B-SimpleRL--Z0 (Zeng et al. 2025). open-source variant enhanced with RLVR for mathematical reasoning. Our evaluation spans diverse mathematical domains, including number theory, geometry, algebra, calculus, counting and probability, using challenging problems sampled from the MATH dataset (Hendrycks al. (021I) We further extend analysis include complex mathematical problems from AMC23 (2023. 2023). The results consistently support our earlier findings: RLVR enhancement shows increase the frequency of high-accuracy reasoning patterns, while less effective patterns appear less frequently. Detailed task-specific analyses are provided in Appendix C.3
YYang et al. 2024) as the base model and compare it
Through reasoning pattern analysis across various tasks, we observe that patterns with the highest success rates consistently become more prevalent after RLVR enhancement, reinforeing our findings from Section 4 and Section 5 Notably, in tasks such as Algebra and AMC23-19 19 in Figure 4 we observe consistent success rates across individual reasoning patterns, which not only aligns with our previous observations but also provides empirical support for Assumption 5.1 in our theoretical analysis.
B.3
vs. QwQ-32B
To further verify theory, conduct experiments larger model, QwQ-32B Team 2025). This model is based on Qwen-2.5-322 2.5-32B-Instrrue (Yang et al. (0024) and greatly enhances the reasoning ability through RLVR. We test the reasoning patterns shift of these models four tasks, i.e.
15

---

Published as a conference paper at ICLR 2026
Number Theory
Geometry
100%
100%
100%
100%
Pottn Dist we RIVR
Pottn Dist we RLVR
80%
80%
80%
Pottn
80%
Pattn Acc wio RLVR
Success Rate
Paltn Acc weo RLVR
Success Rate
Proportion
Acc
Proportion
Pottn
60%
60%
60%
60%
40%
40%
40%
40%
2.0%
0%
0%
0%
0%
3
Pattern
Pattern Calculus
Algebra
100%
100%
100%
100%
Paltin Dist w/o RLV
Pailn Dist wio RLVR
Pattin
80%
Pellin Ace w/o RLVR
80%
Success Rate
80%
Pailtn Ace weo RLVR
80%
Success Rate
Proportion
Proportion
Pettn
60%
60%
60%
60%
40%
40%
40%
40%
20%
20%
20%
20%
0%
0%
0%
2
5
Pattern
Pattern AMC23-19
Probability
100%
100%
100%
100%
Pattn Dist w/o RLVB
Pattn Dist wie RLVR
RLVR
Pettin W/
80%
Aoati Acc W/o RIVR
80%
Success Rate
80%
Pattn Ace weo RIVR
80%
Success Rate
Proportion
AcC RIVR
Proportion
Pattn RIVa
60%
40%
40%
40%
40%
20%
20%
20%
20%
2
3
Pattern
Pattern
Figure 4: Evaluation results for reasoning pattern and corresponding success rate of model with/with- out RLVR enhancement for varying additional tasks, which are aligned well with our experimental insights and theoretical conclusions. The bar (Pattern Dist) represents the proportion of certain pattern in all patterns. The dot (Pattern Acc) represents the accuracy or success rate corresponding to the pattern.
16

---

Published as a conference paper at ICLR 2026
number theory and geometry tasks from the MATH dataset (Hendrycks et al. 2021), task from AIME 2024 (Codeforces), and task from OlympiadBench (He et al. 2024).
We that most of the reasoning patterns of QwQ-32B correspond the most accurate patterns of Qwen-2.5-32B, which is aligned well with our theoretical results. We do not give the accuracy of the API of the models for testing and not extract the CoT data allow the
QwQ because base model to continue to generate answers. In this way, it is impossible to obtain the accuracy of the reasoning patterns (rather than the accuracy of the model itself). For details, see the "Accuracy Analysis" in Appendix
Number Theory
Geometry
100%
Patth Dist Owen2.5
100%
100%
Paltn Dist -wen2.5
100%
Pattn Dist- DwQ
QwQ
80%
Qwen2.5
80%
Success Rate
80%
Pettn Acc Qwen2.5
80%
Success Rate
Proportion
Proportion
60%
60%
60%
60%
40%
40%
40%
20%
20%
20%
20%
0%
0%
0%
0%
2
Pattern
Pattern
AIME24-1
Olympiad Bench-1631
100%
100%
100%
100%
Pottn Dist Qwen2.5
Pottn Dist: Qwen2.5
QwQ
Pettn QwQ
80%
Pottn Acc Qwen2.5
80%
Success Rate
80%
Pottn Acc
Qwen2.5
80%
Success Rate
Proportion
Proportion
60%
40%
40%
40%
40%
20%
20%
20%
20%
2 Pattem
3
Pattem
Figure 5: Evaluation results larger models, Qwen-2.5-32B and QwQ-32B. QwQ's reasoning patterns converge to the most accurate pattern of Qwen-2.5, which demonstrates the applicability of our theory to larger-scale models.
C EXPERIMENTAL DETAILS
C.1 EXPERIMENTAL SETUP
We first select Qwen2.5-3B as the base model and train it on the MATH dataset, using both RLVR and RLIF. We adopt the verl (Sheng et al. 2024) framework and ensure that all parameter settings verl framework. For RLIF, we only modified the reward calculation method. By setting different random seeds, obtain three different training curves for each method. All our settings exactly the same as those in the Verl framework using the GRPO algorithm. The temperature is set to I during training. The learning rate is set to le-6. The train batch_ size is ppo_ mini batch size is set to 256. In group sampling, the number of rollouts for a single prompt is to 5. We disable top-k and top-p I.
Entire Training Procedures:
identical the example the MATH dataset provided by the
1024 while the
We use Qwen2.5-3B and the above RL-enhanced models for this experiment. For GSM8K and MATH, we randomly sample 400 questions from their test sets respectively. For AIME24, its full dataset. We first sample the corresponding answer from the base model for each question. Next, we concatenate this answer to the question itself and input it into the RI-enhanced model
Token-level Analysis:
prompt. We use the interface in the OpenAl library calculate the
17

---

Published as a conference paper at ICLR 2026
probability of each token in the prompt and its ranking among the probabilities of all possible tokens at the current position. By comparing whether the ranking of each token has changed before and after RL, we can calculate the proportion of tokens whose ranking has changed.
Reasoning-patters Level Analysis: For this task, we conduct two types of experiments. We first study the dynamics of the patterns distribution during training using previously trained models, namely Qwen2.5-3B and RL-enhanced Qwen2.5-3B. We evaluate every 20 steps. To examine the applicability of our conclusions on different base models and larger models, we test the open-source QwQ-32B, and their corresponding base models. Therefore, we can only examine the changes in the patterns distribution of model outputs before and after RL, but cannot examine the dynamics of the patterns distribution during the training process.
model Qwen-2.5-7BB 1w..------mmlle---222
C.2 REASONING PATTERNS ANALYSIS PIPELINE
Model Inference: We first randomly sample questions of different task types (such as Geometry, Algebra, etc.) and different difficulty levels (Level 4, Level 5) from datasets including MATH. For each question, our model samples 1024 answers in the patterns distribution dynamic task, and samples answers ranging from 64 to 1024 in the task that only considers the distribution change before and after RL, depending on the task category. These responses will be used for subsequent pattern extraction and classification.
Reasoning Patterns Extraction: We sample subset of responses of the base model and use GPT-4o (Hurst et al. 2024) API to summarize patterns categories from these samples. Full prompt we used for reasoning patterns extraction is given in Appendix [E.1 To ensure the accuracy of patterns extraction, we set the temperature of GPT-4o's API to 0.
Responses Classification: We again employ GPT-4o to classify all responses according to the identified pattern categories, also setting the temperature to O. Full prompt used for responses classification is given in Appendix E.2
Accuracy Analysis: In order to only consider the impact of the patterns distribution on accuracy. we construct new prompts by concatenating the original prompt with partial responses from the base model, assuming these partial responses sufficiently represent specific reasoning patterns. This ensures that the reasoning process is fixed, which makes it easier for us to determine whether the change in model accuracy depends only on the change in the distribution of the reasoning pattern, rather than the change in the reasoning process itself. Specifically, the final answer from the base model's response, add the sentence before it to the end of the question, and input it predict the answer. We will examine the accuracy of the predicted answer as the accuracy of the reasoning pattern.
remove the sentence containing
prompt the RL-enhanced model. The RL-enhanced model will continue to
C.3 DETAILS FOR EACH TASK
Below we detail the experimental procedures for each task, including the task descriptions and examples of reasoning patterns (Tabel 2}. Example task prompts are provided in Appendix E.3
This task presents models with problems involving coin distribution across multiple bags. Initially, bags contain equal numbers of coins. After receiving additional coins and redistributing them equally, the total must exceed a specified value while maintaining equal distribution. Models must determine the minimum initial coin count per bag. We derived this task from a level MATH dataset problem (id: test/number (tanbbrrtteor/105555 (Hendrycks al., 2021). creating 32 variants by adjusting parameters like bag count and coin totals as our evaluation dataset. We employ one-shot prompting with a simple, unrelated example to guide answer formatting using "boxed" notation without influencing reasoning approaches, For evaluation, we first choose 20 questions from our evaluation dataset with 4 responses each for reasoning pattern extraction. For each question sample 64 responses. We then categorize all responses by the extracted reasoning patterns, allowing us to compare changes in the distribution of models reasoning patterns before and after RI.
Number Theory:
18

---

Published as a conference paper at ICLR 2026
Task
Reasoning Patterns Examples
Common Elements Key Words
Modular Congruence with
Uses modular arithmetic to express divisibility conditions.
Coefficient Simplification
Number Theory
Incqualit--Drivel Search
Often involves substituting back
for Minimal Solution
compute the total coins.
Systematic Inequality
Explicitly lists and solves
Application
each of the three inequalities.
Geometry
Verification of Scalene
Adding extra layer of validation verifies the third side length.
Condition
Iterative Floor Division
Uses floor division to
with Leftover Tracking.
compute new cans step.
Algebra
Recursive Recycling with
Combines leftovers with newly
Aggregated Leftovers.
produced cans before recyeling.
Direct Simplification and
Assumes inputs fall within the principal range.
Principal Value Matching
Calculus
Interval Analysis
Splits the whole domain into different intervals.
with Case Splitting
Direct Probability Setup
Explicitly calculates combinations for total and favorable outcomes.
and Quadratic Solution
Counting and Prohahility
Early Simplification and
Early cross- woos-multiplicationn
cross-Muldppiicationn
climinate denominators.
Prime Factorization
Counting Digits
and Simplification
in Large Numbers.
AMC23-Q19
Rewriting (8") (2t9)
Final step of counting digits: (3 (from 243) + 15 (zeros) = 18).
(15%) as (3 5-))
Direct Equation Setup
Straightforward and relies
and Elimination Alternative Equation
on algehraic manipulation.
AIME24-Q1
Expressing variables
Formulation and Solving
terms of others carly on.
Direct Calculation
Relying direct computation and empirical verification.
and Empirical Testing
OlympiadBench-QII
Factorization Based Reasoning
Using algebraic factorization to argue certain terms yn-
Table 2: Reasoning Patterns Examples for Varying Tasks.
19

---

Published as a conference paper at ICLR 2026
Geometry: For the geometry task, will give the model the lengths of two sides of scalene triangle and ask how many different integer centimeters the length of the third side can be. This task comes from a level 4 geometry problem in MATH (id: test/geometry/10460 follows the same pipeline as in Number Theory: we construct synthetie dataset with 32 questions, for
Our evaluation
sample 20 questions and select 4 responses each from the base model (Qwen2.5-7b-Inss, reasoning pattern extraction, and then we compare the responses for model with and without RLVR enhancement with all questions our evaluation dataset with 64 responses each for pattern analysis.
The algebra task we use a can recycling problem. We first have certain number of old cans. It is pre-defined that old cans can be recycled into a new can. The question is how many cans can be produced in the end. The difficulty of the problem is that in each step of the iterative calculation, there may be extra cans that cannot be divided evenly. These cans may eventually be combined together for further recycling. The template for this task comes from a level 4 algebra problem in MATH (id: test/algebra/2768), For all the following supplementary tasks starting from this task, we adopt the same settings as the previous tasks, including the data set size, number of samples, etc.
Algebra:
Calculus: Our calculus task is simple and straightforward. We will present trigonometric equation in single variable, the domain of that variable, and ask the model to determine the number of solutions to the equation. Although this problem is simple, can demonstrate the model's basic ability in calculus problems. Its prototype is level 4 pre-calculus problem in MATH (id: test/precalculus/1 140).
Counting and Probability: We also study the performance of the model on the counting and probability task. There are white balls and black balls. We will randomly sample two balls from these balls and give the probability that one of the two balls drawn is black and the other is white. We also provide the number of balls of a certain color and hope that the model can calculate the minimum number of balls of another color. This is probability theory task, which comes from a level 4 counting and probability problem in MATH (id: test/counting and probability/79).
AMC23 Question: Here choose question 19 in AMC23 (2023, 2023), for reasoning pattern extraction, we randomly sample 48 responses, and then sample 1024 response for reasoning pattern analysis.
C.4 DETAILS FOR EXPERIMENTS ON QwQ-32B AND QWEN-2.5-322 2.5-32B
We conduct experiments on four tasks. Due to the limitations of using the API for testing, we randomly select question per task as input, sample 64 responses from each of the two models, and directly perform reasoning patterns extraction and classification from these answers. Specifically, for the two tasks of the MATH dataset, we choose question 1055 from the number theory task and question 1046 from the geometry task. For the remaining two tasks, we question problem from AIME24 and question 1631 from Olympiad Bench. Examples of patterns are shown in Table
D DEFERRED PROOFS
In this appendix, we provide proofs for our main theoretical results: Proposition Theorems 5,44and and Theorems 5.6
5.2 Theorems 5.3
D.1 PROOF OF PROPOSITION 5,2
Proof of Proposition 5.2 In this proof, we utilize the proof techniques in Rafailov et al. (203)), recall
that the optimization objective of RL
ΦRL
(r, g)] - kL (
(x)
mer [x)]
(D.1)
Under our framework, as state in section 5.1, we consider a policy conduct reasoning by first sample T; R. based To no (r:lg) and then provide the final answer E AA by no (a/r:) the reward dRu.
20

---

Published as a conference paper at ICLR 2026
can be written
(HL( (0) E (alla)
Σ
Tzei(a (
3DKL (TT (x) Trer(ay x)], (D.2)
LaEA
Let R(r) EaEd Trer(a r)r al. then
Te( Jzef (a/r) Tyer( q) Trel r))
@RL( (e R((r) -1n
Bo [((r) 31 (T(rlq)
Tort
(r(g)
(T(
--Bpxxa(-10)) In
In Z
2 exp
1/3 R(()
(Tee(rlq))
3L [x (rlg)
Tret (y
m>) + 3In
Where the third equation is by Assumption 5.1 and partition constant that ensures
ErER exp (HD*(r) To.e (rlq) is the
1 xt (r((q)) Z exp ( R(r) Te(rlq))
a valid probability distribution such that ΣrER x (rll) = 1. Since not a function r.
Therefore, maximizing the objective in Equation D.2 is equivalent to:
max Ea-D.y yoa (a)le) (<x y) 3DL [T(y] x)) Trat (y x)]
π
min 3I. JTT (rlq)
(ref(y)
æ)] 8 In Z.
By the properties of KL-divergence, we know that the optimal policy for the KL-constrained reward maximization objective satisfies:
1 ((rr) Te.
Topt Tope (rlg) = cxp
for all E R.
specifically, for RLVR, (RLVR(T) = pp, for RLIF, we have
[((r)
1
log (| (r(ll)) +
Σ
log (IA) To(agla.r))
R
A
PER
ayEA
1
log (To
log ([ (T(rr)))) + log [A|
R
FER
where the second term is the same for all r E [R], for RLIF, the optimal policy can written as
Topt (rlg)
exp R((rr) Te... (rlq)
exp ( To. (rlq)
Emer exp (B( Ter. (r:|g)
exp (R - c) Te (rlq)
ΣnER exp (R'(rs) c) To.at (r: (g)
So can write BRLIF(F) ('(r) - [ Eajes log (TO (a, 41, r)). This concludes the proof of the theorem.
21

---

Published as a conference paper at ICLR 2026
D.2 DYNAMICS OF TABULAR POLICY
Consider the RL objective (Eq 3.1 using tabular policy (Eq 5.3 with Assumption 5.1 holds, the gradient dynamics of can be computed as
d ((1) V (0(1))
dt
VE 1h (rlx)
Σ [ (rlg)Voocy(r)))) PER
Where (rlg) softmax(O:.qn.. Oggg E R is the column of 0 corresponding to q, and (r(g) KL refers to the reward with KL. divergence, so
(0)
Σ
Tk'K( ( (Vs)))max(0.q)::))))
Öt
TE
Σ
[r (r)lo) (- (r(o))))))))
rER
+ rro (rilg) (r(e)))
where the overall accuracy is defined as ACCo()) ErER Tonc (rlq)x"(r) For (re (r:lg), have
d
(r:lg))))))) (tr),((t)
Feon(rigg)
Σ
dt
(0,,,a(t)
Øt
D.3 PROOF OF THROREM 5.3
here first prove that ACC,() ACCoro) for t > 0 (Eq D.4 then establish lower bound for me(t) (r" (q) (Eq D.5), finally we derive the final bound for stated in Theorem 5.3.
To proof Theorem 5.3
Proof of Theorem 5.3 Setting 3 = 0, Eq D.3 becomes:
a (t) ( (p"(r.) ACCoo)
(D.3)
Øt
For (F:(g)))))) we derive:
d
Σ (To())(i(g)) (0r,.g())
dt Tee Tew (n:lg)
(tr,,g(t)
Ot
ER
Σ (ACCo(t) -p" (T;)) The (0
ER
+ (p" (r:) ACCo() (r.1g))
22

---

Published as a conference paper at ICLR 2026
and for the accuracy (CC
d
ACCo(1)
Σ (ri)
dt (r:1g)
=
dt
PER
Σ
p" (r.))) (ACCo()) - (r²)) (
"RyEER
( ACCo(o) (rila)))
NER
Σ ACCo(1) (ACCB(O) - p" (r,)) (1y
PE
Σ (p'(r:) - -ACCo()) (r:l9)p (r:)
RER
Tot (rilg) (CCCnn)))) 2 0.
(D.4)
As ACCo()) 2 0, so ACCon) > ACCa(o) holds for 0, for the optimal reasoning pattern r", let r' the suboptimal reasoning pattern with the second highest success rate, defined r' r p" (r), we have:
dt (ro)))))))))) = Σ (ACCo()) - (r;)) (ro))
PER
+ (p'(r") ACCo()) TēCH (r"1 g)
2 (ACCo() (r())) T0(1) (r" (r' (1)
+1"(rr)) ACC T Toce((1 (r"1g)
(v'(r') -ACCo() Tē (q) (1 Tores (r"1g)) 2 (p"(r=) p" (r')) TÔ (r" (q) (1 (r*))) 2 0.
(D.5)
As a ( 2 o. (r())))))) 2 (r(o) q) holds for 2 0, define C
(x" (r*)r) - -p (r')) To, (r"()):
d di or 2 C(1 (r=)))))))))*
This differential inequality yields:
(r")g) > 1- Ct + T- (r"
For any 0, there exists T 1 (1-1-1000 =O()), such that
(0()(r')g) > 1-c.
This concludes the proof of the theorem.
D.4 PROOF OF THEBREM 5.4
To proof Theorem 5.4 we first establish an upper bound for (r())(")lg) (Eq D.6). then we analyze
the ratio Po(t) (r:) (r")))
(r:lg) (Eq D.7and and prove that for > To. ACC,g) 2 (r').
23

---

Published as a conference paper at ICLR 2026
Proof of Theorem 5.4
Let r" and r' be the optimal and second optimal reasoning patterns. For
To(n) (r' ()):
di Tec)(r)))a)
= Σ (ACCo() - -p"(r)) (r))
TYR
+
- (ACCrr)) (T'
E (ACCo()) - (r;) (r)
(ER\(())
+ (1'())) Torn (r' q) (1 (r lq))
1 Σ
(r
E////²+
+ (r"(r') - p" (r') T (r l2)) T0 (1 -r(()(r)1)))
/ ()) (1 Toe) (r (q)) + (r (r'1g) (1 (r))))
x2 21 =e(e((""))))
This yields:
(r(e)(r'lg)) S1 2t + 1- (r'1g)
(D.6)
1
Next, define (r(e)(r.)) (r q)
r'}. (r then for all r, R/(r", ///""rr'h we have
dt (r.)) = (rein(""00 a mo( q) (rilg))))) a (r())()))))
d
To()) (r)
(rilq)
(e(c)(*1))) (q) ((P ACCow) To(o) Toce (rilg) - (r(a())))))
(a( (r*) ACCo:)) To()) (q) S
The last inequality is based on the condition for case and ACCom) > ACCo so p" (r:) ACCo()) holds for all r, E R/(r")
< }. 2 0. Then as at (r()))r)) - 0, we have:
d
1- I-TO() (r' (q)
- Te(o)
Toto (q)
EMTER/Ar (r:))
EbEER/Ir Poco)
While ACCa pp(())), define YoTet ErER/(r")
Ne (rla)
we have:
x0,0r (r))
d मi (r(o(rs) = - Poco(r:) (p"(r") -ACCo()) (e(o)(r" (q)
S - Po(o((r) (v" (r"))")- -r"()) 1 Te(e) (r'lq)
7xie
S (r(c)(r:)) p (r*) (r(c)(r:) ((())) - -p"(r""v pr(r")
Yorot
2t -- (r' ②)
Then we have:
गi (r)) < (r.)) p' - (r(")
1
(D.7)
Yaeot
21 Tovooo (r'
this differential inequality leads to:
(1 To,0) (r' (r'lq)) - -/(22))()
(r())):) < (1/ To(o)
+ 26) A/(21= Po(o) (r:).
(r'lg)) To(o)
24

---

Published as a conference paper at ICLR 2026
Where A p²(r") -p" (r'). Recall that:
ACCo(1) - -p"(r') Σ
(r')
Top
FER
= (r")
(p (1r)) - (r()))
E Pocoo(:)) (m:)))(((r))))
TERR/(M)
(p"rr")
(0()))-''011
E (o(o)(:)p(TT)
EERR/(M
Let C2 = 1/4.Cr (r')/ define:
To 2 2Te (r'lq) (IC Yaraa 220 -1)
then:
(r:)p)
(1 To(o) (r'lq) -3/(2
Σ MER/(M)
Po(o) (r:)p"(r)
(1/(1 To(o) (F' (q)) + 2T)) A/
(1- (r' (/2
(r() Σ
Pocu)(r:)
(ITme (1y) 2 /A)
A/ (2
1/(1 (r'lg)))))))
(rER((r))
A/ (27e P (r') Frree
Menaep (mi))
2//r0r/ /A)
A =A = p" (r²) p² (r').
Finally, for To we have:
ACCon - -'()') 2 pr())
(0 -2'r)) Σ Poco(r:))"(r))
MER/((r)
-p - -
This concludes the proof of the theorem.
D.5 PROOF OF THEOREM 5.6
Proof of Theorem
5.6
First, as we consider the case [A] = 2 and Th (rl4) = | Assumption
5.5
holds means
[ EreR (p" (r)) > 0.5. The derivative of overall accuracy can be written as
d ACCoin (r.) d ti (rio(())
PER
Σ P.ER
T (rilq) (p'(r.) - ACCa()
log(Te()) (a/r:, Coneco) a
25

---

Published as a conference paper at ICLR 2026
where Co -
EnER (r:))) Easa (ot) (a/r,,)))) q)). Here we assume A = fa", .a)
where a' is the incorrect answer, and To(o) (r:gg) = [
for all Ty R. Then:
d dt
ACC
Σ HE
1 R
pr(r:)
Σ
(T(1,))
R
PE
Σ
log( log(=o(o)(a)r: 9)) +
Σ
log( (oroo
[
a
TyER REA
(r(r:)
(ry)) EjER
log( (co) (()) --
Σ aEA
R
R
TER
As we consider the case
[R]
+00 and the success rate p" (r) i.i.d. samples from U10, 1]. (r"))) is concentrated
By the Central limit theorem, for large (R], the sample mean [R]
around the its expectation 0.5, the deviation is of order is (1/ [R], We define this deviation as 0.5 5.
5 (1/(RR) 1/2) > 0 such that Tee (P
According to the Gibbs conditioning principle (Dembo 2009) Corollary 7.3.5 and Theorem 7.3.8),
as [R| + oo, the empirical distribution of the p (r) conditioned on their mean [ ΣXER (p" (r)) = 0.5 + 5 converges to the density:
((x) = 1 + 128(x 0.5), [0, I]. As [R]
can approximate the summation over r with an integral over this limiting distribution fzx) The derivative becomes:
dt ACC
- (1 128(x 0.5)) (xx (x- 0.5 - 5) log (x(1 - x)) dx
2
0
3
Since delta > 0, the derivative positive. As R - 0o, this holds with probability 1, proving the first statement.
To prove the second statement, we first analyze the RLIF reward function. For an action space of size | = 2, the reward for a path proportional the negative ligliiitl of the policy:
RILIF (r)
log (To (To(a)lg.r) a,EA
log(p" (r) -p0
A
The function (p) = p(1-p)
- p) is maximized at p = 0.5. Therefore, maximizing the reward RRLIF(F) equivalent to choosing the path whose success rate p" (r) furthest from 0.5.
Let Pimin minreR (r) and Pmax maxreR (r). The path with the highest reward will be the one with success rate Piuin is further from 0.5 than Pmax is. This condition expressed as:
0.5 P'min > Pinax - 0.5 + Pinin + Pinax <
Thus, the event arg max, RRLIF (r) arg min, p" (r) equivalent the event Pini + Pinax 1.
We now calculate the probability of this event. The success rates p" (r) are drawn from the limiting distribution fIx + 128(x 0.5). From the theory of order statistics, the scaled minimum and maximum of = R| samples converge in distribution to independent exponential random variables. Let Y, Piii and Z, n (1 Pinax). As n oo:
Y, converges to Y Exp(A (yr), ). where the rate is the density at the lower bound: xy f(0) = 65.
* 7,, converges to 7 ~ Exp(A2) Az), where the rate is the density at the upper bound: Az f(1) 1 68.
=
The condition Pinia + Pinxx < I can be rewritten in terms of these asymptotic variables:
Yn +
Zn
<
Y < Zn
n
26

---

Published as a conference paper at ICLR 2026
The probability of this event is P(Y < Z). For two independent exponential variables, this is given by:
Ay
1 - 66
1 - 66
P(Y 2
36
A (1 60) (1 (1+66) 66)
2
Since 8 (22) in the limit as [R] + 00, ò 0. Therefore, the limiting probability is:
lim P(arg max (RLIF()) = arg min pr()) = lim +
36
5-+0
This concludes the proof of the second statement.
E
FULL PROMPTS
E.
FULL PROMPT FOR REASONING PATTERNS EXTRACTION
Full Prompt for Reasoning Patterns Extraction
You will analyze multiple Al reasoning processes, showing how different models solve problems.
Analysis Steps:
For each reasoning process, identify:
Key words and recurring phrases Logical structure the argument Problem-solving techniques used Step-by-step progression
Group similar reasoning processes into exactly patterns based on shared:
Vocabulary patterns (common terms and phrases)
Logical frameworks (how arguments Solution approaches (methods used to reach conclusions)
structured)
IMPORTANT! You need categorize general thinking patterns for problem solving, rather than categorizing types of problems themselves. In other words, each type of thinking pattern you give can be applied any problem, rather than being task-specific. What you need to classify is the way of thinking, not the way of solving problems. For example, you can classify them into:
1. derivation based on existing knowledge
derivation combined with verification
3. independent calculation, suitable for learners starting from the basics,
detailed manual derivation, suitable for learners who need to deeply understand each step
5. etc
Or you can classify them into: knowledge call type
2. derivation and verification type
etc.
Output Format:
Pattern 1:
Name and description
Defining characteristies
Example processes from input
Common elements
Pattern 2:
Name and description
Defining characteristics
Example processes from input
Common elements
There reasoning processes:
27

---

Published as a conference paper at ICLR 2026
Group similar reasoning processes into exactly patterns. Please provide classification of reasoning patterns, with each group defined by their common characteristics and examples from these processes.
E.2
FULL PROMPT FOR RESPONSES CLASSIHICATION
Full Prompt for Responses Classification
Please classify each reasoning process into one of the following five patterns. Output the classifications in JSON format.
{patterns_cateed _ }
Here the reasoning processes classify:
(extracted responses}
Please output your classification in this JSON format:
E.3
PROMPT EXAMPLES FOR EVALUATION TASKS
Prompt Example for Easy Countdown
<iim_starb>ssstf
You are helpful assistant. You first thinks about the reasoning process in the mind and then provides the user with the answer.<lim > <lim_start>>
user Using the numbers [1,2,3,7], create equation that equals 5. You only basic arithmetic addition (+) and subtraction (-), and each number from the list can be used at most once. Show your work <think> </think> tags. And return the final answer <answer> </answer> tags, for example, to get from [1,2,3,7], we have <answer> 2 + 3 = 5 </answer>; to get 13 from [1,2,377], we have <answer> 1 2 13 </answer>.<iin endl> atanttaasssstt
Let me solve this step by step. <think>
Prompt Example for Long Arithmetic
<lim_ _ttattt>systtem
You are helpful assistant. You first thinks about the reasoning process in the mind and then provides the user with the answer.clim_ent _lndd dim_startbbuse
Considering the experssion ((((((((((( 47) 92) 84) 5) 49) 82) 33) 99) 69). you need start from the innermost bracket and work your way out. You need to give the specific calculation process. You have to do a certain amount of calculations on each line, and print each calculation the format of a + = c or a c, for example "1 + 2 -llmm <lim_endd> <lim_ bhim_starbbassittaan
Let me solve this step by step.
<think>
Prompt Example for Number Theory
<lim_ tin_sttatbsysttmm
You are a helpful assistant.<lim endl>> hiim_starb>usee
You have four bags of copper coins. Each bag has the same number of copper coins. One day, you find bag of 23 coins. You decide to redistribute the number of coins you have
28

---

Published as a conference paper at ICLR 2026
so that all five bags you hold have the number of coins. You successfully manage redistribute all the coins, and you also note that you have more than 120 coins. What is the smallest number of coins you could have had before finding the bag of 23 coins?
Please reason step by step, and put your final answer within \boxed( | xxlim_eee _in__ndd>> <lim startD>asiistnn
Prompt Example for Geometry
<lim utn_sartbbssstem
You are helpful assistant<<lm_enddt _nnd>>
<lim dim_startbbuse
Two sides of scalene SYbigrriangleup ABCS measure $4$ centimeters and $7$ centimeters.
How many different whole centimeter lengths are possible for the third side?
Please reason step by step, and put your final answer within \boxed( |.llim _nddl> <lim starttassistan
Prompt Example for Algebra
<lim_ him_statb>ssstem
You are helpful assistant.<lim, <lin__ndd>>>__dd>>>> hii_startbbuser
Six aluminum cans can be recycled to make new can. How many new can eventually be made from 200 aluminum cans? (Remember that the first new cans that are made can then be recycled into even newer cans!)
Please reason step by step, and put your final answer within \boxed <.llim _ndd__ <lim_ stassssatbassi
Prompt Example for Calculus
<lim start
You are helpful assistant.clmm_ent
<lim_ stt_sttrbbsser
For how many values of $x$ in S[0,2\piJS is (- 4 x)
[-1 ((sin x)$?
Please reason step by step, and put your final answer within \boxed (1. 1.<lim_endll
<lim starsstarbaassit
Prompt Example for Counting and Probability
<lim, startbsysstmm
You are a helpful assistant -lim--iimmmmei- _nndd>>d>>>>
<lim_ sit_sttrtbbuerr
7 white balls and SkS black balls are placed into a bin. Two balls are drawn at random. The probability that one ball is white and the other is black $3 (66)S. Find the smallest possible value of SkS.
Please reason step by step, and put your final answer within \boxed() <<lim endl> <lim, starttaassssttn
Prompt Example for AMC23-Q19
<lim_ tii__strtb-system
You are helpful aasistant.<lim endl>
diim_starlbuuse
How many digits are in the base-ten representation of '$8"5 ledot 5*(10) Vedot 15^5$"?
Please reason step by step, and put your final answer within \boxed _eslim__eedd endl>
<lim, startbassistant
29

---

Published as a conference paper at ICLR 2026
Prompt Example for AIME24-Q1
<lim_ Siim_sartbsssster
You are helpful cisistant<lim_endd
<lim_starl>user
Every morning Aya goes for 595-kilometer-long walk and stops a coffee shop afterwards. When she walks at constant speed of $s$ kilometers per hour, the walk takes her 4 hours, including $t$ minutes spent in the coffee shop. When she walks Ss+2$ kilometers per hour, the walk takes her 2 hours and 24 minutes, including StS minutes spent in the coffee shop. Suppose Aya walks $s+Afaac (1) (2]$ kilometers per hour. Find the number of minutes the walk takes her, including the $t$ minutes spent in the coffee shop.
Please reason step by step, and put your final answer within \boxed J<liim endl>
atassssaaasss
Prompt Example for OlympiadBench-Q1631
<lim_start>>syne starrtsssssssee
You are helpful assistant<<lim_endl
sin_sttrtbbuerr
For positive integer $aS, define sequence of integers Sx Sx_[1], x_]],__[)),, (2), Vdots$ by letting Sx Sx_(1)=aS and [x_[n++1]=2 x_ [n]+11 for Sn Igeq 1$. (ttS__n)=2Mxx(n)) the largest possible $kS such that, for some positive integer SaS, the numbers Sy_(1), Mdots,
Determine
y_(kj$ are all prime.
Please reason step by step, and put your final answer within (ooxe())<<imm_e
<lim_ atsattttssssstt
30


Reviews:
[{"review_id": "4EJGTX4fJU", "reviewer": "Reviewer_xtTG", "summary": "This paper provides new insights into the reinforcement learning (RL) training dynamics of large language models (LLMs) through a combination of empirical analysis and theoretical modeling. The authors focus on two reward types: verifiable rewards (RLVR) and internal feedback (RLIF). They find that RL primarily optimizes a sparse subset of “critical” tokens, which reshapes the distribution of reasoning patterns. For RLVR the authors show two scenarios, one were optimization converges efficiently and another where weaker base models experience slow convergence. For RLIF the authors show that it is initally improving performance but can degrade performance with continued training.", "questions": "- **Q1:** For generating base model responses in the token-level analysis, was the sampling temperature set to 0?\n- **Q2:** Do the authors expect qualitatively different dynamics for different RLVR variants (e.g., GRPO, PPO, or CISPO), or would the theoretical conclusions remain consistent across these variants?\n- **Q3:** Could the authors elaborate on Assumption 5.1? In practice, the success rate of individual reasoning patterns may change during training. How realistic is it to assume this remains constant, and how might deviations affect the theoretical results?\n- **Q4:** How sensitive are the observed “sparse token optimization” effects to the choice of base model or training dataset?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Provides an explanation for the stability of RLVR and the instability of RLIF\n- Relevant topic: Understanding the behavior of RL training in the context of LLMs\n- Multiple datasets and model scales are used which strengthens the generality of the findings.\n- Introduces an interesting and interpretable analysis framework at both the token and reasoning-pattern levels", "weaknesses": "- The paper would benefit from a more comprehensive description of the experimental setup for the different training/evaluation runs. Important implementation details (e.g. temperature settings, sampling strategies, and other hyper parameters) are missing making it difficult to fully reproduce or interpret the reported results.\n- It remains unclear which exact implementation was used for RLVR training. For example, if vanilla GRPO was used, information about rollout numbers, optimization parameters, and possible comparisons to more recent methods (e.g., CISPO[1] or Dr.GRPO[2]) would help contextualize the empirical findings and assess their robustness.\n\n1: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention  \n2: Understanding R1-Zero-Like Training: A Critical Perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JrPZNIXC1y", "reviewer": "Reviewer_MU4Q", "summary": "The paper analyzes how RL post-training changes LLM reasoning, showing empirically that RLVR promotes higher-success reasoning patterns while RLIF remains unstable. A two-stage q → r → a model and tabular gradient-flow analysis explain these behaviors, and controlled experiments support the overall conclusions.", "questions": "1. How is pattern-level accuracy evaluated? A single response may contain multiple reasoning patterns—how do you attribute correctness to individual patterns in such mixed instances?\n2. How should the proposed training-dynamics framework inform practical RLVR training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper effectively combines empirical findings with a clean theoretical analysis, offering useful insights for understanding LLM reasoning behavior.", "weaknesses": "The experimental scope is narrow: all evaluations are conducted on math reasoning tasks and within the Qwen2.5 model family. Prior work has shown RLIF to produce gains on non-math general domains, so broader experiments across model families and tasks are necessary to substantiate the generality of the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2fh564ejPT", "reviewer": "Reviewer_BMWN", "summary": "This work first conducts analysis on reasoning-pattern and token levels. They claim that there are patterns obtaining stable accuracy, and distribution changes are only limited to a few tokens. Then, basing on the findings with the patterns, they theoretically analyze the training dynamics on RLVR and RLIF. In the experiment section, they conduct simulation on the RLVR and RLIF basing on their assumptions. The empirical results are consistent with theoretical claims.", "questions": "1, Besides the simulation, could you verify your findings on a real dataset (e.g., MATH)?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1, In section 4.2, this work identifies patterns in LLM mathematical reasoning. This could provide inspire further explorations.\n\n2, In section 5, basing on the assumptions, theoretical results are derived for RL training dynamics. Furthermore, these claims are supported by simulations in the experiment section.", "weaknesses": "1, My biggest concern is with the assumptions. Specifically, \n- Assumption 5.1 assumes that success rate for each pattern remains the same. But this assumption is only demonstrated on one dataset; and there is not discussion on why this assumption is reasonable;\n- Assumption 5.5 assumes that correct answer $a*$ has the highest probability across all possible answers, for all answers. I do not think this claim is well supported by Wang et al. (2022). \n- In equation (5.3), the LLM is simplified as a one-step policy, where the distribution of $y_l$ is only determined by its previous token $y_{l-1}$. Can you explain whether this is an reasonable simplification?\n\n2, Claims about the token distribution is not convincing; and is not connected to other parts of the manuscript. First, Table 1 is hard to interpret. Can you elaborate on how to understand this table? Second, in line 220, it claims that less than 10% of tokens experienced shift. But considering the vocabulary size and token distribution (a few tokens occupies significant probability mass), 10% rank shift could be significant. Can you also report the probability mass shift?\n\n3, The conclusion in line 203 actually depends on what patterns are. Could you explain what these patterns are? Will the pattern for a prompt persist or change?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)