ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning
Authors: Ruiyang Zhou
Shuozhe Li
Amy Zhang
Liu Leqi
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Self-improvement
Guided exploration
Reasoning
Reinforcement Learning with Verifiable Reward
Bootstrapping
Primary area: deep_learning

Abstract: Self-improvement via RL often fails on complex reasoning tasks because GRPO-style post-training methods rely on the model’s initial ability to generate positive samples. Without guided exploration, these approaches merely reinforce what the model already knows (distribution-sharpening) rather than enabling the model to solve problems where it initially generates no correct solutions.
To unlock reasoning ability in such settings, the model must explore new reasoning trajectories beyond its current output distribution. Such exploration requires access to sufficiently good positive samples to guide the learning. 

While expert demonstrations seem like a natural solution, we find that they are often ineffective in RL post-training. Instead, we identify two key properties of effective positive samples: they should (1) be likely under the current policy, and (2) increase the model’s likelihood of predicting the correct answer. Based on these insights, we propose \textbf{Self-Explanation Policy Optimization (ExPO)}—a simple and modular framework that generates such samples by conditioning on the ground-truth answer. ExPO enables efficient exploration and guides the model to produce reasoning trajectories more aligned with its policy than expert-written CoTs, while ensuring higher quality than its own (incorrect) samples. Experiments show that ExPO improves both learning efficiency and final performance on reasoning benchmarks, surpassing expert-demonstration-based methods in challenging settings such as MATH level-5, where the model initially struggles the most.

ExPO: Unlocking Hard Reasoning with
Self-Explnntioo--Guieed Reinforcement Learning
Ruiyang Zhou University of Texas at Austin ruiyang. zhou@utexas edu
Shuozhe Li
Amy Zhang
University of Texas at Austin shuozhe. 1i@utexas...... edu
University of Texas at Austin
Liu Leqi
University of Texas at Austin leqiliuButexas..
Abstract
Self-improvement via RL often fails on complex reasoning tasks because GRPO- style post-training methods rely the model's initial ability to generate positive samples. Without guided exploration, these approaches merely reinforce what the model already knows (distribution-sharpenien to solve problems where it initially generates no correct solutions. To unlock reasoning ability in such settings, the model must explore new reasoning trajec- tories beyond its current output distribution. Such exploration requires access to sufficiently good positive samples guide the learning.
rather than enabling the model
While expert demonstrations seem like a natural solution, we find that they are often ineffective in RL post-training. Instead, we identify two key properties of effective positive samples: they should (1) be likely under the current policy, and (2) increase the model's likelihood of predicting the correct answer. Based these insights, we propose Self- Seff-Exllantton Policy Optimization (ExPO)- a simple and modular framework that generates such samples by conditioning the ground-truth answer. It can be integrated with popular RL training methods like GRPO and DPO. ExPO enables efficient exploration and guides the model produce reasoning trajectories more aligned with its policy than expert-written CoTSs, while ensuring higher quality than its own (incorrect) samples. Experiments show that ExPO improves both learning efficiency and final performance on reasoning benchmarks, surpassing expert-demonsttration -basdd methods in challenging settings such MATH level-5, where the model initially struggles the most. Code available in attps://github com Htce/Humaiinn http//HaaannaaabBBeaee onnoaa/ wrw.:oonnnnnoo_ooollaaaaiiiioeea
Introduction
The development of reinforcement learning (RL)-style post-training methods has been a pivotal factor in the advances of large language models complex reasoning tasks. RL training optimizes model outputs using reward signals derived from human preferences, model comparisons, or automated verifiers [16] 25 27
24]. .. RL-style post-training encompasses a broad family of algorithms, including
reward-maximizzig reinforcement learning (c.g., GRPO) [20] [] contrastive preference optimization
methods 23
38 39 and supervised fine-tuning on expert demonstrations or high-quality self-
generated samples g. Despite differences in algorithmic formulation, all these methods share
central mechanism: increasing the likelihood of generating preferred/positive responses and reducing that of dispreferred/negativ ones.
Leading Contributors (Equal Contribution).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GRPO is similar to or not effective time scaling passgiNN large N).
beboz manning
F(oarrect)
GRPO
EsPO
37%
Direct Cul: (uabs few the ditliculty incresso)
Self-Eoglanatian [earret, driailed, lediiribatin) diide 585,
pepiiie Thes allos
Direct Oall: (weeng sice step 1:2 segittooia X thackee Aile).
Diveet Clrl: jarnng enee stpp 1/2
iestitooo A Tharites Wipt.
p
pooa) 120
MATH-Iewa
Molium Difficulty Problem
GRPO does not work for hard rensening problems.
before trainty
Picarrvet)
F(wreas))
GRPO
ExPO
Expl-aapanttioo fearreet, detailed, la distr Butioo)) se and enelie hee and not seim. Now, won
No effective pasitive ir siiinng sample
Birecr Cal: (wrnng strp 2.8) 4-
Direct Carl: (wrang since strp 2/5
X0%
Taal sio
Ex caFo
CRRO
Rooe
Taess
afee tranryy
pumep
MATII
Jevel
Hligh Difficulty Problem
Figure 1: Illustration of the problem and our proposed solution ExPO. On the left, models (base model is Dwen2.5-3B-Instiee
performance on the MATH dataset highlights the issue with GRPO-style methods: they primarily strengthen the model's existing capabilities rather than enabling new ones. On the right, we present the positive and negative samples of both GRPO and our proposed ExPO method for MATH level-4 (top) and level-5 (bottom). ExPO is more effective than GRPO in guiding the model to learn for hard reasoning tasks.
While negative samples are typically abundant during RL post-training, positive samples are scarce, especially in hard reasoning tasks. Thus, key challenge remains unresolved: how to obtain effective positive training samples in settings where the model's initial success rate is low? In practice, positive samples are either drawn from expert demonstrations, which are costly to obtain for complex domains (e.g., MATH level-5 [2),, or
by the model. Yet for hard problems, the model often fails to produce any correct answers initially, leading to pathological regime in GRPO-style training: the advantage term vanishes, the KL term dominates, and the model regresses- an effect sometimes described as "unlearning"
Existing work attempts to circumvent this by discarding training examples with all incorrect re- sponses (37 335. but this merely sidesteps the core issue. Fundamentally, GRPO-style methods assume high-quality samples are likely to be sampled under the model's current policy, but this assumption breaks down in high-difficulty regimes (Figure D.. In other words, these methods exhibit distribution-sharpeninng bias: they sharpen the model's 's output distribution around high-probabille correct responses, but struggle to guide learning on tasks where correct answers are unlikely under the current model. This limitation reflects broader issue noted in recent work [40] current RL- style post-training often reinforces existing capabilities rather than fostering fundamentally stronger reasoning abilities beyond those of the base model.
This motivates a fundamental question:
How can we effectively guide learning for challenging reasoning tasks in RL-style post-training, when positive samples scaree?
Rather than focusing on suppressing negative samples that offer little benefit when the model lacks any useful priors, we argue that progress hinges on identifying and synthesizing effective positive samples. This paper therefore investigates two core questions:
1. How can we obtain positive samples when the base model's initial success rate is low?
2. What properties must such samples possess to provide strong learning signals in RL-style post-training?
Our Approach: ExPO. To answer these questions, we begin by identifying two essential properties of ideal positive training samples:
2

---

The sample should have high probability under the current policy to guide learning effectively.
In-distribution:
Positive learning signal: The sample should increase the likelihood of the correct answer compared to the model's current CoT.
These criteria inform proposed method, ExPO SelffExxllaation Policy Optimization), modular framework for generating and integrating positive samples via self-explanation. Specifically, ExPO conditions the model on both the question and the final answer to generate plausible reasoning chains, which are then used as positive samples for RL-style training.
Our key insight is that conditioning on the ground-truth answer helps the model produce in-distribution reasoning traces that are more aligned with its current policy than expert-written CoTs, while providing better guidance than its own incorrect completions. These self-explanations are more elaborate and contain more correct steps than standard CoT's, serving as form of natural process supervision-as their relatively small deviation from standard CoTs helps the model identify where and how to adjust its reasoning. Because they are more likely under the current policy, these self- explanations guide the model to explore more effectively than expert CoTs Con'senabling the model to learn in settings previously dismissed as unlearnable.
ExPO is broadly applicable and can be instantiated in both contrastive (e.g., DPO) and reward-basee (e.g., GRP)) frameworks. Across both domains, we demonstrate that ExPO improves sample effi- ciency, accelerates learning, and significantly boosts performance-esppeeeee where prior methods fail. Notably, ExPO not only removes the need for expert Co''s, but also outperforms approaches that rely on them.
on difficult questions
2 Related work
RL methods for training LLM reasoning ability. Reinforcement Learning methods are increasingly showing promise in training for complex reasoning tasks. The simple but effective Direct Preference Optimization (DPO) algorithm [23 is used for reasoning and other human preference alignment tasks on pair of positive and negative generations. There are variants to better apply DPO on reasoning tasks, for example, the iterative DPO method [21 29] iteratively applies DPO to make it online; the self-training DPO method [31] iterates between SFT and DPO. On the other hand, the Group Relative Policy Optimization (GRPO) method [25] uses only the outcome verifier to train strong reasoning models. Without the need for process supervision signal (reasoning chain-of-thoon offthhughtt, it is possible to train on a large scale of tasks. We incorporate our method ExPO with DPO and GRPO to further improve their training performance on challenging reasoning tasks.
RL methods for self-improvement without expert-labeled COT. Given the challenge of acquiring expert-labeled CoT for complex reasoning tasks, self- self-improvement through reinforcement learning has become a key area of research. Current approaches include reinforcement learning from Al feedback [18 g 33 which uses powerful "judge" model to evaluate responses; methods with iterative self-correction and self-refinement, where model refines its own generation [2] and the use of self- seff-eenerteed training data (7). Our work contributes to this third category, where the model generates the reasoning steps as signals for its own training.
One important paper in this domain is STaR [T]. which proposes to use self- eeff-generated chain-of- thought during finetuning on reasoning tasks. In the absence of expert-labeled CoT, they prompt the model to regenerate rationales for incorrect predictions given the correct answer. While STaR is limited to SFT training and direct prompting, subsequent work proposes various methods for better self-training with RL. Their methods include using model rules, implicit signals, or external verifiers 13,15 28 42 However, while these approaches demonstrate practical success, they do not explain why StaR-style methods are so effective. Our paper proposes more general theoretical analysis on why -tyle methods can work and sometimes even surpass methods that use expert-labeled CoT, particularly under the RL framework.
3 Characteristics on the Ideal Positive Training Sample
We focus on settings where the model struggles to generate positive samples on its own-though it can still produce negative ones (c.g., incorrect self-generated responses), commonly seen in
3

---

hard reasoning tasks. In such cases, effective positive training samples are essential to guide the model's exploration and reduce the sample complexity of learning. To tackle this challenge, we first analyze what makes positive training sample effective in the context of policy/preference
optimization for reasoning tasks (Section 3.2 3.3 We then demonstrate how our proposed method for generating positive samples satisfies these ideal properties (Section F. Finally, we introduce ExPO Aself-Explanation -aase Policy Optimization), along with instantiations using different base *PO algorithms such DPO and GRPO (Section 5).
3.1 Problem Setup
When training large reasoning model, our (implicit) end goal is the following:
max "=
[r(g, c, a, a'),,
(I)
0
where D a distribution of question y and ground-truth answer pairs. The model my generates a pair of CoT e and answer a, and the verifiable reward is often given by whether the answer is correct, i... r(q, C, a, a²) = =iaa- = a²}. Though many algorithms (e.g.. "PO algorithms like GRPO, DPO, PPO, etc.) have been proposed to train reasoning models with various learning objectives, implicitly, our end goal is Eq. T as indicated by how we have evaluated the performance of the trained model To: evaluate them directly on their correctness reasoning tasks instead of evaluating them their learning objectives. For simplicity, we denote the objective in Eq. (1 as (()).
To understand the effect of data on the training process across different algorithms, we consider two complementary types of analysis: Policy improvement (66), where we use gradient-based analysis to examine how different samples contribute to optimizing the overarching objective Eq. (T); and Probability shifts [5), where we analyze directly how the policy Ta changes in response to different types of training samples (and that these probability shifts are closely tied to policy improvement).
Drawing insights from both analyses, we identify two key properties that characterize ideal positive training samples: (1) They must be in-distribution with respect to the current policy; and (2) They must be better than negative samples in achieving the task at hand (which we will provide a more precise definition for) and thus provide positive learning signal.
3.2 Property 1: In-distribution
Consider a generic gradient g(ğ. C, obtained with a sample (question, CoT, answer) pair (g. è, a). For now, we keep the gradient definition abstract: it can be derived from any policy/preference optimization algorithm, and the sample used for its computation can be obtained on- off-policy In this setting, after taking gradient ascen step with learning rate a > O, we have oc+1 = 0, + (7, E, a),, and approximate the change in the objective ① by first-order Taylor expansion:
AJ = (0 J (0.) aVJ (O) g(
The key to understand the effect of training data (7, 2, a) on the policy improvement AJ is thus to identify the alignment between the true gradient VoJ (0) and the sample gradient g(ā. c, a) used in training. Given the definition in (T). we have the true gradient to be
(0)
(DDaa) [Vg log (c alg)r(a, c, a, a,a"),,
(3)
As discussed in Shao et al. (25) Appendix A, given the training data (q. c, a). the gradient *PO algorithms often takes the form:
g C, a) = w Ve log (oC,
(4)
for some weighting w E R. The weighting depends on the relative quality of the data. When a sample is considered "positive"- higher objective value or is correct) the weighting is positive (w > 0). Conversely, samples that perform worse has the weighting to be w < of
is, it performs better than other samples in the batch (e.g.. it achieves
DDpendding on the exact 'PO algorithm (minimizing or maximizing the learning objective), it can be either a descent or ascent step.
'For certain algorithms (e.g.. PPO), the weighting is applied per-token, i.e. the weighting different for each of the summand in V log To (c, alg) = E, Ve log io (celg. det) + V. log ne (al (alg, c). For illustration simplicity. we keep fixed weighting for each token in a (response) trajectory, but the general properties we illustrate below apply settings with teken-dependent weighting.

---

By injecting the (0J(O) and g(q. e, a) into the objective3.2 and transform the expectation into summation, we argue that the magnitude of the policy improvement AJ (and the alignment between the true gradient VoJ and the actual used gradient g a, 7)) is dominated by the term (details are in Appendix)
wTe(c', a' 14 V log mo('', 3 (''''''''','''"""
(5)
Thus, the training data (g. C, a) will only be effective in improving the policy if To alq) is large, Le., the training sample (q, e, a) is "in-distribution" with respect policy Tg.
We now formalize the first ideal property of a positive training sample: being in-distribution under the current policy.
Property 1 (In-distribution). A sample (ž, a) is considered more in in-distribution relative to another sample (c, a) for given question if its probability under the current policy Ta is higher, i.e.,
Tc alq) > Ty (c, alq).
This is relative criterion, not an absolute one, reflecting the fact that in practice, when determining which samples in given batch are more in-distribution, we compare how likely each sample under the current policy. Samples with high probability and high reward can contribute more effectively to policy improvement. In practice, we want the training sample (q. é, a) to have Tt(C, alğ) sufficiently far from zero, and not too far from max,,a Tp alg). This ensures that the sample is in-distribution enough to meaningfully contribute to policy improvement.
3.3 Property 2: Positive Learning Signal
In the above, we already see the importance of having in-distribution data: gradients for samples (g, é, â) (with positive rewards) where (C, q >> O enable more effective policy improvement. In the following analysis, we examine which in-distribution samples should be given positive rewards and receive positive learning signals to increase their probability.
Consider a batch containing two training samples for the same question: {(g, CI, 01), (g, C2, a2)} When a1 is correct and (2 is not, it is clear that the correct one should be prefered,, i.e., (g, CI a1) > (g, 02; (2); and thus receive the positive reward to increase its probability. However, in settings where both 01 and a2 incorrect (as is often the case in model training for challenging reasoning tasks), which sample should be treated as the positive/preferred one to increase the probability?
We argue that the notion of preference between responses in general should be defined in terms of their ability to increase the likelihood of the correct answer a*. Specifically, we propose the following criterion:
Property 2 (Positive learning signal). A sample (g, C1 (1,01) 01) has a positive learning signal compared to (q, C22 a2), and thus should be preferred, i.e., (g, C1, 01) > (g,02, 02, a2); if and only if
To TT(a") lq, c1) > Th (a" q e2).
We note that this property is also defined in a relative sense: it allows us to compare two chain-of- thoughts, but does not assign absolute quality any individual sample. In the specific case of a batch containing two samples, if (g, C1. 01) carries positive learning signal relative to (g. 622 02). then training algorithms should assign the weighting w for the gradient g(q, C1 a1) ) to be positive (hence, positive gradient), and the weighting w for the gradient (4, C22 a2) to be negative (
Why should we increase the probability of (g. C1, a1) and decrease that of (g. C2, 02), if (g. C1 a1) possess a positive learning signal as defined in Property 2 For a fixed question and ground truth answer pair (q, a"), consider our objective defined in (I)
a(0: = Ena To(cl)) Ta(alg, e) Ma= = a²) E. mo(clg) mo(a")g,c)
(6)
This objective reflects the total probability mass assigned to generating the correct answer a through all reasoning paths c. Now, (a (g, a) > Ta(a" (g. 02). then increasing To vo(cilg) and decreasing Te(c2l4) would increase the overall objective [(I: 9. a²) ).") Therefore, adjusting the probabilities in this way directly improves the model's ability to generate the correct answer-justifying the preference defined in the property. In Appendix, we provide detailed analysis of how increasing or decreasing probabilities under a softmax policy influences others (based on their relative value), showing that choosing the wrong samples to increasc/decrease probability can lead to unintended effects.
5

---

Positive Training Sample Generation through Self-Explanation
As discussed, the ideal positive training samples should satisfy two properties: Property | (in- distribution), which ensures the samples to have relatively high probability under the current training policy no, and Property 2 (positive learning signal), which specifies that within batch, the sample whose Co makes the ground truth answer more probable- rellative to the CoT's of other samples- should receive positive gradient, i.e., its probability should be up- up-weighed.
In scenarios where self-generated responses (Co'T, answer pairs) perform poorly (e.g.. on challenging tasks where the model has not yet learned perform well), a surprisingly simple method can yield positive samples that satisfy the in-distribution and positive learning signal properties- a self-explanation conditioned on the correct answer:
è ~ To (cot [g. a²),
and use (q, a*) as a positive sample. This idea resembles the sampling strategy proposed in STAR [4] It is worth noting that the criteria for ideal positive training samples (Property 2 are more general than this specific sampling approach. As long as a sample is probable under the training policy and leads towards the correct answer more likely on average than the self-generated CoT, then the sample is ideal for providing positive learning signals. For example, beyond generating the sample conditioned the answer, one generate it by conditioning on partial expert CoT with additional hints.
4.1 Sell-Explanation: Natural Guidance for RL Post-training
We begin by showing example of self-explanation and provide some intuition why self- explanations are suitable positive training samples in RL post-training. Example shows a self- explanation for hard math problem. For this problem, conditioning the model on the correct answer enables it to successfully identify the pivotal step and guide the reasoning process correctly. Compared to the model's standard CoT, the self-explanation often contains more correct reasoning steps. Besides, the self-ex exfl-axaannntin''s 's phrasing is more consistent with the model's own language, offering clearer contrast to its incorrect responses.
Intuitively, self-explanation reduces the task difficulty by providing the correct answer as a known condition, shifting the challenge from open-ended problem-solving to generating conditional expla- nations. Because of this, the self-generate explanation has better quality while having a relatively small deviation from the incorrect standard COT. These properties seem well-suited for RL-style training, which, in our understanding, is more geared toward shaping the existing behavior of the base model rather than instilling entirely new knowledge. Therefore, self-explanation helps the model identify where and how to adjust its reasoning, functioning as an effective and natural form of process supervision.
Example I: Standard CoT vs self-explanation for question from MATH
Question: A Conditional probability question... Given that the side you see is red, what is the probability that the other side is red?
Standard CoT: /(Correct step I] Step I:
Self-Explanation /[Correct step 1] To solve this problem... /|Correct step 2] Now, we need to find... /[Explain the answer again] The answer is...
Count... X[Redundant step 2] Step 2: Deter-
mine... x[Wrong computation in step 3] Step 3: Calculate...
In fact, if the base model still fails to generate better reasoning traces with the guidance given by the correct answer, the problem is likely too difficult for the RL post-training approach. Compared to supervised fine-tuning (SFT), RL training is not as efficient at teaching new knowledge to the model. The objective of SFT is to maximize the probability of the target response. so that the knowledge in the response can be memorized. In contrast, the RL post-training objective is make the positive/chosen response more likely than the negative/rejected one, which provides weaker learning signal for memorizing complex, response-relee -eleevat new knowledge. Besides, the KL term in the RL post-training objective suggests that the trained model always stays close to the base model.
We arrived at this sampling strategy independently, and only later recognized its resemblance prior work.

---

Therefore, if the model is unable to articulate improved reasoning even when the correct answer known, it unlikely that reward driven learning alone will yield meaningful gains in reasoning capability- at least not in an efficient manner.
4.2 Comparing Seff-Explanation with Expert CoT and Standard CoT
To more formally understand why self-cxplanations 7 serve as ideal positive samples, we compare them- both empirically and analytically-wihh two other types of data: the expert CoT cæ and the original seff-generated CoT Self-generated explanations satisfy both ideal prop- erties (Table [T]. In contrast, expert CoT lacks Property 1 and, empiriclll, provides less effective guidance for model learning (Section[)) Compared to standard CoT, self-explanations provide positive learning signals. We elaborate on these findings below.
Medhod
dn-disributionn
Posiiive Leoraing Sianal
Ours belli selleoulaaaiio))
Papen Orl
Standard CoT
Table 1: Comparison of different train- ing data with respect to the ideal proper- ties. Our ExPO method verifies both.
First, regarding the in-distribution Property D we observe that, compared to the expert CoT CE. the self-generated explanation č is more likely to be generated under the current policy and thus more in-distribution for training. As shown in Figure 2, the self self-generated explanation has a significantly lower negative log-likelihode than the expert CoT. Surprisingly at least impirically-traiiingg with expert CoT can sometimes result in performance worse than the performance of models trained using self- self-generated explanations (as shown in Section o This observation aligns with our gradient-brr based analysis in Section 3.1. In general, the distribution of ē (-1g, a²) is close the original CoT distribution e ~ π (- [q), since the prompts used to generate these two distributions differ by only a small number of tokens, which describe the ground-truth answer a².
Second, for the positive learning signal Property 2 as illustrated in Example above-and consistently observed across the self-explanation data we have generated- self- generated explanations tend to be more detailed and con- tain more correct reasoning steps than the original self- generated COT. This observation is further supported em- pirically by prompting GPT-4o to evaluate the relative quality of é and GPPTT consistently rated é signifi- cantly higher in quality (Figure
20 scll-optanatiinn
ZZ wilf-oplanatiio direct CIT
Ixpert Cull
3.0
MATH GSMMKK MATIL-500
MITTII
GANHII MATIE-300
Figure 2: Left: Negative log- likelihood of the self-explanation č and expert CoT. Right: Winrate labelled by GPT-4o in terms the number of correct steps of the self-explanation and self-generated CoT Both on the test split of each dataset using Qwen2.5 IB-Instruct.
This intuition is also formalized in the following lemma:
Lemma 1. On average, the self- self-generated explanation is more likely lead the ground-truth truth answer than the original self generated chain-of ofain-of-thoughh thought. That is,
(e- (To(a" (g. 2)) 2 Bewnolchy) [To(a" lq, c)].
To summarize, the self-generated explanation è outperforms the expert CoT cÆ with respect to the in-distribution property (though it is weaker in terms of the positive learning signal property), while it surpasses the self-generated CoT in terms of the positive learning signal property but not the in-distribution one. Thus, our proposed positive sample-tte -he self-generated explanation a-achieves balance between these two desirable properties. In the following, describe how to leverage these generated positive samples within RL-type training for reasoning tasks.
5
ExPO and its instantiations on DPO and GRPO
Building on our method for generating in o--distribution positive samples (Section 4 and the theoretical foundations established in Section 3.1 we instantiate our approach within two widely used poli- cy/preference optimization frameworks: Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO). This leads to two practical algorithms ExP-DPO and ExP-GRPO that integrate explanation-ben based data augmentation into the learning pipeline.
7

---

5.1 ExP-DPO
We design and evaluate two variants of ExP-DPO: an offline and an iteratively online version. In the offline DPO setting, all explanations č and responses c- are generated by the initial policy at the beginning of the training, i.c. č₁ ~ Teer lq. at) andandd e ~ (-9()- In the online DPO setting, č is iteratively generated from the up-to-date policy.
In the offline setting, we implement ExP-DPO using either the expert-provided CoT cg or the model-generated explanation é as the winning response (see Section (6) However, training with cg often results in deceptively low loss values, yet fails to yield strong downstream performance. This occurs because cæ typically has a much lower log-probability compared to the self-generated CoT (the losing response). As a result, during training, even a minor increase in the expert's oog-robbability can suffice to classify it as the correct answer without meaningfully increasing the expert CoT's ss probaaiiity relative to that of the self--enerated one.
Observation 1.
Observation 2. While being effective at the beginning, the offline scheme eventually suffers from distributional drift: as the policy To updates, the fixed explanation è becomes increasingly out-of- distribution, violating the in-distribution criterion outlined in Property [
These observations correspond with our theoretical analysis: nut-of-distribution positive samples can yield ineffective or even detrimental learning signals. To address this, we introduce the iteratively online EXP-DPO, in which fresh explanation é, regenerated after updating mi over a large enough batch of data, i.c. č+ (-1, a"). This formulation ensures that the winning sample remains in-distribution of the evolving model, thereby providing a consistently strong learning signal.
5.2 ExP-GRPO
The Group Relative Policy Optimization (GRPO) method derives its learning signal by sampling responses from the policy model itself. However, this approach has critical limitation: when all sampled responses are incorrect, the model receives no effective gradient signal. such cases, training degenerates into minimizing the KL divergence term alone, which fails guide the model toward better performance. This issue is particularly acute when the sampled questions are too difficult for the model generate valid intermediate reasoning steps. Prior works have circumvented this problem by removing the KL divergence term or excluding such challenging examples from training altogether (37 19 32), but they stop short of addressing the fundamental question- how can a model learn to solve problems it currently fails at? Our method provides principled mechanism to overcome this limitation via the generated eeff-xxllnnttinn and enables the model to explore and learn even when initial sampled responses are incorrect, thereby unlocking the potential of the model to improve on previously unsolvable instances.
In our design, we address such issues by introducing an ExP-SFT term with a scaling coefficient 3. We fix 3 0.04 in all reported experiments and ablation study to justify this choice across multiple 3 values on the full MATH training set is in Appendix. Specifically, we use the initially generated as the CoT that leads the correct answer. Then, we reconstruct the GRPO objective as:
(f) =B (a."">>, foille (om emeo(1l9.")
D Jup Ela min (Fra())Ane cip(r,,(0),115 1,,,,-,,,,)).,) + Blog (ol,
Key Observations. In Section 6 we empirically demonstrate that the EXP-SFT term directly addresses the core challenge of missing learning signals in difficult reasoning settings-where correct responses are exceedingly rare under the initial policy. By injecting in-distribution, task-relevant supervision through é, our method activates the trial-and-error learning loop that is otherwise stalled, enabling the model to make meaningful progress even on instances previously considered unlearnable. Moreover, we show that the addition of the ExP-SFT term significantly enhances sample efficiency and allows the policy to reach a higher performance ceiling than baselines without such augmentation. Further analysis reveals that the majority of the performance gains originate from the harder questions in the test set. This suggests that the reasoning capabilities acquired via the EXP-SFT term not only improve training efficiency but also improve models' reasoning capability on more challenging, unseen instances.
8

---

Qwen2.5-3B-Ir MATH Level-5
Llamm3.2-3R-I MATH Level-5
Qwen2,5-3BBIt MATH
3.amm3.--3B-4 2-3BB-r MATH
Exh-caro GIIPO
700 180 300
Figure 3: Accuracy on the level-5 questions from MATH test set (left 1, 2) and on the whole test set (right 4) for Qwen2.5-3B -nntttsttt- and LLaMA-3.2-38 3B-Instruct across global training steps. ExP- GRPO consistently outperforms both GRPO and GRPO SFT-GT-COT, the latter uses supervised fine-tuning on expert CoT cg. The results show that EXP-GRPO provides more effective and generalizable learning signals, leading to improved sample efficiency and higher overall performance.
We can interpret the EXP-SFT term as form of natural process supervision-it increases the likelihood of generating self-explanations while decreasing that the self-generated CoTs c. The relatively small deviation between these two reasoning traces helps the model pinpoint where and how to adjust. This mechanism is particularly valuable for challenging tasks, where the model may initially fail. In such cases, it is crucial to provide effective positive learning signals that introduce new reasoning traces or knowledge. Compared to GRPO, ExP-SFT provides these signals and guides the model in the right direction, leading greater sample efficiency and performance ceiling.
6 Experiments
We present empirical results demonstrating the effectiveness of ExPO by providing strong on- on-policy learning signals. We begin by evaluating our method the context of ExP-DPO. The detailed experimental results and analysis are in Appendix D.3 We then turn to a more challenging scenario: GRPO 25 training when the initial policy struggles to generate meaningful responses and fails to produce informative learning signals. In this regime, our method ExP-GRPO is especially valuable: jump-starts the learning process by guiding the policy early on, thereby igniting the trial-and-error loop that is critical for sustained improvements in reasoning.
Models and training settings. We conduct preference optimization experiments using two families of models: LLaMA-3.2 (6) and Qwen-2.5 [36). Training and evaluation are performed on two widely used mathematical reasoning benchmarks: MATH [12] and GSM8K B. The self-explanation of ExP-GRPO training is published Other experimental details are in Appendix D.1
6.1 Results on EXP-GRPO
To create scenarios where learning signals are hard to obtain, we picked the difficulty level-5 questions from the MATH dataset. In GRPO training, without extra guidance, these questions are hard for the initial policy to obtain any effective learning signal. In addition, obtaining expert CoT label for hard questions like these are expensive. However, with the help of learning signals from ExPO, the initial policy acquired effective learning signals and soon is able to further improve its performance via online sampling as shown in Figure These results indicate our method's effectiveness on providing helpful learning signals on reasoning training where learning signals are rare.
Further more, in training settings, using the whole MATH training set, that feature a mixture of easy and hard questions- each associated with differing availability of learning signals-we find that augmenting GRPO with the ExP-SFT term markedly improves both sample efficiency and the final performance ceiling, shown in Figure 3 (see Appendix for performance more datasets). This improvement stems from EXP-SFT's ability to consistently inject informative, on-policy learning signals in regimes where standard policy samples fail provide meaningful supervision. To better understand the source of these gains, we conduct a breakdown analysis by question difficulty. As summarized in Table 2
observe that the majority of performance improvements arise from the hardest subset of questions in the test set. This suggests that the reasoning capabilities acquired

---

Level 1
Level 2
Level 3
Level 4
Level 5
# Test Samples
437
894
1131
1214
1324
EXP-GRPO pass @4 GRPO SFT-GT-CoT p
96%
91% %
86%
76% t
23%1
95%
89%
83%
65%
12%
GRPO pass @4
91%
84%
77%
39%
2%
Basc pass @ 64
97%
88%
75%
32%
4%
Basc pass @ 128
97%
93%
80%
42%
9%
Table 2: Accuracy of different methods (base model is Qwen2.5-33B-Instrueee across difficulty levels on the MATH test set. While all methods perform comparably on easier questions, the performance gap widens dramatically on harder levels. Especially on level-4 and level-5 questions, ExP-GRPO yields substantial gains while the standard GRPO fails to Iearn. Moreover, when evaluated under large pass@k
used to reveal the full reasoning capacity of C.LMM--EX--RRE exhibits a substantially broader coverage on difficult questions, effectively harnessing and expanding the model's latent cabblemsslviing ability beyond the reach of conventional RL methods.
via the ExP-SFT signal not only facilitate more effective policy learning during training, but also generalize robustly to challenging, previously unsolvable instances at test time.
7 Discussion and Future Work
Discussion of broader application. While our experiments focus on math reasoning, ExPO's core idea of bootstrapping learning with self-generated. in-distribution samples conditioned on verifiable outcomes applies broadly. For example, in code generation (e.g., Codeforces), correct outputs can be verified via test cases. Given the desired output, the model can be prompted to generate rationale (e.g., pseudocode). enabling ExPO to provide effective training signals. In common-sense reasoning tasks where answers are deterministic, ExPO can generate the multi- step reasoning required to reach the correct answer. More generally, any reasoning task with verifiable rewards (such as physics scientific QA) can benefit from this approach. Ultimately, ExPO demonstrates that model can effectively teach itself without expert-labeled chain-of-thought, paving the way for enhancing language models' complex reasoning ability.
Discussoin of the exploratory role of the advantage-weighie objective. ExPO method uses the relative quality learning signals of the self-generated explanation and direct chain-of-thought. sateer-bttmmybb--iie
While Lemma 2 guarantees that the partial heuristic reasoning paths that still guides the model to learn, we want to further discuss why the imperfect CoTs will not make the model collapse. On the one hand, our method does not collapse to blindly imitating these imperfect CoTs. The ExPO training objective includes reinforcement term with an advantage- advantage-weighted update, which continues to explore around these initial CoTSs This ensures that the model does not simply memorize the self celf-generated explanation č but instead uses as an anchor for exploration, gradually refining the reasoning policy through trial-and-erres On the other hand, we could employ an annealing schedule for the 8 coefficient: gradually decrease B as training progresses. This would make the model rely less on the SFT term in later stages, effectively "fine-tuning" its bootstrapped knowledge during warm-up while reducing the risk of locking in any emergent hallucinations.
self-explanations often provide
Looking ahead, we see two critical areas for enabling complex reasoning capabilities through self- improvement: refining algorithmic approaches and optimizing data curation strategies. From an algorithmic perspective, while RL-based training methods are currently seen as a promising direction for achieving strong reasoning abilities, further investigation into their limitations and underlying mechanisms is necessary. For instance, the distribution- srabbuttoonssaapnnnnngg phenomenon observed in GRPO highlights new challenges that need to be addressed for developing truly capable models. From data perspective, our findings unexpectedly show that expert-generated chain-of-thought examples are not always optimal. This suggests that simply increasing the quality of data does not automatically lead to better performance. Therefore, future efforts should go beyond the focus on expert-annotated high-quality data. Instead, tailoring data to the model's 's current capabilities and designing learning curricula will be crucial for continued model development.
10

---

Acknolwedgement
This research was supported in part by research grant from Coefficient Giving. We thank Victor Wang for his valuable discussions and feedback.
References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
[2] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. arXiv preprint 2xXv:2440.00133 2024.
[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[4] Hugging Face Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https://githut com comm/huggiingffcc//opeee-a
[5] Scott Fujimoto, David Meger, and Doina Precup. Off Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062.
PMLR, 2019.
[6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2177 2024.
[7] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math. Small lims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501. 250400990,,, 2025.
[8] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahemn Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling, arXiv preprint arXiv:2308..890 08998, 2023.
[9] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Direct language model alignment from online ai feedback. arXiv preprint 2rXiv:2402.04700 2024.
[10] hang deng. X-rl. httt:p///ggthaah.ccc///hhoooeppp
2025.
[11] Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:2506.0235 2025.
[12] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL hupp://arxiv. org/abs/2103..mmmm 033.03874 2024.
[13] Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training chain-of-thoughl via latent-variable inference. In NeurIPS, 2023.
[14] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprins 2402:22000064555 2024.
[15] Jiaxin Huang, Shixiang Shane Gu, Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.116. 2022.

---

[16] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, al. Openai ol system card. arXiv preprini arXiv:2412. 16720, 2024.
[17] Hynek Kydlíček. Math-Verify: Math Verification Library. URL https://github. com/
[18] Harrison Lee. Samrat Phatale, Hassan Mansoor, Kellie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023.
[19] meta, llama4. URL httpss//aaimeeaa.ccommomog//////
hhw.://wweeccoeeooooooeeeeeeeene -nttiiimmdddd--telligeneeell 2025.
[20] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 2022.
[21] Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 17:1166171111677 2024.
[22] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how self-improve. Advances in Neural Information Processing Systems, 37:55249 -52285, 2024.
[23] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5333 53741,, 2023.
[24] Amrith Setiur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RL on incorrect synthetic data scales the efficiency of LLM math reasoning by eight-fold. arXiv [cs. June 2024.
[25] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[26] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[27] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12500
2025.
(28) Yongqi Tong, Sizhe Wang, Dawei Li, Yifan Wang, Simeng Han, Zi Lin, Chengsong Huang, Jiaxin Huang, and Jingbo Shang. Optimizing language model's 's reasoning abilities with weak supervision. arXiv preprint arXiv:2405. 4000000 2024.
[29] Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al. Enhancing 11m reasoning with iterative dpo: A comprehensive empirical investigation. arXiv preprint 2tt.:2553.128... 2025.
[30] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning. GitHub repository, 2020.
[31] Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization improves chain-of-thought reasoning. arXiv preprint arXiv:2407. 18248, 2024.
[32] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning large language models with one training example. arXiv preprint arXivv2504.20571, 2025.
12

---

[33] Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judee arXiv preprint arXiv:2407. 2t5i::2207.1999/40 19594, 2024.
[34] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.008 00675, 2024.
[35] Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. 115504111444 1143,, 2025.
[36] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[37] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 1ee122003..467,, 2025.
[38] Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024.
[39] Hui Yuan, Yifan Zeng, Yue Wu, Huazheng Wang, Mengdi Wang, and Liu Leqi. Common pitfalls of margin-based preference optimization in language model alignment. In The Thirleenth International Conference on Learning Representations.
[40] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? 2025. https::/ arxiv. org/ aww.ab///../. 13837
[41] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 1126, 2024.
[42] Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772, 2024.
[43] Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. 2025. URL
https: //tps://arxiv.. h0tg/abs/2504.000
13

---

A Details of Property Derivations
Details of injecting the (0) and g C, a) into the objective3.2 and transforming the expectation into summation. From Equation (3) have
VoJ( VoJ(O)'g(a,ce) ( ( (0 ( Vologmllc.addr(aacccca.
= =E P Vo Vologmcccd 2.6.0
(8)
From we obtain that
VoJ(o) g ( (atrr//aaaa////ooocccccccccca1/a
Ti: when (9.0,)=
w xPP()TTo(,al))) log (c
Vg log Vglosto(cccoaarraac..c.
(4,5,))
Ty: when (g,c,4))((44.)))
argue that the magnitude of the policy improvement AJ (and the alignment between the true gradient (o) and the actual used gradient g(é, a. ÿ)) is dominated by the term Ti.
To be more precise, we characterize when T2 (the sum of cross terms (Va log (C, al Va log (e(c, alg)) in (9) is negligible. Intuitively, these gradients lie in extremely nigh-dimensional spaces (e.g., the order of billions of dimensions when performing full parameter updates, even on relatively small reasoning models). As a result, unless the questions and corresponding CoTs are highly similar, it unlikely that the cross-term gradient inner products contribute significantly. In simplified setting where the logits of the softmax policy we have orthogonal gradients with equal norm, we show that for training samples to which the model assigns lower probability. i.e., To (C, alq) < 2. the cross-term gradient inner products decrease the sample becomes less likely. We formalize this below.
Lemma Let T [(() Cj
a;) a, be finite set, and the policy being a softmax policy over this set, i.e., πT : TC(cj aylq;)))))))))))){{) exp(2)))/
exp(2) where zj = folg; Cj. a.).. Assume the following conditions hold: (1) For all j + L, the gradients of the logits are orthogonal: (Vazj, V 0. (2) All logits have the same gradient norm: || 1| C > for all E [L]. Then for any pair j + j'. the cross-term gradient inner product (V log J V, a log ny) is strictly decreasing in π if and only n
The above lemma implies that when the sample (g. č, a) is assigned low probability under the current policy (i.e., when (2 ā q) is small), the cross-term contributions in Ta become negligible. Simultaneously, the leading term Ti in (9) is also small in magnitude, as it scales proportionally with (2 d)) As a result, such samples contribute minimally to overall policy improvement.
In summary, the policy improvement AJ will be large on gradient step with training data (7. a, a) when To (z, alg) is high that is, when the sample is likely under the current policy-and when the corresponding reward is also high.
B Proofs
Proof of Lemma
Let 2j = ff Sa(g;, az) and define the log-probability gradient as:
L
Vo log n, Voz,
T oz =: Voz,
1=1
Then the inner product becomes:
(V log V Velog
(Vez) Vo2y 2).
14

---

Expanding the above and using the assumptions, we get:
Ty, Ty Vo log
(Voz,, Vozy) Σ nI (Vozj, Vozi)
Σ m Vou)
Σ mm Vozz, Vozm)
= 0 - TCC TyC C+
TCC
1=1
C -T Ty
To study the monotonicity of the gradient inner product in Tjj we check the gradient:
d (Vo log J Vo Volog CC+C
Emp की = + +C2, C(2 - 1).
dr
dry
Therefore, the inner product decreases in ry if and only if πy < 2.
Proof of Lemma 7 We begin by expanding the expectation under the left hand side of the inequality:
(( (T= [2, Σ Telē | 9,a") To(a" (.
(a (. Tolē q) TT(a" (, Ag(a* 9)
Σ Te(e q) (Te(a" a;)))²
To(a"
q)
(elra(ela)
E
[Too(a)) c.q))²].
(a
(g)
where the second equality holds because of Bayes' rule. Note that after applying the Bayes rule, To(a" č, 9) cannot be directly generated by the language model, but it is general distribution (the reverse conditional).
Recall that the right hand side of the inequality is:
E Ee-so(cl)) [To(a" c,, = > To(c) 2))o(aa" | c, To(a" lg).
Thus, the inequality we prove is reduced to comparing between:
E
x
vs.
B[X]
where X =a" | c, c.
E
E[
By Jensen's 's inequality (since the square function x is convex), we have:
E x
E 2
(B[X))²
> E[X],
X
which completes the proof.
C
More discussion on probability shifts
During RL.-type training like GRPO and iterative DPO, negative samples are on-policy and have relatively high probability under the current policy. During training, the probability of these samples
15

---

are pushed down, while the probability of the positive samples will be pushed up. We provide a discussion on how pushing up samples with different probability (under the current policy) influences the change of other sample's probabilities when the policy is a softmax of independent logits. The key takeaways are: (1) Pushing up samples that have low probability under the current policy will not make minimal changes to other samples' probabilities (thus the in-distribution property is needed for effective training). (2) Pushing up and down equal amounts have different effects: the magnitude of changes on other logits depend on the magnitude of the original logits.
To be more specific, let Te perturbation:
ckk eam
be a softmax distribution over logits {zk J and define
2/2 = 24 + Dup.
2j ==2 2j Aown: z/4 = 24 for ls # i,j.
Let e B Aon, and define:
Z Σ e
Z' = ae + B)
Σ E
m
kØL
Then the change in softmax probabilities
ATk Th
Thk is given by:
1
ÅT: et
Z'
Z
B
1
ATy e*
Z'
Z
1
ATH th
for l + i,j-
Z'
First, note that >> and B < 1. Thus, Ar; > 0 and Ar, < 0. The change of other logits are relative to their rank: Ang is proportional to e*t
Thus, we have:
Increasing 2y (pushing up sample k)) increases Ty and decreases ny for all i k.
Decreasing zạ (pushing down sample k) decreases गı and increases n, for all i o k.
The magnitude of these changes is proportional to the product of the original probabilities, implying that updates to high-probability samples have larger impact than those to low-probability candidates.
The effect of only pushing up/down sample probabilities differs from simultaneously pushing up positive samples and pushing down negative ones. Relying solely on negative gradients (i.c., providing signals to decrease the probability of certain samples) will not be effective unless the model already assigns high probability to the correct outputs. This approach is especially ineffective for problems where the model is only partially correct. Conversely, only pushing probabilities up of certain samples can lead to overly aggressive updates. Therefore, it's important to include both appropriate positive and negative samples.
D Additional Result
D.1 Detailed Experimental settings
In the EXP-DPO experiments, we train LLaMA--32-3B-insttne and -.WE--25-3B-insttuue on a single NVIDIA H100 GPU. The optimizer is AdamW with cosine learning rate scheduler and 0.05 warmup ratio, where the maximum learning rate is 5c-7. The training batch size is 16. The basic code frameworks are the trl library (30 https: //git//tttttt thub. com/ hww.hhhggggggcooo //rlllrllle/ and openrl [4] htttss://githuu
(aagggff///pe-rrrr open-r1 The evaluation is performed under the zero- shot prompting, with the simplest prompt for "think step by step" and the answer format. We run the evaluation for passes and calculate the match via Math Verify [17
www.com//aaaee www.ghuggggngggg ace/
com/
huggingface//Mtth- -eeree
Similarly, the ExP-GRPO experiments, we fine-tune L.L.AA-3.--B-iismmmtu and QWEN-2.5- 3B-instruch based on the X-RI [10) https://github. wtwetttub.commhheheeeeeee htwp:///.ccoooocccccceeep// wpp/ R1 GRPO trainer
16

---

NVIDIA A100 GPUs (80GB each). Training is performed for epochs with per-device batch size of and a gradient accumulation step size of 8. We adopt the AdamW optimizer with cosine learning rate schedule, setting the maximum learning rate to 3e-6 and a warmup ratio of 0.1. Mixed-pree Mixed-precision training enabled via b6loat16, and flash attention v2 is used to accelerate attention computation. Gradient checkpointing is applied to reduce memory usage, and training is conducted using the accelerate framework with ZeRO Stage 3 configuration. For both experiments, during training, the generation temperature is set to 0.9, while for evaluation it is set to 0.7.
D.2 Outputs at Initialization
The expert-annotated CoT is mathematically dense and concise, demonstrating the solution by expanding and cancelling terms in single step. In contrast, the self-cxplanation is more pedagogical and compositional; it first defines the core concept, then explicitly states the key logical insight (9! 9 x 8!), and finally performs the simplified calculation. This decomposed, principle-fisy first approach provides a better in-distribution learning signal for RL training.
Example of expert-annotated CoT vs explanation for the question from MATH
Question: Compute without using caleulator: 91/8!
Expert-annotated CoT:
01/8! 8x7x
9x8x--X1
Self-explanation:
The expression 91/8! can be simplified by recognizing that the factorial function rd! (n factorial) defined as product all positive integers up to n. So = 9 x x and 8 = 8 7 ... x 1. Therefore, 9!
-9x 8!. When we divide 9! by 81, we get:
9!
9 x 8!
=
8!
8!
Hence, 91/8!
19.
We also provide the full example of direct CoT vs explanation for the question from MATH in Subsection 4.1
Full example of direct CoT vs explanation for the question from MATH
Question:
A box contains six cards. Three of the cards are black on both sides, one card is black on one side and red on the other, and two of the cards are red on both sides. You pick card uniformly at random from the box and look at a random side. Given that the side you see is red, what is the probability that the other side is red? Express your answer common fraction.
Direct CoT:
To solve this problem, we'll break down into steps. Step 1: Count the total number of sides that red. There are 2 cards that red on both sides, so each has red sides. That's s one side and red on the other, so it has red side. Total red sides = 1 Step 2: Determine the total number of sides in the box. There are 3 cards that are black on both sides, each has 2 sides. That's 3
red sides. There is card that is black on
sides. There are cards that are red on both sides, so each has 2 sides. That's 2 2 4 sides. Total sides 6 4= 10
Step Calculate the probability that the other side red given that the side you see is red.
There are 5 red sides in total, and 5 of those are from cards that are red on both sides. Since we see a red side, we have options for the side we see, and of those are from cards that are red on both sides. The probability that the other side is red given that the side you see
17

---

is red is the number of red sides from cards that red on both sides divided by the total number of red sides. Probability = Number of red sides from cards that are red on both sides Total red sides 5 5
Therefore, the probability that the other side is red given that the side you see is red is Explanation:
To solve this problem, we first need to find the total number of red sides on all the cards in the box.
There are two cards that are red on both sides, so they have 4 red sides in total. There is one card that is black on one side and red on the other, has red side.
In total, there are 4 + = 5 red sides.
Now, we need find the probability that the other side is red given that the side we see is red. This is conditional probability problem.
There are red sides out total of sides in the box.
So, the probability that we If we see red side, could from the 4 red sides of the cards that are red on both sides, or from the red side of the card that is black The 4 red sides of the cards that are red on both sides are out of the red sides we see. So, the probability that the other side is red, given that The answer is based on the idea of conditional probability. We know that there are 5 red red side, it could be from 4 different
red side is 5/6.
side and red the other.
red side, is 4/5.
sides out of total of 6 sides in the box. When cards, each with red sides, or from card with 1 red side. Therefore, the probability that red side, is 4 out of the red sides.
the other side is red, given that
18

---

D.3 Results for ExP-DPO
Offline
Online LLaMA3.2-3-- 3-B-n.tr-ttt
Offline -wen2,5-3B-Insiiee
Online
LL
--en2.5-3-Instruce
Version
C+
Ce.
C,
oz
eg.
6,
Cz
MATH
45.7
38.
50.2
336
54,3
43,7
60.4
49,3
Table 3: Pass @ expert CoT cg+. ExP-DPO consistently outperformed cg trained policy in both online and offline settings.
performance on MATH and GSM8K with models trained with ExP-DPO č, and
We evaluate two variants ExP-DPO: an offline and an iteratively updated online version. For offline, we first compare the impact of using expert CoT's (cg) versus self gelf-generated explanations (č) as the preferred response. As shown in T training with cø leads to only marginal improvements in downstream performance. This observation aligns with our earlier analysis: although expert CoT' are semantically correct, they often in low-probability regions under current policy Ty, violating the in-distribution criterion (Property D]. Consequently, they induce misaligned gradient signals. Moreover, from visualization of its loss (see Appendix for details), we observe that even minimal increases in the log-probability of cg sufficient to yield near-zero loss, creating the illusion of rapid convergence. Therefore, we transition to the online ExP-DPO setting, where the preferred explanation is regenerated periodically from the current policy. These improvements underscore the importance of maintaining distributional alignment between positive training samples and the evolving policy. By continuously adapting è to reflect the model's 's current beliefs, ExP-DPO ensures that learning remains anchored in the regions where the model can most effectively improve, thereby enabling robust optimization without relying external expert demonstrations.
Offine EXP-DPO MATH Performance
Online EXP-DPO MATH Performance
Owen2 1-38
Cwen2. 5
00
40
Figure 4: We compare ExP-DPO performance on the MATH dataset across training steps for two base models: Qwen2.5-3B-Insttree and Llama..2-3B-Instte its offline counterpart, confirming that updating the explanation-basee positive samples improves learning efficiency and final accuracy. Qwen2.5 shows higher sample efficiency and peak accuracy than Llama3.2 under both settings.
Online EXP-DPO consistently outperforms
19

---

Performance
Performance
Owend 5-Iii-instratt
QuenG 5-1B-4rddrrtt
Uemmzz2222 20
Lerna3.2 2 3D instruct
puSSor Neturary
25
75
200 125
150
175
200
25
50
75
100
125
150
125
200
Figure 5: EXP-DPO performance on the GSM8K dataset for (0wen2.5-3BIInstreee and Llama3.2 2-B-BB Instruct models. Online ExP-DPO achieves stronger performance and faster convergence compared to the offline setting. Qwen2.5 benefits more from the online explanation updates, attaining over 85% accuracy, while Llama3.2 2 saturates earlier.
D.4 Deceptive Loss Dynamics in DPO Training
Offline ExP-DPO GT-CoTT MATH Performance and Lass
50
1.0
ExP-DPO
GT-2UT
RXS5- (ccurac)
40-
0.6
0.4
23
Shep 150 0.0
Figure 6: Training curves the MATH dataset using LLaMA-3.2-3B-Insttee We compare standard DPO (with ground-truth answer as the preferred response) against our EXP-DPO method. The solid lines denote model performance (PASS accuracy), while the dashed lines represent the loss. We observe that under standard DPO. the training loss rapidly decreases to a low value; however, this does translate into improved downstream performance, revealing a phenomenon of deceptively low loss. In contrast, ExP-DPO exhibits a smoother loss decay and leads to significantly higher task accuracy. This demonstrates that ExP-DPO provides more reliable learning signal and avoids the overfitting or shortcut behaviors often associated with poorly aligned positive preference supervision.
D.5 Choice of 3 for ExP-GRPO
In our implementation of EXP-GRPO, the guidance term of self-explanations on the ground-truth answer is incorporated into the overall objective via scaling coefficient beta: 3 log ( a a²1g) This is designed to provide an additional learning signal when the model's sampled responses fail to yield effective gradients, particularly in challenging reasoning settings. We fix 3 = 0.04 in all reported experiments. justify this design, we conducted an ablation study across multiple values on the full MATH training set (1 epoch).
Model 9
0.5
0.1
0.08
0.04
0.01
0
Owen2.5-3B-Innste Instructt
47.3%
48.6%
50.5%
68.7%
60.4%
51.6%
LLLMMA332-3B-Inmtte
33.5%
34.8%
35.3%
58.9%
46.6%
44.3%
Table 4: Accuracy of EXPO-GRPO with different values the MATH training set after epoch of training. We can see that B = 0.04 yields to the best training result.
20

---

From Table we observe clear trend: the addition of the ExP-SFT term significantly boosts performance over the baseline fails to provide meaningful updates. However, excessively large 3 values (e-g., 3 = 0.5) degrade performance, likely due to over-reliance on imperfect guidance signals. contrast, smaller values (e.g., B = 0.01) yield slower learning.
0), particularly in early-stage training where standard GRPO
Thus, 3 0.04 offers a favorable trade-off between initial learning efficiency and final performance. It balances the model's ability to leverage the structured supervision from self-generated explana- tions without overwhelming the exploration driven by the advantage term in GRPO. These results empirically support our design choice and highlight the complementary role of the ExP-SFT term in bootstrapping reasoning capability where standard policy optimization struggles.
D.6 EXPO-GRPO results across different base model sizes
Owen2,5-1.5B-1t
LLaMAA332-1B-It
Owen2.5-3B-It
LLaMA-3.2-3B-I
Qwen2,5-7B-It
LLaMA-3.1-SB-I
ExP-GRPO
45.2%
40.5%
68.7%
58.9%
83.2%
70.9%
GRPO SFI-GT-CoT 58.8%
35.8%
61.9%
48, 2%
79.8%
64, 64.9%
GRPO
Table 5: Accuracy of ExPO-GRPO across different base model sizes on the MATH test set. For >7B model, we use the LoRA parameter-effiiiet fine-tuning. ExPO consistently outperforms other baselines across all tested scales.
To better demonstrate the generality and robustness of ExPO, we conducted a broader evaluation across model scales. The results show that ExPO performance gains hold even in >=7B models, suggesting that ExPO's design, which is grounded in in-distribution gradient alignment and positive learning signal, scales effectively with model capacity. These results strengthen claim that ExPO a scalable and general reinforcement learning method for reasoning, even in the absence of expert CoT supervision.
D.7 Additional Result for ExP-GRPO
Qwe12.5-38-
Llama-3.2-30
EXP-GRPP
(2890
100
202
330
100
500
630
700
120
200
300
-00
500
500
700
Stop
Step
Figure Test accuracy on GSM8K for Owen2.5-3B-Instnee (left) and LLaMA--32-3B-instreee (right) across global training steps. ExP-GRPO consistently surpasses both GRPO and GRPO SFT-GT-CoT, the latter of which incorporates supervised fine-tuning on expert CoT cg. These results highlight the effectiveness of ExP-GRPO in delivering stronger and more generalizable learning signals, resulting improved sample efficiency and superior final performance.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our paper contains contain both theoretical and experimental results to support the main claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limirations in the discussion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We present the properties and short proof in section 3 and the complete proof
in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental setup including dataset, model and training in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We will open source the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu http:///ggiddeeeooouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We will open source the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We run the experiment several times for averaging and error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We will open source the code and indicate that in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform the neurips code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss social impact the discussino part.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use open source publicly available models and dataset.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We do original work and cite properly.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is based on open source code base and will open source it. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method is not from LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
28


Reviews:
[{"review_id": "nL4jEtG1ng", "reviewer": "Reviewer_2NX2", "summary": "This paper introduces ExPO (Self-Explanation Policy Optimization), a framework designed to improve the reasoning abilities of large language models (LLMs) without relying on costly expert CoT demonstrations. These expert-curated CoTs are essential to the traditional model. But these CoTs are scarce, especially during early training stages when self-generated CoTs (using question only) perform poorly. ExPO uses self-generated explanations (CoTs generated using both question and correct answer) to create positive samples under the following property: The training example, including the question + its reasoning steps + its answer, should already look fairly plausible to the current model (in-distribution property). And following those reasoning steps should make the model more likely to produce the correct final answer than if it followed its own possibly wrong reasoning (positive learning signal). ExPO improves reasoning performance on hard tasks like MATH and GSM8K. Iworks better than using expert CoTs in many cases. It helps models learn even when they initially fail to generate good answers. ExPO shows that models can bootstrap their own reasoning ability—they don’t need to rely on expensive human-written explanations.", "questions": "1. How regeneration frequency and batch size trade off model improvement against added compute? The online ExP-DPO variant requires periodically regenerating self-explanations with the up-to-date policy, adding lots of computation.\n\n2. Conditioning on the ground-truth answer to generate explanations may inadvertently teach the model to “memorize” rather than truly reason. How does this method perform on out-of-domain reasoning tasks.", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths :\n\nAppropriate Methods: The methods used, including reinforcement learning and policy optimization, are appropriate for the problem at hand. The integration of ExPO with existing algorithms like DPO and GRPO is well-executed.\n \nThe approach addresses a difficult task in a novel way, potentially advancing our understanding and capabilities in training large language models.\n \nThe paper provides new insights with the properties of effective positive training samples into existing [self-explain approach](https://aclanthology.org/2021.emnlp-main.64.pdf).\n\n\nWeaknesses:\n\nStrong assumption required: The gradient‐alignment analysis relies on Lemma 1’s orthogonality and equal-norm assumptions for logits’ gradients, which rarely hold exactly in large transformer models.\n\nEvaluation limited on math reasoning: While the method shows improvements in specific benchmarks, its broader applicability across diverse reasoning tasks remains uncertain, such as Musique, Phantomwiki, and Game of 24.\n \nThe paper compares ExPO primarily with expert-CoT baselines but might not sufficiently address how it stands against other recent papers. For example, [Satori](https://arxiv.org/abs/2502.02508v2) treats CoTs (or action-thought sequences) as trajectories and use RL to maximise answer rewards.", "was_revised": true, "final_justification": "The reiview addressed my questions, and hence I am giving a positive score.", "rebuttal": "Thank you for your positive evaluation on our paper and your supportive comment that our work could “potentially advance our understanding and capabilities in training large language models.\"\n\n>**Weakness 1**: Strong assumption required: The gradient‐alignment analysis relies on Lemma 1’s orthogonality and equal-norm assumptions for logits’ gradients, which rarely hold exactly in large transformer models.\n\nWe thank the reviewer for raising this point. We first want to highlight that the main contribution of our paper is the proposal of ExPO for guiding models to learn to solve hard reasoning tasks. We empirically find that using self-generated explanations outperforms expert CoT in guiding the model in these settings (Figure 3 and Table 2). To understand why these samples can be helpful, we observe that self-explanations have higher likelihood under the policy being trained (Figure 1). Lemma 1 (discussed in Section 3.1) is intended to provide intuition for why higher-likelihood samples may lead to gradients that are more aligned with the true gradient. We agree that the assumptions in Lemma 1 are strong, but emphasize that they are used only to illustrate an idealized setting to build intuition. Our empirical results and observations do not rely on these assumptions.\n\n>**Weakness 2**: Evaluation limited on math reasoning: While the method shows improvements in specific benchmarks, its broader applicability across diverse reasoning tasks remains uncertain, such as Musique, Phantomwiki, and Game of 24.\n\nWe appreciate the reviewer’s concern regarding the broader applicability of our method. While our experiments focus on mathematical reasoning, we emphasize that the core idea behind ExPO—bootstrapping learning via self-generated, in-distribution positive samples conditioned on verifiable outcomes—extends naturally to other reasoning tasks.\n\nIn common sense reasoning tasks like Musique, the final answer is deterministic and can be verified against the ground-truth output. Therefore, ExPO can similarly generate training signals that satisfy both the in-distribution and positive learning signal properties discussed in Section 3.1. More broadly, any reasoning task with a verifiable reward—such as the retrieval-augmented common sense reasoning task Phantomwiki. We agree that adapting ExPO to these domains may require domain-specific prompting strategies, and we view this as an exciting direction for future work.\n\nWe will clarify this point in the revised version and appreciate the opportunity to elaborate on the method’s generalization potential.\n\n>**Weakness 3**: The paper compares ExPO primarily with expert-CoT baselines but might not sufficiently address how it stands against other recent papers. For example, Satori treats CoTs (or action-thought sequences) as trajectories and use RL to maximise answer rewards.\n\nThanks for pointing us to the interesting paper “Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search”. It proposes a new reasoning format COAT that allows for self-reflection and iterative training to achieve impressive results.\n\nBut our work aims to address this problem from a different perspective: we propose the simple framework ExPO that is easy to integrate with different RL training algorithms, accompanied by theoretical proofs. The in-distribution property and positive learning signal property provide a formalization to the intuition of why bootstrapping could work, and is supported by our experiments of ExP-DPO and ExP-GRPO.\n\nThank you for this excellent suggestion and we will add the discussion of current work like Satori in the revised version.\n\n> **Question 1**: How regeneration frequency and batch size trade off model improvement against added compute? The online ExP-DPO variant requires periodically regenerating self-explanations with the up-to-date policy, adding lots of computation.\n\nThank you for the question regarding the trade-off between computational cost and model improvement, particularly for our online ExP-DPO variant.\n\nWe would like to first explain that, among our experiments, the efficiency gains are most pronounced for online methods like ExP-GRPO. During the standard GRPO training process, only a correctly generated CoT could provide the positive training signal. This process is highly inefficient for difficult tasks. For instance, on MATH Level 5 questions, the qwen-2.5-3b model achieves only 4% accuracy for pass@64 and only 9% for pass@128. **This means over 90% of the computational effort in generation during training is wasted** as the responses cannot be used for positive reinforcement. \n\nIn contrast, as shown in Figure 2, our self-explanation method generates significantly higher-quality CoTs. By producing more valid reasoning paths, a much larger fraction of the sampled responses can be utilized as positive signals, making the training process substantially more sample-efficient and computationally effective.\n\nRegarding the concern about the added computation in iteratively online ExP-DPO, we would like to explain that in fact, DPO is an offline method by nature. It operates on a static, fixed dataset of preference pairs that must be available at the start of training. For complex reasoning tasks, the \"chosen\" samples are typically human expert annotated CoTs. However, if we want to train the model without relying on expert annotations, we must enable the model to generate its own positive signals and force the use of an iteratively online DPO framework. It is true that this iteratively online process increases the computational cost compared to the classical offline DPO method, but this comparison is not entirely fair, as the two approaches solve different problems—one relies on existing expert data, while ours generates its own. And the most appropriate way to evaluate the computational overhead of our ExP-DPO method is to compare it against other iteratively online DPO methods.\n\n> **Question 2**: Conditioning on the ground-truth answer to generate explanations may inadvertently teach the model to “memorize” rather than truly reason. How does this method perform on out-of-domain reasoning tasks.\n\nThank you for raising the point of potential memorization and out-of-domain generalization. Regarding the concern of memorization, all of the results presented in our paper are evaluated on the held-out test set, for example MATH500 (the subset of the test split of MATH) and the test split of GSM8K. The strong performance that our model achieves on this unseen test set provides compelling evidence against simple memorization. \n\n> **Paper formatting concern**:\n\nWe sincerely thank the reviewer for this valuable comment and we will revise the dense writing by providing more explanations and illustrative examples."}, {"review_id": "wCgEg2UQOH", "reviewer": "Reviewer_UZkv", "summary": "This paper investigates the properties of chain of thought (CoT) that make them useful in improving the reasoning abilities of models. The main theoretical hypotheses are: 1) CoTs that are in-distribution are more useful than other CoTs, and 2) CoTs that increase the likelihood of the answer are more useful than other CoTs. Formal proofs for these statements are presented. Based on these hypotheses, the authors generate CoTs conditioned on the gold answer, also referred to as \"self-explanations,\" thus meeting both criteria — being in-distribution and increasing the likelihood of producing the target answer. The proposed method ExPO demonstrates that these self-explanations outperform expert CoTs and other CoTs in both offline and online policy optimization using two training methods, DPO and GRPO.", "questions": "1. Please explain the statement: \"They must be better than negative samples in achieving the task at hand.\" What does it mean for a negative example to achieve a task?\n2. Please clarify: \"In settings where a+ and a_ are incorrect\"—what's an example of this? Why would anyone train on CoTs that have generated wrong answers in the first place? Neither DPO nor GRPO does this.\n3. What happens if the likelihood of the correct answer is high but occurs under a \"wrong CoT\"? One could easily generate such data to meet both criteria (using an LLM as a judge and selecting incorrect CoTs), and the trained model would likely be worse. Doesn't this contradict your claims even though the conditions are met?\n4. What if the self-explanation CoT yields a higher likelihood for a wrong answer, i.e., p(a- | q, c_a*) > p(a* | q, c_a*), even though c_a* is conditionally produced using p(. | q, a*)?\n5. In Table 1, why does online Llama-3.2 perform better with c_e than with c_+?", "limitations": "- The paper fails to address important scenarios:\n    - What happens when the likelihood of the correct answer is high but occurs under a \"wrong CoT\"?\n    - What if the self-explanation CoT yields a higher likelihood for an incorrect answer (i.e., p(a- | q, c_a*) > p(a* | q, c_a*)), even though c_a* is conditionally produced using p(. | q, a*)?\n- The paper doesn't clearly state its underlying assumptions.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well-structured (though it contains vague statements that make it harder to follow—more on this later).\n- The experimental results substantially support the theoretical framework.\n- Many of the underlying intuitions in the paper are common knowledge, and it's valuable to see these concepts formally articulated.\n\n## Weaknesses\n\n- Many statements are vague:\n    - For example, \"They must be better than negative samples in achieving the task at hand.\" What does it mean for a negative example to achieve a task?\n    - The statement \"In settings where a+ and a_ are incorrect\"—what's an example of this? Why would anyone train on CoTs that have generated wrong answers in the first place? Neither DPO nor GRPO does this.\n- The formal part of the paper does not clarify its assumptions. It would be useful to discuss what happens if these assumptions are not met.\n- Connections to curriculum learning are not discussed. Essentially, if a problem is too out-of-distribution for the current model, the model will not be able to learn.\n- No qualitative examples are shown. Contrasting expert CoTs with self-explanations would be useful. The paper doesn't include even one example.", "was_revised": true, "final_justification": "See my response.", "rebuttal": "We sincerely thank the reviewer for their thorough and constructive feedback. We appreciate the reviewer's positive feedback on the paper's structure, formalization of common intuitions, and abundant experiments. We also appreciate the insightful questions that help us to significantly improve the clarity of the manuscript, and we want to address them in detail below.\n\n**Weakness 1, Question 1, Question 2: vague statements.**\n>\"They must be better than negative samples in achieving the task at hand.\" What does it mean for a negative example to achieve a task?\n\nThe sentence \"better than negative samples in achieving the task at hand.\" is trying to explain the intuition of property 2. For a pair of CoT $(c_+, c_i)$, we say that $c_+$ is better at achieving the task if $\\pi_\\theta(a^\\star | q, c_+)  \\gt \\pi_\\theta(a^\\star*|q, c_-)$. That is, $c_+$ is more probable for $a^\\star$ to appear under the current policy than $c_-$.\n\n>\"In settings where a+ and a_ are incorrect\"—what's an example of this? Why would anyone train on CoTs that have generated wrong answers in the first place? Neither DPO nor GRPO does this.\n\nWe would like to mention our paper’s setting: (1) difficult tasks that the untrained model can hardly solve, for example qwen-2.5-3b-instruct only achieves 4% accuracy on MATH-level 5 dataset with pass@64; (2) only the model self-generated CoTs are used to train the model itself. In such cases, it is probable that the CoT sampled from the model might lead to an incorrect final answer. And it is exactly the case where standard DPO or GRPO would fail, as there are no \"winning\" samples to provide a positive learning signal. Some people do skip these batches where all CoTs lead to wrong answers and wait long for a correct one to appear, but the training will become extremely inefficient ( qwen-2.5-3b-instruct only achieves 4% accuracy on MATH-level 5 dataset with pass@64).\n\nOur ExPO method can overcome this problem. We argue that even within a batch of incorrect CoTs, some will be of higher quality than others. Our paper aims to formally define what \"higher quality\" means in an RL context and provide a principled way to treat these better-but-may-be-imperfect CoTs as positive samples $c_+$ relative to even worse ones $c_-$. This allows the model to make incremental progress on difficult problems where it would otherwise have no learning signal at all.\n\t\nIn addition, our method doesn't just blindly copy these imperfect CoTs. The ExPO training objective includes a reinforcement learning term that encourages the model to explore beyond the initial explanation. This ensures the model doesn't simply memorize the self-generated explanation $\\tilde{c}$. Instead, it uses the explanation as an anchor for further exploration, gradually refining its reasoning policy through trial and error.\n\nWe hope our explanations above have provided the necessary clarity on our methodology and motivation. Should any aspect of our response still seem unclear, or if further questions arise, we would be more than happy to provide additional details to ensure our arguments are well understood.\n\n> **Weakness 2**: Connections to curriculum learning are not discussed. Essentially, if a problem is too out-of-distribution for the current model, the model will not be able to learn.\n\nThank you for this comment. We agree that if a problem is too far beyond the model's current capabilities, the learning signal is often negligible, and the model will fail to improve. Indeed, a key implication of our work is the critical importance of careful data curation. Our finding that models learn more effectively from data tailored to their current capabilities directly supports the core idea of a curriculum, where task difficulty is gradually increased as the model becomes more competent.\n\nWhile a full exploration of curriculum learning strategies was beyond the scope of this paper, we agree that it is a very important and natural direction for future work. We will add the discussion to address this connection with curriculum-based approaches.\n\n> **Weakness 3**: No qualitative examples are shown. Contrasting expert CoTs with self-explanations would be useful. The paper doesn't include even one example.\n\nThank you for the suggestion. We provided a qualitative example comparing self-generated CoT and self-explanation in Appendix C.1 and will add the following one for comparing against expert CoT in the future version of the paper:\n\n- Question: \"Compute without using a calculator: $9!/8!$\"\n- Expert-annotated CoT: “${9!}/{8!} = \\\\dfrac{9 \\\\times 8 \\\\times 7 \\\\times 6 \\\\times \\\\cdots \\\\times 1}{8 \\\\times 7 \\\\times 6 \\\\times \\\\cdots \\\\times 1} = \\\\boxed{9}$.”\n- Self-explanation: “The expression $9!/8!$ can be simplified by recognizing that the factorial function $n!$ (n factorial) is defined as the product of all positive integers up to $n$. So, $9! = 9 \\\\times 8 \\\\times 7 \\\\times 6 \\\\times 5 \\\\times 4 \\\\times 3 \\\\times 2 \\\\times 1$ and $8! = 8 \\\\times 7 \\\\times 6 \\\\times 5 \\\\times 4 \\\\times 3 \\\\times 2 \\\\times 1$. \\n\\nTherefore, $9! = 9 \\\\times 8!$. When we divide $9!$ by $8!$, we get:\\\\[\\\\frac{9!}{8!} = \\\\frac{9 \\\\times 8!}{8!} = 9\\\\]Hence, $9!/8! = 9$.”\n\nThe expert-annotated CoT is mathematically dense and concise, demonstrating the solution by expanding and cancelling terms in a single step. In contrast, the self-explanation is more pedagogical and compositional; it first defines the core concept, then explicitly states the key logical insight ($9! = 9 \\times 8!$), and finally performs the simplified calculation. This decomposed, principle-first approach provides a better in-distribution learning signal for RL training.\n\n> **Question 3 & Limitation 1**: What happens if the likelihood of the correct answer is high but occurs under a \"wrong CoT\"?\n\nWe thank the reviewer for this insightful question. First, we want to mention that **lemma 2** guarantees that the better-but-may-be-imperfect CoTs is on average better than self-generated CoT in producing the correct answer (as also verified empirically in Figure 2). However, it is indeed the case that the self-generated CoTs conditioned on the ground-truth answer may be imperfect and contain wrong steps. In such cases, the generated self-explanations $\\tilde{c}$ , although potentially imperfect, often provide partial or heuristic reasoning paths that still guides the model to learn. \n\nCritically, our method does not collapse to blindly imitating these imperfect CoTs. The ExPO training objective includes a reinforcement term with an advantage-weighted update, which continues to explore around these initial CoTs. This ensures that the model does not simply memorize the self-generated explanation $\\tilde{c}, but instead uses it as an anchor for exploration, gradually refining the reasoning policy through trial-and-error.\n\nAs shown in Section 4.2 and Table 2 of our paper, this mechanism is particularly effective on level-5 MATH questions, where baseline methods fail to generate any learning signal. ExPO, by contrast, enables the model to bootstrap and continue improving—demonstrating that imperfect CoTs can still serve as valuable scaffolding for RL-based learning. We agree that this is an important point and will clarify the exploratory role of the advantage-weighted objective in the final version.\n\n> **Question 4 & Limitation 2**: What if the self-explanation CoT yields a higher likelihood for a wrong answer, i.e., p(a- | q, c_a*) > p(a* | q, c_a*), even though c_a* is conditionally produced using p(. | q, a*)?\n\nThank you for the thoughtful question. We would like to clarify that when we compare self-explanations to self-generated CoTs, our focus is on their **relative quality** as learning signals. More specifically, from Lemma 2, we know that on average, the chance of generating a wrong answer under self-explanations is lower than that of self-generated CoTs: $1 - E_{\\tilde{c} \\sim \\pi_\\theta(\\tilde{c} | q, a^\\star)}[\\pi_\\theta(a^\\star | q, \\tilde{c})] \\leq 1 - E_{c \\sim \\pi_\\theta(c | q)}[\\pi_\\theta(a^\\star | q, c)]$. This inequality implies that although a self-explanation may still lead to a wrong answer, it is on average a more reliable guidance than normal CoT. In such cases, self-explanations provide effective guidance to the model in exploring better reasoning paths during training as demonstrated in our empirical results. \n\n> **Question 5**: In Table 1, why does online Llama-3.2 perform better with c_e than with c_+?\n\nThank you for the question. It is indeed the case that in Table 1, the online DPO experiment with LLaMA-3.2 shows slightly better performance with $c_e$ compared to $c_+$. However, the performance gap is small and should be interpreted in the broader context of our findings.\n\nAcross the majority of our experiments—including those with other models and setups—$c_+$ consistently outperforms $c_e$, both in terms of final performance and training efficiency. Specifically, training with $c_+$ improves faster than with $c_e$, which we attribute to the in-distribution property outlined in Property 1. As shown in Figure 1, the negative log-likelihood of the self-explanation $c_+$ is lower than that of the expert CoT $c_e$.\n\nTaken together—results across other models and setting, faster performance improvement over training trajectory, and the empirical observation that c_+ is more “in-distribution”—we believe the overall evidence still supports our claim that $c_+$ offers a more effective learning signal than $c_e$."}, {"review_id": "muHlHE1MrZ", "reviewer": "Reviewer_uMrA", "summary": "This paper focuses on RL-style post training method, and deals with high cost and sparsity of positive responses. The authors identify two key properties of effective positive samples, high probability under the current policy and increase the model's likelihood for the correct answer. And then, this paper develops the self-explanation policy optimization, ExPO, which generates such positive samples from the model itself. Experiments confirm the effectiveness and efficiency.", "questions": "1.The paper argues in introduction that \"Perhaps surprisingly, empirically, we observe that these \"gold\" responses—though containing correct CoTs—may fail to provide effective learning signals\". I suggest adding such empirical experiments in the intro to make the motivation clearer.\n\n2.Like DeepSeek-R1, only relying the system prompt (<think>) and ground-truth answer can start RL training. Does it conflict with the claims (e.g., in Line 50 \"Unlike prior work that relies on fixed expert CoTs\") in this paper?\n\n3.Some details can be enhanced, 1) in figure 1-2, the base model is qwen2, while the main experiments use qwen2.5, what is the reason behind it? 2) in table 1, can you provide a detailed introduction on the offline and online?\n\n4.Typos: 1) Line 48: \"positive training samples These samples\" ->\"positive training samples. These samples\" 2) Line 227: \"self-generated CoT c\" -> \"self-generated CoT c˜\"", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strength\n\n1. Continuous improvement and exploration of effective principles for RL on LLM reasoning are of great research significance.\n\n2. This paper provides a comprehensive theoretical analysis for exploring the effectiveness of RL-style post training.\n\n3. The authors analyze both the offline DPO and onlie GRPO to confirm the reliability of ExPO.\n\n- Weakness\n\n1. From the technical perspective, the contribution of this paper is somewhat weak. Generating a self-explanation conditioned on the correct answer in Eq.(8) is not new for existing work.\n\n2. The sampling efficiency is not well consolidated and validated.\n\n3. Some details are not clear, referring to questions.", "was_revised": true, "final_justification": "Comprehensively considering the original paper's contribution and the rebuttal, I think the original score is appropriate and I tend to keep my score.", "rebuttal": "We sincerely thank the reviewer for their time and insightful feedback. We are especially grateful and very encouraged for their recognition on the work's \"great research significance,\" our \"comprehensive theoretical analysis,\" and the experimental reliability confirmed through \"both the offline DPO and online GRPO.\"\n\n> **Weakness 1**: Generating a self-explanation conditioned on the correct answer in Eq.(8) is not new for existing work.\n\nThank you for this comment. We agree that the concept of generating explanations conditioned on the correct answer and using it for supervised fine-tuning (SFT) has been explored in STaR. However, our work builds directly on that foundation and addresses a deeper, unresolved question that STaR itself identifies.\n\nWhile STaR demonstrated the empirical effectiveness of self-explanation during SFT, the authors explicitly state that the underlying reasons for this effectiveness remain unclear. To quote from their paper: “Future work should establish more connections between rationalization [self-explanations] and these RL objectives, and examine more generally when and why rationalization [self-explanations] improves learning.” Our work is a direct response to this open problem.\n\nOur primary contribution is not only the act of conditional self-explanation generation, but rather the easy-to-use ExPO framework that integrates these self-explanations into RL training, together with a formalization on what may enable self-explanations to provide better learning signals to language models. We identify two key properties that high-quality positive samples must satisfy. This formalization makes our approach more general and extensible than previous methods: it clarifies that many other self-explanations (e.g., explanations based on additional hints) can serve as a positive sample, as long as they conform to the two fundamental properties we propose.\n\n> **Weakness 2**: The sampling efficiency is not well consolidated and validated.\n\nThank you for this question regarding the sampling efficiency. In standard GRPO training, only a correctly generated CoT could provide the positive training signal (if we disgard the formatting rewards or if all samples have the same formatting rewards). This process is highly inefficient for difficult tasks. For instance, on MATH Level 5 questions, the qwen-2.5-3b- instruct model achieves only 4% accuracy for pass@64 and only 9% for pass@128. This means over 90% of the computational effort in sampling the model responses for these hard tasks during training may beis wasted  as the responses cannot be used for providing learning signal. \n\nIn contrast, as shown in Figure 2, our self-explanation method generates significantly higher-quality CoTs. By producing more valid reasoning paths, a much larger fraction of the sampled responses can be utilized to provide positive signals, making the training process more sample-efficient and computationally effective as shown in the test performance trajectory during training (Figure 3 and 4).\n\n> **Question 1**: The paper argues in introduction that \"Perhaps surprisingly, empirically, we observe that these \"gold\" responses—though containing correct CoTs—may fail to provide effective learning signals\". I suggest adding such empirical experiments in the intro to make the motivation clearer.\n\nThank you for this suggestion. The statement you highlighted is a reference to one of our experimental findings: in the case of math reasoning tasks of MATH and GSM8K, the self-explanation approach can outperform training on \"gold\" expert-annotated CoTs.\n\nThis was at first an unexpected observation during our early experiments, which motivated us to investigate the underlying reasons more deeply. This investigation led to the formulation of the in-distribution property. It emphasizes the importance of tailoring training data to the model's current capabilities rather than simply using human-annotated 100% correct data that may be too far beyond the model's current policy.\n\nThanks for your comment and we will revise the introduction to make this connection explicit and point the reader to the specific results in the experiment section. We appreciate you helping us make the paper's motivation clearer.\n\n> **Question 2**: Like DeepSeek-R1, only relying the system prompt ([object Object]) and ground-truth answer can start RL training. Does it conflict with the claims (e.g., in Line 50 \"Unlike prior work that relies on fixed expert CoTs\") in this paper?\n\nThank you for this question and we want to clarify the context of our work and its primary contribution. Our claim in Line 50, \"Unlike prior work that relies on fixed expert CoTs,\" refers to a common practice in the field. When high-quality, human-annotated CoTs are available, researchers often use them to train reasoning models. This is prevalent in both major training stages:\n\n- In the SFT stage: for example LLaVA-CoT and ReFT use expert-annotated CoTs to train the model; DeepSeek-R1 uses a small dataset of gold CoTs to warm up the model.\n\n- In the RL stage: imitation learning and behavioral cloning methods often depend on expert demonstrations, for example in the BREAD and FingER papers.\n\nOur work addresses the scenario where such expert data is NOT available (or only a few), so that the GRPO method is commonly used, which is also the case for DeepSeek-R1. Our ExP-GRPO experiment is designed to operate in this setting. \n\nUnder the \"no-expert-data\" scenario, our work emphasizes the importance of model self-generated data and formalizing the principles to explain why self-generated data could provide a better signal than expert-annotated data. We formalize this intuition in Property 1 and Figure 1 empirically supports the intuition that self-explanations are indeed more in-distribution than expert CoTs. Furthermore, the experiments of ExP-DPO $c_+$ vs $C_E$ in Table 1 and ExP-GRPO vs GRPO SFT-GT-CoT in Figure 4 show that these in-distribution samples provide effective learning signals and lead to better performance.\n\n> **Question 3**: Some details can be enhanced, 1) in figure 1-2, the base model is qwen2, while the main experiments use qwen2.5, what is the reason behind it? 2) in table 1, can you provide a detailed introduction on the offline and online?\n\n1)Thank you for your careful reading and this was a typo in the figure captions. The Initial model used throughout all of our experiments is Qwen2.5. We will correct this typo in the revised manuscript.\n\n2)We explain in detail the offline and online DPO methods here.\n\n- Offline DPO is the classical DPO method. The entire dataset of preference pairs is generated once at the beginning of training using the initial reference policy π_ref. For offline ExP-DPO, both the preferred winning and dispreferred losing responses are sampled from this initial policy, i.e. $\\tilde{c_+} \\sim \\pi_{ref} ( \\cdot | q, a^\\star)$ and $\\tilde{c_-} \\sim \\pi_{ref} ( \\cdot | q )$.\n\n- Iteratively online DPO method has a dynamic training process. At the beginning of each iteration, the preferred winning explanation is regenerated using the current updated policy, as well as the dispreferred samples. This allows the model to learn from progressively better positive examples as it improves.\n\n> **Question 4**: Typos.\n\nWe sincerely thank the reviewer for their meticulous attention to detail in identifying several typos. We will correct them in the revised version. We appreciate their effort in helping us improve the overall readability and quality of the paper."}, {"review_id": "ypmWM1dZNZ", "reviewer": "Reviewer_D1iU", "summary": "This paper introduces ExPO (Self-Explanation Policy Optimization), a modular RL method based on DPO or GRPO that can improve LLM learning efficiency and final performance on math reasoning benchmarks. By bootstrapping the LLM with the question and ground truth answer, ExPO obtains positive samples that satisfy two criteria: (1) in-distribution and (2) positive learning signal, which are considered essential for the main optimization objective. ExPO leverages these positive samples to provide more effective learning signals for DPO and GRPO. Through experiments that compare ExPO with expert CoT baselines, this paper demonstrates the effectiveness of ExPO.", "questions": "1. Bootstrapping the LLM with the ground truth answer is effective for some questions since it reshapes the search space, allowing the LLM to \"think outside the box\". However, it may also introduce some bias. For questions that the LLM can not figure out a real correct solution even with the hint (which can be common for complex reasoning questions), it still generates a CoT solution although it's not correct (falsely copying the ground truth answer). Learning from these CoT data may have negative influences. Have you noticed or considered this problem?\n2. In the reconstructed GRPO objective, the scale of the original part and the additional SFT term seem unclear, making the new objective design seem unreliable. Are there any justifications or observations that support this design?\n3. As mentioned in the Weaknesses part, could you elaborate on the method you use to obtain expert CoT data, as well as the compared GRPO SFT-GT-CoT baseline? \n4. Are there any RL results for models at different scales (7B, 32B, etc.)? Considering that currently displayed results are all for 3B LLMs, it may not reflect the complete picture.", "limitations": "1. As mentioned in the Weaknesses, the generalizability of this approach is limited.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- **Strengths**\n1. Significance: This paper provides some insights in the selection of training data and the optimization process of reinforcement learning algorithms, discovering a new approach for improving current RL algorithms like GRPO and DPO.\n2. Originality: The idea of incorporating STaR-style rationales [1] into RL optimization is novel. This paper seems to have made an effective attempt on extending the STaR methodology.\n- **Weaknesses**\n1. Rigor: One major issue of this paper is insufficient rigor concerning mathematical derivations and conclusions. For example, the claim in expression (7) is not supported by convincing mathematical proofs (Lemma 1 is actually very weak). The conclusion that data with higher probability leads to greater gradient is also erroneous empirically.\n2. Insufficient experimentation: The displayed experimental results are very limited. It can not suffice to justify the ExPO approach.\n3. Clarity: This paper also faces clarity issues. For example, the source of expert CoT data is unclear, although it is considered as an important baseline. Besides, The implementation of the GRPO SFT-GT-CoT baseline is also ambiguous.\n4. Generalizability: The proposed bootstrapping method based on ground truth answer may not be applicable for other reasoning tasks such as code generation.\n\n[1] Zelikman, E., Wu, Y., Mu, J., & Goodman, N. (2022). Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, 15476-15488.", "was_revised": true, "final_justification": "I have raised my score for the author's supplementary experimental results, which improved the solidity of this paper. However, the proposed method may introduce bias or cause hallucination in a long term perspective, since it relies on self-generated explanations. Besides, the generalizability of this method towards other domains needs further experimentation.", "rebuttal": "We thank the reviewer for their thoughtful comments and appreciate the recognition of our contributions to training data selection and RL optimization, as well as the novelty of incorporating STaR-style rationales into RL.\n\nThe core contribution of our paper is **ExPO**, a method to bootstrap learning in **hard reasoning tasks** where positive samples are rare (e.g., Qwen2.5-3B achieves only 4% pass@64 on MATH Level-5). In such settings, GRPO often fails (Fig. 3, Table 2). ExPO addresses this by using **self-explanations** on the correct answer to warm-start training. Empirically, these signals are more effective than expert chain-of-thoughts (CoTs) for hard problems.\n\nTo understand why, we build on the open question in STaR about when and why self-explanations[rationales] help. Our **empirical analysis** (Sec. 3.2, App. C.1) shows that: (1) self-explanations have higher likelihood under the current policy than expert CoTs (Fig. 1), and (2) they contain more correct reasoning steps than self-generated CoTs (Fig. 2). Our theory (Sec. 3.1) provides **intuition** on why these properties support effective policy improvement.\n\nWe now address each reviewer's point in detail below:\n\n>**W1**: “One major… empirically.”\n\nThank you for the question. Sec. 3.1 is intended to provide intuition for why the two key properties of self-explanations (identified in Sec. 3.2) may lead to better policy improvement, compared to self-generated and expert CoT. L139-134 (incl. Eq. 7) and Lemma 1 aim to provide intuition on why when a sample has low likelihood, its gradient may have low inner product with the true gradient. We do not claim that high-likelihood samples have high gradient norms, and we will clarify this in the revision.\n\n>**W2 ^ Q4**: “Insufficient …approach.” “Are there … complete picture.”\n\nWe thank the reviewer for raising this concern. We agree that a broader evaluation across model scales will better demonstrate the generality and robustness of ExPO. While our original submission focused on 3B-scale models due to resource constraints in academia settings (maximum available compute: 4×L40S, 48GB each), we took the reviewer’s comment seriously and conducted additional experiments under these constraints.\n\nTo scale to ≥7B models, we adopted parameter-efficient fine-tuning using LoRA (rank 32) with DeepSpeed ZeRO-3. Despite hardware limitations, this allowed us to run controlled RL training experiments on larger models such as LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct.\n\n\n| Method\\Model   | Qwen2.5-1.5B-Instruct | LLaMA-3.2-1B-Instruct | Qwen2.5-3B-Instruct | LLaMA-3.2-3B-Instruct | Qwen2.5-7B-Instruct (LoRA)| LLaMA-3.1-8B-Instruct (LoRA)|\n|-|-|-|-|-|-|-|\n| ExP-GRPO | 45.2%     | 40.5%     | 68.7%     | 58.9%     | 83.2%     | 70.9%     |\n| GRPO SFT-GT-CoT | 38.8%     | 35.8%     | 61.9%     | 48.2%     | 79.8%     | 64.9%     |\n| GRPO | 32.5%     | 28.3%     | 50.3%     | 44.4%     | 78.1%     | 63.9%     |\n\nExPO consistently outperforms other baselines on MATH[2] across all tested scales, particularly on difficult questions (e.g., MATH[2] level-5 from table 2 of our paper). Notably, the performance gains hold even in >=7B models, suggesting that ExPO's design—grounded in in-distribution gradient alignment and positive learning signal—scales effectively with model capacity. These results strengthen our claim that ExPO is a scalable and general reinforcement learning method for reasoning, even in the absence of expert CoT supervision.\n\nWe believe this addresses the reviewer’s concern regarding the scope of the experimental validation. Furthermore, we are committed to expanding these large-scale experiments and will include more results in the future version of the paper, where possible.\n\n>**W3a & Q3**: “The source … GRPO SFT-GT-CoT baseline?”\n\nThank you for raising this point.. The expert CoT annotations used in our baselines for GSM8K and MATH are sourced from the official dataset releases associated with the following references:\n\n*[1]Cobbeet al. Training verifiers to solve math word problems. 2021.\n*[2]Hendrycks et al. Measuring mathematical problem solving with the math dataset, 2021. \n\nThese datasets are publicly available and widely adopted in prior work on mathematical reasoning. Due to the rebuttal policy, we cannot include direct hyperlinks, but we confirm that these datasets (including expert CoTs) are accessible through their official HuggingFace repositories. We have also ensured that our implementation uses these expert CoTs exactly as provided in the original dataset releases, without any modifications.\n\nWe will clarify this in the revision to ensure transparency and reproducibility.\n\n>**W3b**: “The implementation … ambiguous.“\n\nWe apologize for the ambiguity and thank the reviewer for pointing this out. To clarify, the GRPO SFT-GT-CoT baseline corresponds to a modified version of the GRPO objective described in Sec. 3.3.2 (ExP-GRPO). Specifically, in the ExP-GRPO formulation, the ExP-SFT term:\n\n$\n\\beta \\log \\pi_\\theta(\\tilde{c},a^{\\star} | q)\n$\n\n(where $\\tilde{c}$ is the self-explanation on the ground-truth answer) is replaced in the GRPO SFT-GT-CoT baseline by:\n\n$\n\\beta \\log \\pi_\\theta(c_{E},a^{\\star} | q)\n$\n\nwhere $c_{E}$ is the expert CoT annotation provided in the official release of the MATH or GSM8K datasets. All other components of the GRPO training objective remain the same. We will make this detail more explicit in the revision to prevent confusion.\n\n>**W4**: “Generalizability … generation.“\n\nThank you for raising this point. While our experiments focus on math reasoning, the core idea of ExPO—bootstrapping learning with self-generated, in-distribution samples conditioned on verifiable outcomes—applies broadly. For example, in code generation (e.g., Codeforces), correct outputs can be verified via test cases. Given the desired output, the model can be prompted to generate rationale (e.g., pseudocode), enabling ExPO to provide effective training signals, as discussed in Sec. 3. More generally, any reasoning task with verifiable rewards—such as physics or scientific QA—can benefit from this approach. We agree that applying ExPO in these domains may require tailored prompting or tools, and we see this as a promising direction for future work. We will clarify this in the final version.\n\n>**Q1**: “Generates … problem?”\n\nWe thank the reviewer for this insightful question. Indeed, self-explanations conditioned on the ground-truth answers (denoted $\\tilde{c} \\sim \\pi_\\theta ( \\cdot | q, a^\\star)$) are not guaranteed to be logically correct or aligned with optimal reasoning. We clarify both their intent and role in the learning process.\n\n**The core goal of our method is not to teach perfect reasoning, but to jump-start learning when standard RL objectives (e.g., GRPO) fail to provide gradients—especially on hard questions where models can't produce correct responses.** In such cases, $\\tilde{c}$ often provides partial or heuristic reasoning that:\n\n* Increases the likelihood of predicting the correct answer (Property 2: Positive Learning Signal), and  \n* Is more in-distribution than expert CoTs under the current policy (Property 1).\n\nImportantly, ExPO does not blindly imitate these imperfect CoTs. The advantage-weighted update continues exploration around them, allowing the model to refine its reasoning policy through trial-and-error.\n\n**As shown in Sec. 4.2 and Table 2, this mechanism is highly effective on MATH level-5 questions where other methods fail.** ExPO enables bootstrapped learning and continued improvement, showing that imperfect CoTs can still serve as valuable scaffolds.\n\nWe agree this is an important point and will clarify the exploratory role of the advantage-weighted term in the final version.\n\n>**Q2**: “In … design?”\n\nWe thank the reviewer for the question and apologize for the lack of clarity in our description. In our implementation of ExP-GRPO, the guidance term—derived from self-explanations on the ground-truth answer—is incorporated into the overall objective via a scaling coefficient $\\beta$:\n\n$\n\\beta \\log \\pi_\\theta(\\tilde{c},a^{\\star} | q)\n$\n\nThis term is designed to provide an additional learning signal when the model’s own sampled responses fail to yield effective gradients, particularly in challenging reasoning settings. We fix $\\beta = 0.04$ in all reported experiments. To justify this design, we conducted an ablation study across multiple $\\beta$ values on the full MATH[2] training set (1 epoch):\n\n| $\\beta$ \\ Model | Qwen2.5-3B-Instruct | LLaMA-3.2-3B-Instruct |\n|-|-|-|\n| 0.5 | 47.3%     | 33.5%     |\n| 0.1 | 48.6%     | 34.8%     |\n| 0.08 | 50.5%     | 35.3%     |\n| **0.04** | **68.7%**     | **58.9%**     |\n| 0.01 | 60.4%     | 46.6%     |\n| 0.0 | 51.6%     | 44.3%     |\n\n(Pass@4 accuracy is reported on MATH[2] test set)\n\nWe observe a clear trend: the addition of the ExP-SFT term significantly boosts performance over the baseline ($\\beta = 0$), particularly in early-stage training where standard GRPO fails to provide meaningful updates. However, excessively large $\\beta$ values (e.g., $\\beta = 0.5$) degrade performance, likely due to over-reliance on imperfect guidance signals. In contrast, smaller $\\beta$ values (e.g., $\\beta = 0.01$) yield slower learning.\n\nThus, $\\beta = 0.04$ offers a favorable trade-off between initial learning efficiency and final performance. It balances the model’s ability to leverage the structured supervision from self-generated explanations without overwhelming the exploration driven by the advantage term in GRPO. These results empirically support our design choice and highlight the complementary role of the ExP-SFT term in bootstrapping reasoning capability where standard policy optimization struggles."}]

Author rebuttal:

Meta-review: The paper introduces ExPO (Self-Explanation Policy Optimization), a novel framework for improving large language model (LLM) reasoning without relying on costly expert-generated chain-of-thought (CoT) demonstrations. ExPO leverages self-generated explanations conditioned on ground-truth answers to create positive training samples that are both in-distribution (high probability under the current model policy) and increase the likelihood of correct answers (positive learning signal). The authors provide theoretical insights, supported by empirical results, demonstrating that ExPO outperforms expert CoT baselines on challenging mathematical reasoning tasks (e.g., MATH Level-5 and GSM8K) in both offline (DPO) and online (GRPO) settings. The method enhances training efficiency and final performance, particularly in scenarios where models initially struggle to generate correct responses.

Reviewers point serveral weakness:

Limited Generalizability: The evaluation focuses primarily on mathematical reasoning (MATH, GSM8K). While the authors argue that ExPO’s principles extend to other domains (e.g., code generation, commonsense QA), no experiments validate this claim, leaving generalizability uncertain.

Theoretical Assumptions: The gradient-alignment analysis (Lemma 1) relies on strong orthogonality and equal-norm assumptions, which may not hold in practice for large transformer models. While these are used for intuition, their limitations could be better clarified.

Clarity Issues: Some sections, particularly the methodology and theoretical analysis, are dense and lack intuitive examples, making them less accessible to readers without a strong RL background.

Potential for Bias: Conditioning on ground-truth answers risks introducing biases or hallucinations, especially if self-explanations contain incorrect reasoning steps. While the authors address this through advantage-weighted exploration, long-term effects need further exploration.


The authors’ rebuttal effectively addressed most concerns, particularly by adding experiments to validate scalability and clarifying theoretical assumptions. The generalizability concern remains partially unaddressed due to the lack of non-math experiments, but the authors’ arguments about applicability to other domains are compelling. The risk of bias/hallucination is mitigated by the proposed annealing schedule and exploration mechanisms, though long-term effects warrant further study. Overall, the paper’s strengths outweigh its weaknesses, justifying the acceptance for its innovative approach and strong empirical results.
Decision: Accept (poster)