Learning to Reason under Off-Policy Guidance
Authors: Jianhao Yan
Yafu Li
Zican Hu
Zhi Wang
Ganqu Cui
Xiaoye Qu
Yu Cheng
Yue Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning with verifiable rewards
off-policy guidance
Primary area: deep_learning

Abstract: Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(RLVR). 
However, existing RLVR approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. 
To address this issue, we introduce LUFFY (Learning to reason Under oFF-policY guidance), a framework that augments RLVR with off-policy reasoning traces. 
LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. 
Specifically, LUFFY combines the Mixed-Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. 
Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks.
Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.

Learning to Reason under Off-Policy Guidance
Jianhao Yan 123- Yafu Li क Zican Hu
Zhi Wang Ganqu Cui?
Xiaoye Qu²
Yu Cheng Yue Zhang
Zhejiang University? Shanghai Al Laboratory Westlake University" Nanjing University
The Chinese University of Hong Kong
Corresponding to: chengyu@cse. cuhk. edu. yue. zhang@wias. org.
Abstract
Recent advances in large reasoning models (LRMs) demonstrate that sophisti- cated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards (RLVR). However, existing RLVR approaches are inherently "on-policy", limiting learning to a model's own outputs and failing acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce LUFFY (Learning to reason Under oFF-policY guidance), framework that augments RLVR with off-policy reasoning traces. LUFFY dynam- ically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Specifically, LUFFY combines the Mixed- Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling avoid superficial and rigid imitation during mixed- policc training. Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribttion tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.
Introduction
Recent breakthroughs in large reasoning models, including OpenAl-ol [Л]. DeepSeek-RI (2), and Kimi-1.5 [3] have demonstrated remarkable capabilities in complex reasoning tasks. These models have shown unprecedented proficiency in generating extensive Chains-of-Thoure (CoT, 44) re-
sponses and exhibiting sophisticated behaviors, such as self-reflection and self-correction. Particularly noteworthy how these achievements have been realized through reinforcement learning with purely verifiable rewards (RLVR), as demonstrated by recent efforts including Deepseek RI [2] 77. The emergence of long CoT reasoning and self-reflection capabilities through such straightforward reward mechanisms, termed the "aha moment", represents a significant advancement in the field.
Nevertheless, the reinforcement learning methods behind the success have fundamental limitation worth highlighting: is inherently on-policy, constraining learning exclusively to the model's self- generated outputs through iterative trials and feedback cycles. Despite showing promising results, on-policy RL is bounded by the base LLM itself [8] [ In essence, reinforcement learning under this setting amplifies existing behaviors rather than introducing genuinely novel cognitive capacities. Recent study (10) corroborates this constraint, demonstrating that models like Llama 3.2 [TT] quickly reach performance plateaus under RL training precisely because they lack certain foundational
Co-first authors. Work was done during Jianhao Yan's internship Shanghai Laboratory. Yafu the Project Lead.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cognitive behaviors necessary for further advancement. This inherent limitation provokes critical questions about the effectiivnness and scope of RL for reasoning: How can we empower LLMs to acquire reasoning behaviors surpassing their initial cognitive boundaries?
Learning to reason Under oFF-polic guidance, addressing this issue by introducing external guidance from stronger policy (e.g.. from DeepSeek-R1). The strong policy serves as guidance for diverging the training trajectory beyond the limitations of the model's initial capabilities. Unlike on-policy RL, where the model can only learn from its own generations, our approach leverages off-policy learning to expose the model to reasoning patterns and cognitive structures that might otherwise remain inaccessible. This external guidance functions as a form of cognitive scaffolding. allowing the model to observe and internalize reasoning strategies from more capable teacher model, thereby expanding its reasoning repertoire beyond what self-im relr--eepeeeenttt alone could achieve.
In this paper, we introduce LUFFY:
In particular, LUFFY extends on GRPO 12 to Mixed-Policy GRPO by introducing a new off- policy objective with importance sampling to calibrate policy gradient, and combining off-policy reasoning traces with models' on-policy roll-outs during advantage computation, as illustrated in Figure I Intuitively, since off-policy traces consistently obtain positive rewards, LUFFY enables the model to selectively imitate these high-quality reasoning traces when its own roll-outs fail to achieve correctness, while preserving the capacity for self-driven exploration whenever its generated reasoning steps are successful. In this way, LUFFY achieves a dynamic and adaptive equilibrium between imitation and exploration. To avoid overly rapid convergence and entropy collapse, causing the model to latch onto superficial patterns rather than acquiring genuine reasoning capabilities, we further introduce policy shaping via regularized importance sampling, which amplifies learning signals for low oww-oobayillty yet crucial actions under off-policy guidance. This mechanism encourages the model to preserve exploration throughout training, ultimately enabling to internalize deeper and more generalizable reasoning behaviors.
LUFFY achieves significant improvements of +6.4 points on average compared with previous RLVR methods, across AIME24/25 [33, AMC [I3]. OlympiadBench [14]. Minerva [[]] and MATH- 500 [16] benchmarks, establishing the effectiveness of off-policy learning in RLVR paradigms. Moreover, LUFFY demonstrates superior generalization capability, i.e., an advantage of over +6.2 points on average, on cuttof-distributionn tasks, where other off- off-policy methods fall short. Critically, we show that LUFFY successfully trains weak foundation models, i.e., LLaMA3.1 1-... while on- policy RL fails, providing evidence of LUFFY transcending the limitation of model capacity. Our in-depth analyses demonstrate that LUFFY encourages the model to imitate high-quality reasoning traces while maintaining exploration of its own sampling space, resulting in more reliable and generalizable reasoning capabilities.
Our contributions can be summarized as:
We introduce LUFFY, an approach that incorporates off-policy guidance into GRPO and integrates policy shaping through regularized importance sampling to address entropy collapse, effectively transcending the limitations of on-policy RL. (Sec.
We empirically demonstrate LUFFY's effectiveness across various foundation models, achieving an average gain of +6.4 points across six math benchmarks and +6.2 points on fifto---ditributionn tasks against previous RLVR methods, establishing a new state-of-the-art on RLVR with Qwen2.5-Math-1
(Sec. 5.T)
We demonstrate that LUFFY successfully trains weaker foundation models where On-Policy RL fails. Specifically, while On-Policy RL, can only train Llama3.1 on simplified datasets, LUFFY effectively trains these models across varying difficulty levels, overcoming capability- limitations (Sec. 5.2)
2 Reinforcement Learning with Verifiable Rewards
Verifiable Reward Function. The verifiable reward emphasizes the comparison between the extracted answer from the models' output and the predefined golden answer. For instance, the model instructed output the final answer certain format, egg. \boxed(); and a regex function used to extract the answer from \boxed((). Formally, given model's output to question q. the

---

Solution
Advontage
Probability
A1
Policy Model
F2
Verifier & Group Compute
A2
q
&
Tw--1
Ag-1
Off-policy
Model
Tti
A
f(TT)
Palicy Shaping
Figure 1: Overview: LUFFY integrates off-policy reasoning traces into reinforcement learning by combining them with on-policy rollouts. Policy shaping emphasizes low-probability but crucial actions, enabling a balance between imitation and exploration for more generalizable reasoning.
reward is defined by,
{1 if outputs the correct final answer to otherwise
R(T)
This reward design avoids the risk of reward hacking successful scaling of RL training 2
17
18
19
to a great extent and thus leads to
Group Relative Policy Optimization (GRPO). GRPO 12 shows exceptional performance in various tasks, especially enable effective scaling within the RLVR paradigm 20 6
uses the reward scores of N sampled solutions from query to estimate the advantage and thus remove the need for an additional value model. Formally, we denote the policy model before and after the update as Tee and Ne. where both represent probability distributions over possible actions/tokens at each position. Given question q. set sampled solutions т; generated by Te.. and the reward function R(-), the advantage A; of each in GRPO is computed by normalized rewards inside the group.
4 R(T.) mean( (R(T.) T. Ta (T), 1,,,...,NNN
(2)
std({ (R(T.) Ti ~ New (T),
Then, the RL. objective is inherited from the clipped RL. objective proposed by PPO 21
N Imd
JGRPO =
CLIP(r..(0), - -3 DDKL metd
E
TT
where (0) = TT(Tr, Ti, / Te (T., Ti.<)) is a importance sampling term to calibrate the gradient based on policy gradient theory [22] as the solutions are generated by wewe instead of π..
The KL. divergence IDR and the clipped surrogate objective CLIP(r, A,c) min(r- A, clip(r;
1 c) A] empirically ensures that the current policy mo is within the trust region (3 of the old policy Thw We loosely categorize this method as On-Policy RI., indicating that the model is optimized using samples drawn from distributions closely aligned with its current policy. Nevertheless, recent have increasingly omitted the KL divergence term, making these methods somewhat less "On-Policy".
practices 224 25 7
Learning to Reason under Off-Policy Guidance
To facilitate exploration beyond the model's own capabilities, we incorporate off-policy guidance, i... off-the-sheif reasoning trajectories generated by a stronger reasoning model such as Deepseek R1, into the RLVR learning. We expect the model to learn generalizable knowledge from off- off-policy beyond superficial imitation and maintain effective and efficient exploration as in RLVR training. In the following sections, we introduce LUFFY: mixed-polieg GRPO ($ 3.1b with policy shaping (3..).. The former one adaptively integrates off-policy trajectories into advantage estimation while the latter encourages continuous exploration throughout training.
3

---

3.1 Mixed-Policy GRPO
We incorporate off off-po rollouts in GRPO by adding them directly to the group of on-policy rollouts generated by the model itself. Provided an off- aff-policy distribution Te, this would affect the advantage computations in the following way.
A. R(::) mean Gon U Gorr
Std(Gon U Gan)
where Gon (R(T:) Ti (e 1,2,.... ...,, Non and Gort = {[((;)
(T(T),
No....,,oo]]] As the quality of off-policy rollouts is high (yielding high rewards), this group computation naturally assigns higher advantage to off- off-policy rollouts when the model struggles to generate correct solutions independently. Once the model begins producing successful reasoning traces, on-policy rollouts take precedence, thereby encouraging self-driven exploration.
Extending from the GRPO objective (Eq- 3). introduce the off-policy objective incorporate off-policy rollouts. Similar to GRPO and PPO [21 12), we introduce an importance sampling term
(0,((0,0) x) to([Talla Tj. Tt,<t)/ ///////Taaa (Tzelg, Tj,<t) calibrate gradient estimates 22
We refer this
approach as Mixed-Policy GRPO:
Nail
Nen Ird
1
IMsied (0)
C)
CLPP(ru(O))
Z
(5)
off aolppoicy ubjective
policy objective
where (T T3 <t// o (Tyelg Nat |ry1 Emn |T: is the normalization factor.
Ts,st) and ro,)()) = To
Th (Tr,elg Ti,
Tr,
2 = Σド=
The newly introduced importance sampling term f3.i contrasts with the importance sampling fi, used in on-policy RL (Eq. where the denominator corresponds to the pre-update roll-out model policy Te Since the divergence between सe and The typically much smaller than that between Te and the off-policy policy TG. the off-policy importance sampling ratio fz.e tends to be smaller, serving to calibrate gradient estimates from a distinct distribution.
Based the theoretical analysis of stochastic gradient descent in nonconvex optimization 26 policy gradient estimator in Eq. (5) stabilizes and converges to stationary point, and the convergence rate is O(1/VK) where K is the total number of iterations. The proof can be found in Appendix B.
give a convergence analysis in Theorem [ to show that our importance-weighee
Theorem 1. Suppose the objective function of the policy gradient algorithm J E In where Jn is the class of finite-sum Lipschitz smooth functions, has g-bounded gradients. and the importance weight w = /T/ a is clipped to be bounded by [w, W. Let 0g = aa- = = e where e 2( J(09) and 0* is an optimal solution. Then, the iterates of our algorithm in Eq. (5) satisfy:
Lo w
2(J(0-) - ((0))))) Lt
min 0S
Bilv
Kw
The importance sampling ratio in off-policy learning typically involves Te. representing the behavior policy's probability in off-policy trajectories [27]. Theoretically, our derivations and guarantees hold for any well-defined मę distribution. In practice. to facilitate direct integration of high-quality demonstrations from large, powerful models (e.g., DeepSeek- R1), we adopt T = 1 for computational efficiency. This practical choice not only avoids the complexity caused by different tokenization between the on-policy and off- off-policy models. It also facilitates the easy incorporation of off the-- datasets without recomputation of Tæ. as well as preserving theoretical guarantees. We omit the clip operation for the off-policy rollouts, as the clip operation will be imbalanced when J =
3.2 Policy Shaping via Regularized Importance Sampling
While Mixed-Policy GRPO incorporates off-policy guidance successfully via group computation and importance sampling, practical challenge emerges: Mixed-Policy GRPO accelerates

---

Eniropy during Training
Weighting
Gradient Weighting
On-policy
0.5
Mixni-policy
Minani Mollei-polley + Shapking
0.4
Entropy
Weights
10.6-
Wrights
0.15
0.3
0.4
0.30
0.2
10:2
0.06
On polke//Mised
Mined policy
policy Shappng
0.0
0.00
100
200
300
400
300
0.3
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
Probability
Probability
Figure 2: Left: generation entropy during training; Middle: shaping function value w.r.t. action probability; Right: gradient weights w.r.t action probability.
convergence but significantly reduces exploration (Figure 2 left). Specifically, entropy collapses
much faster than on-policy RL, indicating increasingly deterministic rollouts and diminished capacity for exploring diverse reasoming trajectories.
This originates from the "hacking" of the Hixed-Policy objective. When combining both learning off- off-policy and on-policy signals, the model tends to quickly converge toward reinforcing off-policy tokens that are also likely in the on-policy my distribution, and ignoring off-policy tokens that are deviated from the model's original policy, i.e., low-probability tokens that may represent essential reasoning capabilities the model has yet to acquire. We empirically analyze this issue detail in Section 5.3
To address this issue, introduce policy shaping via regularized importance sampling, technique that re- weights the gradient off-policy distributions to enhance learning from low-probabilit tokens. In particular, approach replaces the importance sampling ratio ( Φ) with f( (0, Φ)), where f(.) represents a transformation function that alters the dynamics between off-policy and on on-policy distributions, thereby increasing gradient emphasis tokens with low probability in the model's standard distribution. Recall that we omit the clip operation for the off-policy rollouts. The loss function with policy shaping can be written below:
Now Ins
Now ΣΣ (0)
JSHAPING (0)
((0,,(0,0)) ÀG +
(6)
To further illustrate the meaning of shaping function f, we derive the gradient of off-policy objective,
V OrT))))))c.oor(( = EE (f
To
Vo log To Ag.
importance sampling
We write [TT), Tj,<e) as n for simplicity. From Eq. 7 we can see ('(ma) is weighting function of the gradient. The vanilla mixed -olicy GRPO can be regarded as using a linear shaping function, i... f(T)) ग, where the original importance sampling ratio тe TT is applied.
We decompose the log probability and derive the gradient on each output logit (action):
JJSHAPING (F) ETiTO
[((TT) To [A(T),0 = Tz.e) - -me] A,]
(T,,))
(8)
(SHAPING-OFF(())
S Bynno [Is xe - To) -A,1]=
(M,(T),)
where Ty,r is any possible action/token on in the action space at the j-th trajectory and t-th position, and (T) denotes the logits of that action. The identity case represents the gradient when the action the off- off-policy action r r', which elevates the probability of predicting the off policy action. From Eq. 8 if ( тe and thus f'(T))
1, we can see that the scale of gradient is upper- uppeerbounded
by To n.), leading to small values when ue + and To > 1. We opt for ((x) = x/ (r + +7) as
our shaping function (middle part of Figure 2) where is set as 0.1. As shown in Figure (Right), the shaping function reweights the gradients to assign more importance
owwprobabillity actions,
thereby improving learning from unfamiliar yet effective decisions from the off-policy traces.
5

---

Further, we provide an informal analysis on the variance of our importance weights regularized by f()) in Appendix B.2 With a first-order approximation and a special case derivation, we show that smaller variance can be achieved for the sampling weights, thus proving more stable training for leveraging off- off-policy guidance.
4 Experimental Setup
[28] of which the prompts are collected from NuminaMath 1.5 [29], and the off-policy reasoning traces are generated by Deepseek-R1 R1 (2). We use the default subset, which contains 94k prompts, and we filter out
Dataset Construction.
Our training set is a subset of OpenRI
generations that are longer than 8192 tokens and those that are verified wrong by Math-verify) resulting in 45k prompts and off policy reasoning traces.
RL Practice. We remove the KI. loss term by setting 3 = 0 and set the entropy loss coefficient to 0.01. Following Dr. (R. we remove the length normalization and standard error normalization of GRPO loss (3 for all experiments. For policy shaping, we empirically set the as 0.1 and study the value of y in Appendix E..4 Our rollout batch size is 128, and the update batch size is 64. We use 8 rollouts per prompt. Specifically, for on-policy RL, we use 8 on-policy rollouts. For our methods, we use off-policy and 7 on-policy rollouts to ensure fairness. We use lemperature= for rollout generation. We use Math- Verify as our reward function and include no format or length reward. We use Swwe2..5-Math-7- [30] by default, following previous work [24] 5] In addition, we extend
LUFFY to Qwen2.5-Math- 1.5B [30] and @0wen..5-Instructt
31, and LLaMA 3.1-8B 32)
Evaluation. For evaluation, we mainly focus on six widely used math reasoning benchmarks, including AIME 2024, AIME 2025, AMC [13], Minerva [15 OlympiadBench [14], and MATH- 500 [T6]. For AIME 2024, AIME 2025 and AMC, we report avg @ 32 as the test set is relatively small, and for the other three benchmarks, we report pass @1 1. As our RL training mainly focuses math reasoning, we further validate the generalization capability on three namely ARC-c denoted as GPQA"), and MMLU-Pro [35 (Reasoning- focused Questions from Academic Exams and Textbooks). We shuffle the multiple- choice options to avoid contamination. For testing, the temperature is set as 0.6.
-iittiibiittttionn benchmarks, Domain Reasoning), GPQA-diamont (Science Graduate Knowledge, ceasonnig-focused
For RLVR methods, we consider the following methods: (1) Simple-RL RI. (5): training from Qwen2.5-Math-78 using rule-based reward; (2) Oat-Zero [ō]: training from Qwen2.5- Math-7B and rule-based reward, proposing to remove the standard deviation in GRPO advantage computation and token-level normalization in policy loss computation; (3) PRIME-Zero (24]: using
Baseline Methods.
policy rollouts and outcome labels through implict process rewards; (4) OpenReasonerZenn [7]: recent open soun-source implementation of RLVR methods. Except RLVR approaches from previous work, consider two kinds of baselines with our setting (1) On-Policy RL- we train on-policy RL within RLVR paradigm using GRPO with the same reward and data. (2) Alternative Methods to Incorporate Off-Policy Guidance- We consider three methods, namely SFT, we train the model with the same prompts and reasoning traces as LUFFY using SFT; RL w/ SFT Loss, using SFT loss during RL training; SFT + RL, two-stage training that continues RL training after SFT. For detailed setup for training these methods, we refer readers to Appendix
Experimental Results
5.1 Main Results
SOTA performance on RLVR with Qwen2.5-Math-7B. Our main results are presented Ta- We first compare LUFFY against other RLVR methods and our RLVR replication. All prior methods are built upon Qwen2.5-Math.
base models, differing in dataset composition (source and difficulty) and optimization strategies, e.g., removing length and standard error normalization [6] or incorporating process-level rewards [24]. Evaluated on six challenging competition-
benchmarks, LUFFY achieves an average score of 50.1, significantly outperforming existing RLVR methods by substantial margin of +6.4 points, establishing a new sfate-of-the-art Notably, while LUFFY exhibits comparable performance in AIME 24, it demonstrates a significantly greater advantage the newly
-Math- 220k

---

Table I: Overall in-distribution and out-of-distribution performance based Qwen2.5-Math-7B.
We compare with the following baselines: (1) Qwen2.5-Math- IB-Instruct (Qwen-Instruct), (2) prior RLVR approaches, (3) replication of on-policy RL, and (4) alternative off off-policy learning methods. All models are evaluated under a unified setting. LUFFY denotes training with extra steps (Table 2). Bold and underline indicate the best and second-best results, respectively. represents significantly better than baselines (p < 0.05).
In-Distribution Performance
Out-of- Duttof--Ditributtin Performance
Model
AIME 24/25
AMC MATHI-50 Minerva Olympiad
Avg.
ARC-c GPQA*
MMLU-Pro
Avg.
Qwen-Base
30
11.5/4.9
31.3
43.6
7.4
15.6
19.0
18.2
11.1
16.9
15.4
Qwen-Instruct
30
12.5/10.2
48.5
80.4
32.7
41.0
37.6
70.3
24.7
34.1
43.0
Previous RLVR methods
SimpleRL-Zeee
27.0/6.8
54.9
25.0
34.7
30.2
23.2
34.5
29.3
OpenReasoner-Zeee Zero
16.5/15.0
52.1
82.4
33.1
47.1
41.0
66.2
29.8
58.7
51.6
PRIME-Zero
17.0/12.8
54.0
39.0
40.3
73.3
32.7
Oat-Zero 6
33.4/119
61.2
78.0
34.6
43.4
43.7
70.1
23.7
41.7
45.2
Our Ou-policy RLVR Replication
On-Policy RL
25.1/15.3
62.0
84.4
39.3
46.8
45.5
82.3
40.4
49.3
57.3
Alternative Off- polico Learning Methods
SPT
22.2/22.3
52.8
82.6
40.8
43.7
44.1
75.2
24.7
42.7
47.5
RL w/ SFT Loss
19.5/16.4
49.7
80.4
34,9
39.4
40.1
71.2
23.7
43.2
46.0
SFT+RL
25.8/23.1
62.7
87.2
39.7
50.4
72.4
24.2
37.7
44.8
Our Methods
LUFFY LUFFY+
29.4/23.1
65.6
87.6
37.5
57.2
50.1
80.5
39.9
53.0
57.8*
30.7/22.5
66.2
86.8
41.2
55.3
50.4
81.8
49.0
54.7
61.8*
released AIME 25 test set (+8.1), demonstrating its generalization to internalize nuanced reasoning behaviors from off-policy traces. Compared to On-Policy RL, LUFFY improves performance by +4.6 points on average, demonstrating the benefit of integrating high-quality off-policy traces.
Regarding disto--distriuution performance, LUFFY also demonstrates strong performance gain. Over three challenging -it--ffiitriiuttion benchmarks, LUFFY achieves an average score of 57.8 and outperforms the best RLVR method OpenReasoner- for +6.2 points, These findings underscore the effectiveness of LUFFY in leveraging off-policy reasoning guidance for enhanced generalization across diverse, out-of-distribution tasks.
Comparing against other Off-Policy Learn- ing Methods. Comparing LUFFY against alter- native methods to incorporate Off-Policy Guid- ance, we can see that LUFFY beats all three off- policy baselines in in-distribution math reason- ing tasks and achieves substantial improvements over OOD tasks (+10.3 points). Compared to SFT+RL, LUFFY is advantageous in both in-distribution tasks (+1.9 points) and out-of- distribution tasks (+16.1), with only 59% GPU hours and much less off-policy data usage (Table 2). The additional GPU hours in SFT+RL and RL w/ SFT Loss are largely attributed to ex- cessively long generations induced by rigid im- itation from SFT (Appendix F.), which substan- tially increase the computational overhead dur- ing the RL roll-out stage. With matching GPU hours, LUFFY! further enlarges the advantage, providing a more robust and effective alternative for distilling knowledge from stronger I.MMs, except for supervised fine-tuning 2 36 37). In
Table Comparison of resource requirements between LUFFY and other off-policy methods.
Model
GPU Hours
Data Usage (On/Off)
LUFFY
77 8
64K x /
LUFFY!
130 x
110K x 7/110K
SFT
24 8
64K
RL w/ SFT Loss
133 x
64K X 7/K
SFT+RL
130 x 8
64K x 1/5K
Intrect
SFT
On pobey RC
LUTTY
Qwen2.
Qwen2. 20122.5-788 Justruut
45.7
3um.
Avercage Score
15..0
Figure 3: Average performance on six mathe- matical reasoning benchmarks of LUFFY on different backbones (Details in Appendix F.2
7

---

On-policy RL
LUFFY
Oft-policy Reasoning Traces
Outeome Training Rewards
Mall
wwll
Response Length
Contimmous Exploration
M
Entropy
Tmitatiing Offipaalicy Traees
Adapting External Guidance
Steps
Steps
Steps
Figure 5: Training dynamics of LUFFY compared with on on-policy RL. Left: outcome training rewards; Middle: generation length; Right: generation entropy.
particular, we notice the SFT training causes the model to learn superficial and rigid imitation of off-policy traces, and costs the model on out-of-distribution performance, while LUFFY selectively and strategically learn from off-policy traces (Sec. 5.3 and App. to enhance its own policy rollouts.
Extending LUFFY to More Models. We further investigate whether LUFFY can be applied to small models, instruction-tumne models, weak models. To answer this question, train LUFFY on three more models, i.e., 0wen2.5-Math-11 1.5B (small models), Owwen..5-Instruct-77 (instruction-tune models), and LLaMA-33 1-8B (weak models), and compare with their respective Instruct models (black bar in Figure 3). LUFFY achieves consistent and substantial improvements, surpassing both SFT and On On-Policy RL for all three models, demonstrating the general applicability of LUFFY. Specifically, LUFFY improves over on-policy RL for +8.0 points on Qwen2.5-Math-1.55 +3.6 points 1lama-3.1-8B, and +5.5 points on Qwen2.55-Insruuct-l
5.2 LUFFY Succeeds Where On-Policy Fails
Easy Training Set
Training Rewards
More interestingly. observe that LUFFY can successfully train models in scenarios where on- policy RLVR fails. We conduct experiments using LLaMA-3.1-40 1-8-- on two subsets of vary- ing difficulty (Easy and Hard), with details pro- vided in Appendix C As shown in Figure [4] on-policy reinforcement learning performs well on the Easy subset but fails on the Hard subset, where training rewards collapse zero, since on-policy rollouts struggle to obtain positive feedback signals. In contrast, LUFFY achieves stable reward improvements on both datasets, highlighting its robustness and its ability to over- come limitations imposed by model capacity.
Lluna- Ou- Policy Easy Llams-LUFFY-EAL
Steps
Hard Training Set.
Training Rewards
0.2
0.1
MJama-On- Caana-Or-Poliice Oo-Pooicg-HHar
Liuna- Llnna-LUFF' Hard
0.0
100
200
300
400
500
Steps
Figure 4: Training rewards of LLaMA-3.1 8B on the Easy and Hard training set.
5.3 Training Dynamics
Strategically Learning from Guidance. Figure 5 illustrates the training dynamics regarding training rewards, generation length and entropy for On-Policy RL and LUFFY. Initially, LUFFY primarily imitates off-policy trajectories, as indicated by the increasing generation length gradually aligning with the off-policy reasoning traces (middle part of Figure 5). this early stage, imitation dominates, causing an initial performance dip (left part of Figure 5) as the model adjusts to external guidance and potentially sophisticated cognitive behaviors [8]. As training progresses, on on-policy rollouts gradually become more prominent, fostering independent exploration within the model's own sampling space while effectively retaining insights gained from off-policy demonstrations. This guided exploration brings growing advantages (training rewards) over On-Policy RL. uently, LUFFY achieves a dynamic balance between imitation and exploration, leading to more effective off-policy learning (Section These results highlight that LUFFY selectively adopts valuable reasoning patterns

---

rather than blindly imitating off off-pooicy traces. Such strategic oof-po learning is further evidenced in reasoning behaviors during inference, such as generation length and exploration (Appendix (F).
Maintaining Exploration. Figure P (Right) illustrates that LUFFY consistently sustains higher entropy compared On-Policy RL throughout the entire training process. Specifically, the gen- eration entropy of On-Policy RL rapidly converges to nearly zero after approximately 200 steps, indicating a highly deterministic policy with limited exploration potential. Conversely, the elevated entropy observed in LUFFY allows continuous exploration of less confident yet potentially superior policies, facilitating the discovery and learning of novel cognitive behaviors. Interestingly, we observe entropy fluctuations and even occasional increases, such as between steps 200 and 250, reflecting ongoing exploration of low-probability but crucial actions, also referred to as pivotal lokens 33 25 This strategic exploration enables the model to escape local optima, thus improving its convergence towards more globally optimal solutions.
5.4 Policy Shaping Encourages Continuous Exploration
Figure illustrates the validation performance over the course of training, shedding light on the impact of pol- icy shaping. Mixed-Policy achieves rapid early gains, significantly outperforming On-Policy RL at the start. However, its performance soon plateaus and eventually converges with On-Policy RL, whereas LUFFY continues to improve steadily. These trends consistent with our earlier analysis of entropy collapse (Section 3.2). under- scoring the role of policy shaping larizer that prevents premature convergence and sustains performance gains in later stages of training.
Validation Accuracy (%%
201
10
Out
0.5
Entropy
Miii RL.
Maay Suasix (LUFF))
effective regu-
Steps
6 Related Work
Figure 6: Effects of policy shaping.
RL for LRMs Recent advances have demonstrated remarkable progress in enhancing L.MSs' reasoning capabilities through RL approaches [2] 1 3 40). including DeepSeek-R1, OpenAl- 01, and Kimi-1.5. Subsequent work systematically investigate RL with purely verifiable rewards (RLVR) [5] 7 24 6 41). providing insights into how this approach enables complex reasoning, Various advances have been proposed for reasoning enhancement. Test-time adaptation mecha- nisms (36 42 demonstrate potential in dynamic optimization, but remain bounded by inherent model knowledge. While structured reasoning approaches
have demonstrated that complex reasoming capabilities can emerge from strategically designed prompting and training strategies. The development of RL optimization techniques [46] 47
43
37
6 has contributed novel training paradigms and optimization objectives that specifically target the enhancement of reasoning capabilities. How- ever, recent studies X 9 demonstrate that on-policy learning limited by vast exploration space and primarily amplifies existing behaviors. Existing approaches optimize within model boundaries rather than expanding reasoning horizons. In the meantime, SFT replaces old capabilities [49] but might not generalize to other domains (50]. Our approach leverages off- off-policy reasoning traces to transcend these cognitive constraints while preserving self-driven exploration capabilities.
48
On-Policy and Off-Policy RL Reinforcement learning algorithms are fundamentally distinguished by their approach to experience utilization during policy optimization. On-policy methods (e.g.. TRPO [23]. A2C/A3C [51] PPO (11) strictly update using trajectories from the current policy, ensuring training stability but potentially constraining the exploration space. In contrast, off-policy algorithms (e.g.. DQN ID3 (53). SAC (54)) leverage experiences from diverse policies, offer- ing superior sample efficiency while introducing optimization challenges due to distribution shift. Extending to LLM training, on-policy methods are more commonly adopted, with approaches like GRPO 12. REINFORCE (55] and PPO 21 demonstrating strong performance through various op- timization techniques. PRIME (2 and NFT (56] models the implicit policy to utilize self- generated answers. Meanwhile, off-policy approaches like DPO [57 offer alternative optimization frameworks by reformulating preference learning as classification. To leverage advantages from both paradigms, our work bridges these approaches through policy shaping with regularized importance sampling, effectively combining on-policy optimization with off off-policy guidance.
9

---

7 Conclusion
We presented LUFFY, simple yet powerful framework that integrates policy reasoning guidance into the RLVR paradigm. By dynamically balancing imitation and exploration, LUFFY effectively leverages external reasoning traces without sacrificing the model's s ability to discover novel solutions. Our method outperforms strong baselines across competitive math benchmarks and generalizes robustly to out-of- -iitrii-ittiittionn tasks, surpassing both on on-policy RLVR and off-policy baselines. These results highlight the promise of off-policy learning as a scalable and principled path toward building more general, capable, and self-improving reasoning models. Future work may focus on extending LUFFY to broader domains or modalities (58] and further refining policy shaping to maximize exploration under off- aff-policy guidance.
Acknowledgement
This publication has emanated from research conducted with the financial support of the National Key RD program of China (grant No. No.20222YF0204900 and the National Natural Science Foundation of China Key Program under Grant Number 62336006. This paper focuses on learning to reason from off-policy, which enhances the generalization compared previous off-policy methods and in-domain reasoning capability compared to on on-policy RL.
10

---

References
[1] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
[2] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.11244: 2025.
[3] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
[4] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[5] Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning both effective and efficient.
notion. site/ http:ssmmmeeerrrre reasonn 2025, Notion Blog.
[6] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2013 2025.
[7] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero. An open source approach to scaling up reinforcement learning on the base model, 2025.
[8] Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining, 2025.
[9] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity 1lms beyond the base model?, 2025.
[10] Kanishk Gandhi. Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025.
[11] Meta AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, September 2024. 15 minute read.
[12] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[13] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. https
//huggingface... www.aaaaaeetaaee http//uuiinnnaaaaaa 2024. Hugging Face repository, 13:9.
[14] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3828-3850, 2024.
[15] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solas et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[16] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0388 2021.

---

[17] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, arXiv preprint (XXv::666600665, 2016.
[18] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-
9471, 2022.
[19] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866, PMLR, 2023.
[20] Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (0..1707700.0 06347, 2017.
[22] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[23] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897.
PMLR, 2015.
[24] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv 2025.
preprint arXiv:2502.0144
[25] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Iim reinforcement learning system at scale, 2025.
[26] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In ICML, pages 314-323, 2016.
[27] Wenjia Meng, Qian Zheng, Gang Pan, and Yilong Yin. Off-policy proximal policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9162-9170,
2023.
[28] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025.
[29] Jia II, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath.
htt.p/ggg.n....aeoooooooo/////aa (htphais11..itttttss////iiaaaaa com/ project-numina/ wwwp:aaaaaaaaaaaaaaeaeem http:a/wwww.........peeaapaaaaaa 2024.
dataset.
[30] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-man -mth technical report: Toward mathematical expert model via self--mprrvemmen, 2024.
[31] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint
15X1:2442..5115, 2024.
[32] Meta Team. The llama 3 herd of models, 2024.
[33] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv:1803.05457v.2 2018.
12

---

[34] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-prost proof q&a benchmark, In First Conference on Language Modeling, 2024.
[35] Yubo Wang, Xueguang Ma. Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 2024.
[36] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint 0rXiv:2501.19393 2025.
[37] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chem, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.0333 233873337 2025.
[38] Xiao Hu, Xingyu Lu, Liyuan Mao, YiFan Zhang, Tianke Zhang, Bin Wen, Fan Yang. Tingting Gao, and Guorui Zhou. Why distillation can outperform zero-rl: The role of flexible reasoning. 2025.
[39] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024.
[40] Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond, 2025.
[41] Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy with optimal reward baseline, 2025.
[42] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuckai Zhu, Biqing Qi. Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504. 16084, 2025.
[43] Bo Pang. Hanze Dong, Jiacheng Xu, Silvio Savarese. Yingbo Zhou, and Caiming Xiong. Bolt: Bootstrap long chain-of Mhain-of-thowe of-thought in language models without distillation. arXiv preprint arXiv:2502.0:335 033800 2025.
[44] Mchdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.0511 05885 2025.
[45] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint MXXiv:2505.03335 2025.
[46] Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Rengian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, and Yuting Liu. Trust region preference approximation: simple and stable reinforcement learning algorithm for Imm reasoning. arXiv preprint arXiv:2.04. 045544, 2025.
[47] Liang Wen, Yunke Cai, Fenrui Xiao, Xin Hc, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503. 10460, 2025.
[48] Wang Yang, Hongye Jin, Jingfeng Yang, Vipin Chaudhary, and Xiaotian Han. Thinking preference optimization. arXiv preprint arXiv:2502.13172 2025.
[49] Neel Rajani, Aryo Pradipta Gema, Seraphina Goldfarb-Tarme Tarrant, and Ivan Titov. Scalpel vs. hammer: Grpo amplifies existing capabilities, sft replaces them, 2025.
[50] Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general IIm capabilities? understanding transferability of Ilm reasoning, 2025.
[51] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PmL.R, 2016.
13

---

[52] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 131.5563 2013.
[53] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596.
PMLR, 2018.
[54] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018.
[55] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[56] Huayu Chen, Kaiwen Zheng, Qinsheng Zhang, Ganqu Cui, Yin Cui, Haotian Ye, Tsung-Ys Yi Lin, Ming- Yu Liu, Jun Zhu, and Haoxiang Wang. Bridging supervised learning and reinforcement learning in math reasoning. arXiv preprint arXiv:2505. /8/16, 2025.
[57] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
[58] Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and
Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint ar arXiv:2501. .05444, 2025.
[59] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprini
1atp::11220000000 2017.
[60] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 2020.
[61] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press,
2 edition, 2018.
[62] Philipp Koehn. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empiricul methods in natural language processing, pages 388-395, 2004.
[63] Kishore Papineni, Salim Roukas, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311 11,3318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[64] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or RL is suboptimal. In ICLR 2025 Workshop: VerifAl: Al Verification in the Wild, 2025.
[65] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xic, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: comparative study of foundation model post-training, 2025.
[66] Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1 rike reasoning large vision- vision-language models. htttps://githee www/ VLAAVVLAA Thinking 2025.
14

---

Appendix
Limitations
15
Theoretical Proof
15
Convergence Rate of the Importance- Importance-Weighted Policy Gradient Estimator
15
B.2
Informal Analysis on Variance of Regularized Importance Sampling
17
Experimental Details
18
Detailed Setup
18
System Prompt
19
C,3
Significance Test
20
Case Study
20
Additional Results
20
E.1
Removing On On-policy Clip
20
Extension More Models
................................
20
Ablation Study
20
E.4
Hyperparameter Study
21
Analysis
F.1
LUFFY Learns Strategically from Off-Policy Traces, While SFT Imitates Rigidly
F2
LUFFY Can Explore During Test-time While SFT Cannot.
22
A
Limitations
Firstly, we mainly focus on math reasoning RL training that has the golden answer and support verifiable rewards. Other tasks, especially ones without verifiable rewards, are not discussed in our manuscript. For instance, in open-ended generation tasks, the reward may be evaluated by an in-domain reward model. Secondly, we focus on 7B and smaller foundation models, due to the limited computational resources. Scaling LUFFY to larger models could be an interesting topic, given scaling law [59 60 is one of the most powerful principles the area of large language models. Finally, as we are the first to incorporate off- off-policy guidance into the RLVR paradigm, we are limited to only including one off-policy trajectory per question, and find that one trajectory is already strong. However, extending off-policy guidance to multiple trajectories and multiple teachers could help the performance even further.
B Theoretical Proof
B.1 Convergence Rate of the Importance--Wighhted Policy Gradient Estimator
We study the nonconvex finite-sum problems of the form
72 1
max J(0) OER
71
where both J and J; (i E (n]) may be nonconvex. We denote the class of such finite-sum Lipschitz smooth functions by Jn. Here, we optimize functions in J, our sepoortance-eiighee policy gradient estimator.
15

---

The vanilla policy gradient algorithm maximizes the expected advantage function (equivalent to minimizing the negative expected advantage function) as
12 [A(r)),
max
J(0) E, [A(T))
(10)
OER
Tt
According to the Policy Gradient Theorem 61 the vanilla policy gradient estimator has the following form:
1 V log mT( A(T.)).
( En [V (e() A(T) ~
(11)
where VJ(()) to denote VoJ(0) for simplicity. Our algorithm draws samples from another behavior policy Tæ. resulting in an importance-wiighted policy gradient estimator as
no T2)
J(0)) By Br
Te))
AA(T) ~
jay V.J(
(12)
(T,)))
where w/
(m())) (n)
is the importance weight assigned sample i.
Let on denote the learning rate at iteration k. and Wig be the instance weight assigned to sample i by
algorithm. By stochastic gradient ascent, our algorithm has the following update rule:
0 g++ = 0 + W. V.., (0*), E [n]]
(13)
Definition 1. For J E Jn, our algorithm takes an index E [n] and a point Rd, and returns the pair (J,(O), (O)))
Definition 2. We say J Rd +R is Lipschitz smooth (L-smooth) if there is constant L such that (VOO)- s L|I0 0||,
V0, 0 Rd.
(14)
Definition 3. A point is called c-accurate if [ < c. A stochastic iterative algorithm is said to achieve e-accuracy stochasticity of the algorithm.
iterations if J )12 < e, where the expectation is over the
Definition 4. We say J E Jn. has a-bounded gradients if J(())II S for all i E [n]] and O E Rd.
Definition We say the positive instunce weight w our algorithm is bounded if there exist constants and w such that w, S for all i [n].
Theorem 1. Suppose the objective function of the policy gradient algorithm J E Tno where Jn is the class of finite-sum sum Lipschitz smooth functions, has g-bounded gradients, and the importance weight
w. = /o/ n is clipped be bounded by [w, w. Let og = a = / where e e= 2[ ((0))) and 0* is optimal solution. Then, the iterates of our algorithm in Eq. (??) satisfy:
Lo wm
2(J(0*) - JJ
min DSkck-1
6.
Kw
Proof. According to the Lipschitz continuity of V.J, the iterates of our algorithm satisfy the following bound:
( 1 gh) T. 110tt1 1x1
[[[(0++1]] (()(e)) + +( (V.((*)),
(15)
After substituting [3 into 15
we have:
Lafug
E[J(0++1))
B(J(0*) t..
E V ( (0+)
2
La 0²
E(J(0*) , BJ
(16)
2
16

---

The first inequality follows from the unbiasedness of the stochastic gradient B... (0*)
V.(O"). The second inequality uses the assumption on gradient boundedness in Definition Re-arranging (16) we obtain
([J(@)t)--J(0)) - ) + Lag 0².
V ((* )|
(17)
2
Summing
from k = 0 to K - and using that og is fixed we obtain
K-1
min B|
[[[
E
1
K
Lowi
B|J(O f(0h)]
o²
OWK
K
2
k=0
0
LaW
(J(OK) -J(0"))
o²
Kow
2
Low o²
(J(0*) ((()))
Kow
2
Lew
(J(0*) )(0))
(18)
CI
The first step holds because the minimum is less than the average. The second step is obtained from (17). The third step follows from the assumption on instance weight boundedness in Definition [ The fourth step is obtained from the fact that J(0") 2 J(OK). The final incquality follows upon
using c/ K/ K. By setting
2(J(0))- - J(0)))
(19)
Lodww
in the above inequality, we get the desired result.
As seen in Theorem M our importance- weighted policy gradient estimator has a convergence rate of O(1/ ((1/KK). Equivalently, the time complexity of our algorithm to obtain an c-accurate solution is ((1/e). Note that our choice of step size requires knowing the total number of iterations K in advance. A more practical approach is to use time-decayed step size of og 1 v/Vk or og 1/k.
B.2 Informal Analysis on Variance of Regularized Importance Sampling
Importance sampling is a widely spread Monte-Carlo technique that adopts a reweighting strategy to estimate the so-called target distribution using samples from another distribution. A major drawback of vanilla importance sampling is the large variance of the weights, which known impact the
accuracy of the estimates badly. In Sec. 3.2 we regularize the importance weights to enhance learning from low- pwwbbabbbillty tokens with the shaping function:
x
f(x)
TE 0,
(20)
where x = ne E (0, +o0) is the original weight. We consider the first-order approximation f(x)
Fe
by Taylor expansion as
(() u)"
f(x) f(u) + f'u)(( u) Σ
(x
n!
(21)
==2
R f(u) + (x)((= - u)
Suppose that T dominates ne. and we have E[x]
EE x
1. We consider the Taylor expansion
point u 1 as
f(x) ~ f(1) (()((x - 1) I+. +
7 (1 (1+ 7)+ (2 - 1)
(22)
17

---

The variance of the first-order approximation of fzx is
Var[s(x]] & War (1 71+ (x 1) = (1+
Var[æ]
(23)
Since (1+7}²
< have [ar] < Varlz]]
Further, we analyze the variance of the regularized importance weights f(x) = x4x E (0, 1) given special case that the original weight variable follow a specific distribution as p(æ) = e e= > 0. This distribution makes sense for the importance weight
me(r) in our setting. First, with the =a(r)
distribution p(z) E- the expectation of is E pp(z) [x] Jo "d
This matches the me(r) 1, given that TT(T) dominates (T()). [
expectation of the importance weight as E|r]
Second, the probability density p(z) decreases gets larger, which matches the intuition that the importance weight x
xa(F) tends to have smaller values. (T( is the probability of an expert trajectory under the assumed optimal policy, which is likely to produce large values TT(T) - 1. (T( is the probability an expert trajectory under the current policy, which is usually small, especially at the early learning stage.
mm(r)
In this case, the variance of the original importance weight is
Var( p(x)) [x²] [x]=
x² - =dr
(24)
1.
The variance of the regularized importance weight is
War[f(z)] = Ep(x)]f(x) - Ep(x) [f()]
dr - -1, re xe-
+ dr (2 27e Ej (7) - -2y²e" (xfe)))(7)) (1-76" (1l-7"EE(()) Ei())) 27²2" 2e Bs(1) 22 Ei(7))
e
x+4)
(25)
<
where - - du > 0 and Es(r)= d > O. Hence, we have Varlf < Var[æ].
In summary, with the above informal analysis, we show that our regularized importance weights can achieve reduced variance, thus providing more stable training for leveraging off-policy guidance.
C Experimental Details
C.1 Detailed Setup
Easy and Hard Training Set These two datasets of different difficulties are generated from subsets of the OpenR 1-MATH-2208 dataset. We first filter the questions for which DeepSeek-RI -1 can generate a correct answer. Then, we split the data according the length of DeepSeek-BR RI's solution. We coin questions RI can solve within 2k tokens as Easy set and those within 4k tokens as the Hard set, respectively. Intuitively, the more tokens needed for Deepseek-R1 to generate correct answer, the more difficult the question is. Finally, the Easy dataset contains 7.3k prompts, and the Hard dataset contains 25.4k prompts.
Training. In addition to Qwen2.5-Math. --tth----hh----- 7B, we extend LUFFY Qwen2.5-Math- 1.50 (30 and Swen2.5-Instruut-. B11, and LLaMA 3. 18B [32]. To ensure faimess, we maintain samples per prompt for all Rl-trained models. The learning rate is constantly set as le-6. All training experiments are conducted using 8 A100 GPUs. We train 500 steps for all RL models and three epochs for SFT models. The only exception is LUFFY), which is trained for 860 steps to match the GPU hour of SFT + RL.
18

---

Our implementation is based on ver for these open- repositories.
which uses vLLM as the rollout generators. We are thankful
Qwen2.5-Series Models. Since the context length of Qwen2.5- Mathh is 4096 and the generation length of off- off-policy samples could be lengthy, we change the rope theta from 10000 40000 and extend the window size to 16384. For 1pwe2..5IInstruce the context window is large enough. Hence, we do not change the model configurations. For all Qwen2.5-Series models, we use the same dataset as described in Sec.
Llama-3.1-8B. For Llama3. 1-8B, we follow Simple-RI-Zooo and use a simplified prompt, and we do not ask the model to generate <think>/</think> tokens.
The dataset used for 1.L.MA3.1-PP is the subset of OpenRI-Math Series models, selected by the length of DeepSeek-RI's correct solution, i.e., 0-2k tokens (Easy training set described in Sec. 5.2). We find that on-policy RL fails on other subsets, such as the Hard training set (0-4k) or the same data used in Qwen2.5-Series
we used with Qwen2.5-
SFT. For all SFT models, we train on the same DeepSeek-RI generated traces and prompts as LUFFY. We follow the SFT setting from open-r1 open-rl/OpenRt--oken performance of Deepseck-RI" RI's distilled model. We train each model for epochs. The train batch size is 64, and the learning rate 5e-5. We use learning rate warmup ratio 0.1 and set the max length to 16k.
which reproduces the
RL SFT Loss. For multi-tasking RL and SFT objectives, we compute the on-policy loss on on-policy samples and SFT loss on off-policy sample per prompt. The other setup is the same as other RI. experiments.
We use the same SFT model described earlier and further conduct RLVR training for 500 more steps. Following previous literature [24), we use the held-out dataset OpenR OpenRI-Mate Math-2206 resulting in around 49k prompts.
SFT + RL
C.2
System Prompt
All our trained models, except 1LLaMA-3.1-8B. share the same system prompt for training and inference:
Your task is to follow systematic, thorough reasoning process before providing the final solu- tion. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: "<think>> An thoughts </think> \n". Each thought should include detailed analysis, brainstorming, veri- fication, and refinement of ideas. After "</th //think> \n" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. applicable, include the answer in \boxed{} for closed-form results like multiple choices mathematical solutions.
User:
This the problem: [QUESTION} Assistant: <think>
For ILLMA-3.11 1-8B.B we do not use the above system prompt as we find the model cannot follow such instruction. Thus, we use simplified version that only includes the CoT prompt and do not include <think> token.
User: {QUESTION}
Answer: Let's think step by step.
http:://gghuub.coo/oloeegggg
hhtps://gittbb.omm/llt com/vllm project//lom
bbtpss/hogggingfcc.coo http::/wwgggiiiiaa...oeeeerrollp
19

---

C.3
Significance Test
We report the significance test results in our main results, i.e., Table The significance test 66 calculated by paired bootstrapping resampling. and the sample size is 1000 times. The null hypothesis asserts that any observed difference is merely due to random sampling variation rather than representing genuine effect or difference between the two groups.
From our results, we can that LUFFY and LUFFY! significantly outperform all baseline methods.
D Case Study
demonstrative case study (Fig comparing our proposed approach (LUFFY) against baseline methods (SFT and GPRO) mathematical problem solving reveals distinct characteristics in rea- soning patterns. SFT demonstrates redundant and circular reasoning with excessive repetition (over 8,129 tokens), while GPRO shows concise but unfounded deduction (1002 tokens), both leading to incorrect conclusions. In contrast, LUFFY presents well-balanced approach (2623 tokens) that combines systematic decomposition with clear mathematical calculation. Through rigorous reasoning and proper verification steps. LUFFY successfully reaches the correct answer, demonstrating the effectiveness of our methodology in achieving both accuracy and efficiency.
E
Additional Results
E.1
Removing On-policy Clip
0.20
On-policy
Mixed-policcy Shaping
The clipping mechanism is introduced to constrain policy updates within trust region [23], thereby ensuring stable training. However, when incorporating off-policy guid- ance, the target behavior may deviate significantly from the model's current policy, especially early in training.
Clip Ratio (%)
0.15
0.10
As shown in Figure LUFFY experiences more frequent clipping compared to On-Policy RL, which can suppress learning from high-quality off-policy traces. To address this, we remove the on-policy clip allow greater flexi- bility in updating toward unfamiliar yet effective actions, thereby unlocking the model's capacity to better integrate off-policg reasoning behaviors.
0.05
h www.
0.00
200
400
Steps
Figure 8: Ratio of clipped signals.
E.2 Extension to More Models
Table presents the detailed performance across six chal- lenging competition-level benchmarks for Qwen2.5-Math- 1.5B, 1wen2.5-Instruut-- and LLaMA-3.. 1---BBB On all three models, LUFFY achieves consistent and substantial improvements, surpassing both SFT and On- On-Policy RL. On -wen2.5-Matt-11550 LUFFY attains an average score of 38.0, demonstrating notable gains of +6.1 1 and +8.0 points over SFT and On-Policy RL, respectively. Similar and
50.0
49.5
Accuracy
49.0
48.5
advantages are observed on the Qwen2.5-Instrucc, ALaMA-3.1-8D where LUFFY consistently outperforms baselines across all benchmarks.
48.0
47.5-
47.0
E.3 Ablation Study
0.05
0.2
7 in Policy Shaping Function
In this section, we perform an ablation study to examine the contributions of LUFFY components, as summarized in Table Shaping and NoClip both positively contribute to the final performance of Mixed-Polic training. However, applying these enhancements without off-policy guidance (On-Policy No Clip/Shaping) does not yield improvement, underscoring the necessity of external signals to acquire nuanced and generalizable reasoning skills.
Figure 9: Accuracy versus the choice of 7.
20

---

Table Overall performance on six competition benchmark performance on Qwen2.5 --th2.5-MMth-
1. 1.5B, Qwen2.5- Owen2.5-Instruct-77 7B and LLaMA-3.1 -.1-BB.
Model
AIME 24
AIME 25
AMC
MATH-500
Minerva
Olympiad
Avg.
Qwen2.5 M.Mhh 1.5B
Owen2.5-Manl 5--at--MM..... 1.5th-1.5--Bsee 330
7.2
3.6
26.4
28.0
9.6
21.2
16.0
Qwen2.5 Matn2..-Math--11 1.th--..5--Istruuc
300
12.1
8.9
48.1
77.4
28.7
39.1
35.7
SFT
11.7
13.2
37.8
70.6
26.8
31.3
31.9
On-Policy RL
11.8
7.7
40.2
61.8
26.8
32,0
30.0
LUFFY
16.0
13.1
47.1
80.2
30.5
41.0
38.0
Qwen2.5 Seee2.3--Instreee I--nstrruct-7
Qwen2.5-7B 1-en2.5-7B-instruet 31
11.7
7.5
43.8
71.8
30.9
40.4
34.4
SFT
7.9
36.0
68.6
21.3
31.1
29.0
On-Policy RL
14.1
8.3
43.5
74.0
33.8
37.6
35.2
LUFFY
17.7
14.8
50.9
82.0
31.3
47.4
40.7
LLaMA 1-8
LLaMA-3.1 LLMMA-3.1-B-IInttuuc 32
5.1
0.4
18.6
44.6
19.5
14.1
17.1
SFT
0.5
0.1
5,4
20.2
4.0
5.3
5,9
On-Policy RL
0.3
0.5
9,4
23.4
17,6
6.1
9.6
LUFFY
1.9
0.1
13.5
39.0
15.1
9.6
13.2
Table 4: Ablation study on LUFFY components.
Model
AIME 24
AIME 25
AMC
MATH-500
Minerva
Olympiad
Avg.
Mixed-Police
19.4
17.7
58.9
84.6
35.7
44.4
Shaping
27.4
21.7
37.1
47.8
Shaping + NoClip
29.4
23.1
65.6
87.6
37.5
57.2
50.1
On-Policy
15.3
62.0
39.3
45.5
Shaping
21.3
13.6
58.0
80.6
36.8
41.8
42,0
No Clip
21.5
17.4
61.1
83.4
36.8
49.0
44.9
E.4
Hyperparameter Study
In this section, we study the choice of in policy shaping function. The results are shown in Figure g trained from Qwen2.5-Math-71, We choose value from [0.05, 0.1, 0.2, 0.3, 0.5]. When 0.1, the model performs the best with 50.1 accuracy scores, and increasing or decreasing this value leads notable decline in model performance. Therefore, consistently use 7 0.1 throughout our experiments.
Analysis
Correct Solutions Length Distribution
In this section, analyze how LUFFY effectively leverages off-policy guidance, i.e., imitating to illuminate, improve reasoning quality and gen- cralization.
LUFFY
Count
60-
30-
1000
2000
3.00
4000
5000
6000
2000
S000
Length (# tokens)
F.1 LUFFY Learns Strategically from Off-Policy Traces, While SFT Imitates Rigidly
Incorrect Solutions Length Distribution
300
We compare the generation length distributions of LUFFY and SFT on the combined set of six mathematical reasoning benchmarks. As shown in Figure I0, LUFFY produces significantly shorter generations on average (2,832 tokens) compared to SFT (4,646 tokens), suggesting a more effec- tive reasoning process that balances imitation and exploration. This observation helps explain the
Count
200-
100
3000 1000
2000
Length (# tokens)
Figure 10: Generation length of correct and incorrect solutions.
21

---

excessive training costs of methods that naively combine SFT and RL, e.g., SFT+RL and RL w/ SFT Loss (), as these models spend substantially more compute on producing unnecessarily lengthy CoT's during the RI roll-out stage. In contrast, SFT often mimics the surface form of off- policy demonstrations without genuinely engaging problem-solving This behavior especially evident in incorrect outputs, where SFT frequently generates overly long and ultimately unproductive reasoning traces. These results indicate that while both methods are exposed to similar off-policy signals, LUFFY learns to selectively internalize useful reasoning patterns, whereas SFT tends to overfit to superficial features of the off-policy data.
We further analyze the generation lengths of w/ SFT Loss and LUFFY during training, as shown in Figure [I] RL w/ SFT Loss quickly imitates the off off-policy traces, exhibiting steep increase in generation length early in training. However, it soon becomes trapped in the superfi- cial patterns of the demonstrations, leading to excessively long outputs that even surpass the length of the original off-policy traces. In contrast, LUFFY's 's dynamic advan- tage balancing between on-policy and off off-policy rollouts encourages more strategic learning. As result, its gener- ation length grows more gradually and steadily, reflecting more selective and grounded adoption of reasoning be- havior.
Mwngy www. mmmm
Response Length
LUFFY
Off-pulicy Ressoming Trmess wy
Beyond generation length, imitation behavior can also observed through the similarity between model outputs and off-policy traces. quantify this, we compare genera- tions from SFT, On Policy RL, and LUFFY against those from DeepSeck-RR R1 on a held-out set of 1,000 samples, using BLEU 633 the similarity metric. The resulting BLEU scores are 57.5 for SFT, 8.8 for Un-Policy RL, and 44.8 for LUFFY, reflecting the strong imitation behavior of SFT and the more selective, yet substantial, imitation in LUFFY. We present case study in Appendix D
Steps
Generation length of RL w/ SFT Loss and LUFFY.
Figure 11:
F2 LUFFY Can Explore During Test-time While SFT Cannot.
We compute pass@8 accuracy on the combined AIME 2024 and AMC datasets, varying the generation temper- ature from 0.1 to 1.0. As shown in Figure 12 both RL- based methods (On-Policy RL and LUFFY) exhibit strong exploratory capabilities, with pass @8 improving the temperature increases, showing potential scaling test- time compute (64). In contrast, although SFT performs comparably to LUFFY under near-deterministic decoding (temperature 0.1), its performance deteriorates higher temperatures, failing to uncover additional correct reason- ing paths. This highlights the fragility and limited adapt- ability of SFT, which aligns with prior findings [65] 66) that suggest SFT tends memorize reasoning patterns rather than learning generalizable reasoning capability.
Performance w.r.t. Temperature
80.0
Accuracy (Pass(8)
77.6
75.0
67.5
On-pollcy RI.
65.0
LUFFY
SPT
62.5
0.1
0.2
0.3 0.1 0.5
0.6 0.7
0.5 0.9 1.0
Temperature
Figure 12: Pass@8 accuracy (on the merge sets of AIME 2024 and AMC) under different generation tempera- tures.
22

---

Case Study
Prompt: <system prompt> This the problem: diagonal of a regular 2006-gon called odd if its endpoints divide the boundary into parts, each composed of an odd number of sides. Sides are also regarded as odd diagonals. Suppose the 2006-gon has been dissected into triangles by 2003 nonintersecting diagonals. Find the maximum possible number of isosceles triangles with two odd sides.
SFT
On-Policy RL
LUFFY
Response:
Response:
Response:
<think>
<think>
<think>
[Problem Restatement Not
[Formulaic Decomposition] To solve this problem, we need to understand ...
[Clear Problem Definition] To solve this problem, we need to understand how it can be We are specifically interested in ... [Systematic Decomposition and Analysis]
Analysis]
Okay, let's try tackle this problem.
and
[Pseudo-Systematic Struc- ture] Let's step-by-step.
So, have regular 2006- gon, and we need to find the maximum number of isosce- les triangles...
1. Unddrstandinn the ... we label the vertices of the poly- gon as __...... V_2006)... [Circular Property Descrip- tion]
First, let's 's define some terms and properties...
[Endless Definition Loop] Let me break this down step by step. First, let's recall some basics about regular polygons and their diago- nals...
Step 1: Understanding Odd Diagonals..
Properties of Odd Diago- nals... Properties of trian- gulation... Properties of the polygon...
Step 2: Properties of Isosce- les Triangles
Regular
Polygon...
Hmm, maybe they mean that sides are considered as diago- nals here?...
Step 3: Counting Odd Diago- nals...
[False Authority Appeal] The number of isosceles tri- angles with two odd sides in a triangulation of regu- lar polygon is well-known problem in combinatorial ge- ometry...
Step 4: Dissecting the Poly- gon into Triangles... [Precise Mathematical Calcu- lation]
Hmm, maybe they ing "diagonal" different sense here...
Alternatively, maybe the problem is using diagonalão mean any chord...
"For each vertex, there are 1002 vertices at odd dis- tance...
[Unsubstantiated sion] For 2006-gon, the maxi- mum number of such trian- gles is 501.
Conclu-
[Wait-Pattern Paralysis]
[Logical Verification] Given that each isosceles tri- angle with two odd sides uses one odd diagonal and one side...
Wait, but in the problem state- ment...
Wait, but sides are chords... Wait, but in the problem's definition, sides are odd diag- onals...
</think> [Final Answer]
[Consistency Check] "Let's verify the logic:
The maximum possible num- ber of isosceles triangles with two odd sides in a triangula- tion of regular 2006-gon 501
[Small-Case Trap]
1. Odd Diagonals..
Let's take a regular polygon with small even number of sides, say (a square).
2. Dissection
</think>
[Final Answer]
triangulation of square has 2 triangles...
Tokens Length: 1002 Correctness: False
Thus, the maximum possible number of isosceles triangles
[Complete Breakdown] but diagon diagon polygon.< odd diagon in the odd of odd diagon = think... wait diagon diagon diagon diagon...
with two odd sides is 1003
Tokens Length: 2623
Correctness: True
Tokens Length: > 8192
Correctness: False
Answer: "$1003$"
Figure 7: Comparison of three approaches (SFT, On-Policy RL, and LUFFY) for geometric problem.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have provided extensive experimental results show that introducing off-policy effective enhance the performance of reinforcement learning and alleviate the limitation posed by model capability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include section discussing limitations (Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer:
[Yes]
Justification:
We have provided complete theory and proof in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We present the experimental details that disclose reproduction information in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide our code and data in supplementary files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttp:ccppppcccceeeeeeeeee..c.ett http:////ggeeee/eoeSSuuiiiiisss (Cooeooomiissssseeeeeeee ssinsoliccyyyyycyccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have specify all the details Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We add significance test in main results Table and explain how to compute the significance test result in Appendix C3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have discuss the resource used in Table 2 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of
Answer: [Yes]
Justification:
We strictly conform with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Here no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited assets including vLLM and verl in Appendix C with licenses properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
30


Reviews:
[{"review_id": "mVqtSJwSmx", "reviewer": "Reviewer_VkPb", "summary": "This paper proposes the idea of introducing policy guidance in the RLVR framework. The idea of using off policy demonstrations have been very effective in RL literature and this paper introduces a similar term in GRPO objective especially for hard datasets where online RL is not able to take off and does not provide any rewards. Introduction of this objective further leads to entropy collapse where the model focuses only on off policy trajectories. This is further addressed by introducing policy shaping in the objective. Experiments with open source Qwen and Llama models show significant performance improvements. The experimental setup looks solid with comparison with many baselines including alternatives to off policy guidance.", "questions": "Confidence intervals are missing for all the results. It is difficult to ascertain gains without confidence intervals. How much is variance when sampling results especially performance @32", "limitations": "It would be important to highlight and include discussions on dataset where Luffy didn’t perform that well compared to other methods e.g Arc-C or Minerva", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* This paper proposes the idea of using off policy trajectories with RLVR which is very relevant to the contemporary advances in LLMs for reasoning so it is very timely and addresses an important issue how to bootstrap RL methods when on policy trajectories fetch no rewards\n* The experimental setup is solid with comparison across a variety of methods including other methods which can use off policy trajectories including variations of using SFT and experiments on 6 different benchmarks.  The ablations are complete and comprehensive\n* The section to compare against other off policy learning methods is really interesting especially from GPU usage point of view and gives an interesting alternative to distillation methods\nWeaknesses:\n* There is some confusion in the presented theory. Equation 4 mentions mean and std in advantage is computed jointly over both off policy and on policy trajectory. Was the same used in equation 5 (2nd term). How is A_j computed in equation 5 ? Would it be better to clarify this aspect clearly with each term clearly explained for reproducibility ?\n* The paper presentation on theoretical parts is unclear although complete and could be improved significantly. For example, the jump from eq 7 to eq 8 could be described in more detail.\n* The limitations in datasets where there is performance loss should be discussed and pointed out clearly especially in table 1", "was_revised": true, "final_justification": "I have read other reviews and other feedback. I would encourage authors to include the rebuttal results and discussion in the revision and would like to maintain my current score.", "rebuttal": "We would like to sincerely thank you for your detailed and constructive review. We are very encouraged by your positive feedback and appreciate that you recognized our work as \"**very timely**\" and addressing an \"**important issue**.\" We are especially grateful for your comments highlighting our \"**solid**\" and \"**comprehensive**\" experimental setup, the significant performance improvements, and the completeness of the ablation studies. It was also wonderful to hear that you found our comparison to other off-policy methods \"**really interesting**\" and a valuable alternative. Your encouraging assessment is greatly appreciated, and we will carefully address your suggestions for improvement to further strengthen the paper.\n\n\nThe following is our response to your questions:\n\n**Q1: There is some confusion in the presented theory. Equation 4 mentions mean and std in advantage is computed jointly over both off policy and on policy trajectory. Was the same used in equation 5 (2nd term). How is A_j computed in equation 5 ? Would it be better to clarify this aspect clearly with each term clearly explained for reproducibility ?**\n\n- The advantage in equation 5 is the same as equation 4. We use \\hat{A}_i and \\hat{A}_j to denote that they are from off-policy and on-policy trajectories, respectively. Thanks for your comment, and we will clarify this aspect in our next version.\n\n\n\n**Q2: The jump from eq 7 to eq 8 could be described in more detail.**\n- Thanks for your advice. The derivation from Equation 7 to Equation 8 makes use of the derivative of the softmax function. We will introduce more intuition about how we derive it and put a more detailed derivation in the appendix for reference.\n\n**Q3: It would be important to highlight and include discussions on dataset where Luffy didn’t perform that well compared to other methods e.g Arc-C or Minerva.**\n\n- That’s an insightful question! For Minerva, we manually checked the dataset and find the dataset is sometimes noisy and the golden answer is not perfect. We observe that all methods bring steady improvements on in-domain benchmarks, except for Minerva, which fluctuates a lot. \n- For OOD benchmarks like ARC-C, We do not interpret LUFFY’s performance on ARC-C as a loss. In fact, compared with SFT-based methods (SFT, SFT+RL, RL w/ SFT loss), which show significant drops on ARC-C, LUFFY performs on par with on-policy RL, maintaining strong OOD generalization. This highlights that our mixed-policy approach effectively leverages in-domain knowledge without erasing out-of-domain capabilities—a key advantage over naive off-policy distillation.\n- We will add more discussion about this in our next version to further illustrate this issue.  \n\n**Q4: Confidence intervals are missing for all the results. It is difficult to ascertain gains without confidence intervals. How much is variance when sampling results especially performance @32.**\n\n- Thanks for your thoughtful comment. We have repeated our evaluation with AMC, AIME, AIME 25 for 32 runs and other benchmarks with 3 runs to compute the confidence interval. The shown results are the 95% confidence interval and computed by 1.95 * standard deviation. The results with confidence intervals are as follows:\n\n| Model | MATH | Minerva | Olympiad | AIME | AIME25 | AMC | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Qwen2.5-Math-7B | 43.9 ± 2.5 | 9.1 ± 2.0 | 14.5 ± 1.5 | 11.1 ± 2.0 | 4.7 ± 1.3 | 31.4 ± 1.8 | 19.1 ± 1.9 |\n| Qwen2.5-Math-7B-Instruct | 81.0 ± 2.0 | 36.6 ± 3.3 | 40.1 ± 2.1 | 10.9 ± 2.0 | 9.1 ± 1.8 | 48.3 ± 1.9 | 37.7 ± 2.2 |\n| PRIME | 81.7 ± 2.0 | **40.0 ± 3.4** | 40.9 ± 2.1 | 18.1 ± 2.4 | 14.4 ± 2.2 | 55.9 ± 1.9 | 41.8 ± 2.3 |\n| Oat-Zero | 79.4 ± 2.0 | 35.7 ± 3.3 | 43.4 ± 2.2 | **32.9 ± 3.0** | 10.1 ± 1.9 | 59.4 ± 1.9 | 43.5 ± 2.4 |\n| Simple-RL-Zero | 75.5 ± 2.2 | 26.7 ± 3.0 | 35.2 ± 2.1 | 26.6 ± 2.8 | 8.3 ± 1.7 | 55.6 ± 1.9 | 38.0 ± 2.3 |\n| Open-Reasoner | 82.6 ± 1.9 | 35.3 ± 3.3 | 46.6 ± 2.2 | 17.0 ± 2.4 | 13.2 ± 2.1 | 52.0 ± 1.9 | 41.1 ± 2.3 |\n| On-Policy | 84.5 ± 1.8 | 38.2 ± 3.3 | 47.2 ± 2.2 | 22.2 ± 2.6 | 14.2 ± 2.2 | 62.1 ± 1.8 | 44.7 ± 2.3 |\n| SFT | 82.1 ± 1.9 | 38.7 ± 3.3 | 42.0 ± 2.1 | 23.0 ± 2.7 | 22.3 ± 2.6 | 53.5 ± 1.9 | 43.6 ± 2.4 |\n| SFT + RL | 86.5 ± 1.7 | 39.8 ± 3.4 | 48.3 ± 2.2 | 29.4 ± 2.9 | 22.3 ± 2.6 | 62.2 ± 1.8 | 48.1 ± 2.4 |\n| LUFFY | **87.3 ± 1.7** | 38.7 ± 3.3 | **55.9 ± 2.2** | 29.1 ± 2.9 | **24.0 ± 2.7** | **64.7 ± 1.8** | **50.0 ± 2.4** |\n\n- We will update our next version with these results. \n\n**Q5: The legend in Figure 6 is not clear. Although it can be understood that dotted lines represent entropy, it should be labelled. Line 296: typo as “uently”**\n\n- Thanks for your advice and reminder. We will fix these issues in our next version. \n\nWe hope our responses answer your questions, and again, thanks for your advice to improve our paper!"}, {"review_id": "49UE10vf4b", "reviewer": "Reviewer_5T3H", "summary": "This work presents a new algorithm that blends on-policy GRPO with off-policy policy optimization from expert demonstrations. This algorithm avoids pure on-policy RL training that can fail to learn novel behavior. Hence, the authors propose to use guidance from stronger models (off-policy reasoning traces) to improve performance. They empirically evaluate their method in Math reasoning tasks.", "questions": "- From Equation 6 to 7, the function f changes from being a function of the ratio \\pi_theta / \\pi_\\phi to only depend on \\pi_\\theta. Why is that?\n- Why does your on-policy RLVR replication have an overall improvement of performance with respect to the baselines? \n- Why is setting \\pi_\\theta = 1 a reasonable choice?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well written. The authors clearly motivate their algorithm design.\n- The authors provide a set of comprehensive baselines to contextualize the performance of their method.\n- The authors provide empirical evidence of the algorithms training dynamics which provide further insight of their method. \n\nThe authors provide comprehensive evaluation of their method and introduce a practical shaping mechanism to avoid collapse of the policy’s entropy that their basic mixed-policy GRPO. Moreover, they show empirically that they are able to exploit the expert’s guidance to improve upon the SFT+RL pipeline that is common practice.", "was_revised": true, "final_justification": "The author addressed the question I had during the review process. I find the paper to be well motivated and addresses a problem that might be of interest in the community. Moreover, their evaluation show empirical merit to their method.", "rebuttal": "We would like to extend our sincere thanks for your positive and encouraging review. We are delighted that you found the paper to be \"**well written**\" and our algorithm design \"**clearly motivated**.\" We particularly appreciate you highlighting our \"**comprehensive evaluation**,\" the inclusion of strong baselines, and our analysis of the training dynamics which you noted provide \"**further insight**\" into our method. Finally, we are very grateful for your acknowledgment that we were able to empirically show improvement upon the common SFT+RL pipeline. Your thoughtful feedback is incredibly motivating.\n\nThe following is our response to your questions:\n\n***Q1: From Equation 6 to 7, the function f changes from being a function of the ratio \\pi_theta / \\pi_\\phi to only depend on \\pi_\\theta. Why is that? Why is setting \\pi_\\theta = 1 a reasonable choice?***\n\n- In our methodology, we simplify the policy ratio from p_theta / p_phi to solely p_theta. This simplification stems from treating p_phi as an optimal reference policy and is a pragmatic choice driven by two key practical considerations.\n- Conceptually, we regard the reference policy p_phi as an oracle or an \"expert\" that generates the ground-truth or target responses. Under this assumption, the actions (tokens) in the reference sequence are considered the definitive optimal choices. Therefore, the probability of the optimal policy taking these actions is implicitly treated as 1 (i.e., p_phi = 1), simplifying the ratio to just the probability of our own policy, p_theta.\n- This design choice is motivated by the following challenges:\n  - *Tokenization Mismatches:* The expert policy we aim to learn from (e.g., a powerful proprietary LLM) often employs a different tokenizer than our trainable policy p_theta. This tokenization mismatch makes a direct, token-by-token probability comparison infeasible. For instance, the same text could be segmented into token sequences of different lengths by the two models, rendering the computation of a meaningful probability ratio impossible.\n  - *Compatibility with Existing Datasets:* This simplification facilitates the direct use of popular off-the-shelf datasets (e.g., instruction-following or dialogue corpora). These public resources typically provide high-quality text demonstrations but don't include the generation probabilities from the original source model. Our approach bypasses the need to recompute or approximate p_phi—which is often an impractical or impossible task—thereby significantly enhancing the real-world applicability of our method.\n- In L142 to L149 we discuss our reasons and we will modify this part to make it clearer.\n\n***Q2: Why does your on-policy RLVR replication have an overall improvement of performance with respect to the baselines?***\n\n- The main reason we think is the **better choice of dataset with** **proper difficulty**. SimpleRL-Zero and Oat-Zero use the MATH dataset. OpenReasoner collects both in-domain math problems (e.g., AIME (up to 2023), MATH, Numina-Math collection, etc.) and synthesizes general reasoning tasks using programmatic approaches.  PRIME uses an 860K subset of Numia-Math.\n- In comparison, we use a subset of the OpenR1 dataset by filtering the correctness of Deepseek-R1's trajectories. There is recent work discussing how the difficulty of the dataset affects the successful training and final performance of RLVR[1,2]. Their findings demonstrate that either the dataset is too difficult or too easy for the model would harm the final performance of RLVR. \n- As our on-policy baseline has a better performance, it further validates the effectiveness of LUFFY.\n\nWe hope our responses answer your questions, and again, thanks for your advice to improve our paper!\n\n_[1] Chen, M., Liu, H., Liang, H., Huang, H., Zhang, W., & He, R. (2025). Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning. arXiv preprint arXiv:2505.13261._\n\n_[2] Wang, Y., Yang, Q., Zeng, Z., Ren, L., Liu, L., Peng, B., ... & Shen, Y. (2025). Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571._"}, {"review_id": "fGJvSFbR7F", "reviewer": "Reviewer_ubx4", "summary": "This paper focuses on off-policy RL for LLMs. Specifically, the proposed method mixes in some off-policy rollouts during RL training (e.g., 7:1 on-policy to off-policy in some setups), and applies a shaping function x / (x + \\gamma) to the importance sampling ratio in the policy objective. They show that with this setup, the model outperforms baselines on a math dataset after RL.", "questions": "In Table 2, why do the data setups vary so much across different experiments?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Pros:\n- Clear and easy-to-follow writing\n- Shows positive results empirically\n\n### Cons:\n- The method is pretty straightforward—more like a numerical trick. The actual effectiveness and necessity need more ablations. For instance, could the “policy shaping” part just be done with a well-tuned clipping strategy?\n- The ablation study setup feels messy. The gains are questionable. The method is not purely off-policy; it uses a mix of off- and on-policy data. Most baselines use different rollout sources. In other words, compared to the RLVF baseline, this method uses extra distillation from DeepSeek-R1 (so the performance boost is somewhat expected). Compared to SFT, it also benefits from online rollouts, which could help with OOD robustness. It is not apples-to-apples.\n- More metrics like pass@k would help better reflect actual effectiveness, instead of just reporting average reward.", "was_revised": true, "final_justification": "This work directly uses distillation data as off-policy RL rollouts to perform GRPO, and the reported numbers look quite decent. They introduce some tricks, such as transforming the importance sampling ratio to avoid strictly adhering to imitation-labeled data.\n\nThe method is very straightforward, with limited novelty.\n\nA key concern is that, given the maturity of existing SFT+RL infrastructures, a more carefully designed pipeline may perform just as well or even better than this trick-based approach.\n\nThe authors argue that their two-stage method saves GPU hours, but their experiments do not align the consumed data across different settings, which calls for more controlled experiments to validate the claim. I remain cautious about this point.\n\nTherefore, I give a weak positive score.", "rebuttal": "We thank you for your time and valuable feedback. We appreciate you highlighting the **\"Clear and easy-to-follow writing\"** of our paper and for acknowledging that our method **\"Shows positive results empirically.\"** Also, we are grateful that **other reviewers** (we refer to VkPb as R1 and 5T3H as R2) **are quite affirmative to our overall contributions, including the motivation and novelty** (”is very relevant to the contemporary advances…addresses an important issue” by R1 and “ clearly motivate their algorithm design” by R2), **the empirical evaluation** (”the experimental setup is solid…the ablations are complete and comprehensive” by R1 and “provide comprehensive evaluation…a set of comprehensive baselines” by R2), **and the writing** (”the paper is well written” by R2). \n\nBefore addressing your question, we first want to offer this concise clarification of our research question and contributions to ensure we are on the same page.\n\n***Research Question:***\n\n- On-policy Reinforcement Learning from Verifiable feedback (RLVR) is the de facto method for training reasoning models, shown by Deepseek-R1, but its performance is capped by the base model's capability, limiting the usage of RLVR for LLMs[1,2], especially ones without strong initial performance. Our research question is: **How can we use off-policy guidance from a stronger model to break this performance ceiling?** This is a timely and critical issue for advancing LLM reasoning, as recognized by Reviewer VkPb.\n\n***Our Contributions:***\n\n1. **A Novel and Effective Mixed-Policy Method for RLVR.**\n    - Our method, LUFFY, is the first to use a mixed-policy framework to **meaningfully improve** RLVR performance, not just accelerate it.\n    - This is a non-trivial contribution. We show that naive mixed-policy fails to improve final results (as discussed in Section 3.2 and especially L156-L161). Our key insight is to introduce **policy shaping** to solve the *entropy collapse* problem, forcing the model to learn from informative but low-probability tokens.\n2. **Strong Empirical Validation Proving Our Thesis.**\n    - We establish a **new SOTA** on RLVR with Qwen2.5-Math-7B.\n    - We demonstrate **generalizability** across various foundation models.\n    - Crucially, we show LUFFY **successfully trains weaker models where on-policy RLVR fails**, directly proving that our method overcomes the base model's capacity limits (our RQ).\n- We hope this clarifies the significance and novelty of our work.\n\n**Q1: *The method is pretty straightforward—more like a numerical trick. The actual effectiveness and necessity need more ablations. For instance, could the “policy shaping” part just be done with a well-tuned clipping strategy?***\n\n- Our method is far more than a mere numerical trick. Foremost, we propose **a pure RL-based method to address the significant challenge** of empowering LLMs to acquire reasoning behaviors surpassing their initial cognitive boundaries. We provide an elegant way to harness the benefits of both worlds (the external guidance and self-driven exploration) via a unified mixed-policy update rule. **Unlike SFT, we novelly formulate the external guidance as an off-policy objective within a pure RL framework**. It enables the model to internalize deeper and more generalizable reasoning behaviors through the lens of RL. Then, we propose policy shaping via regularized importance sampling to enhance exploration by emphasizing low-probability tokens. Theoretical analyses are provided to highlight the convergence rate of our mix-policy update and the better training stability achieved by our policy shaping.\n- Furthermore, we have the experimental results to justify why policy shaping is both **necessary** and **non-trivial**:\n- *Ablation Study Demonstrates Necessity (Table 4):*\n  - Our ablation study (Table 4) shows that neither mixed-policy GRPO nor policy shaping alone yields substantial improvements. Only when both are combined does the model achieve significant gains. This highlights that policy shaping is critical for encouraging the model to learn from low-probability but crucial decisions. Without shaping, the model tends to overfit high-probability actions from the off-policy traces, leading to entropy collapse.\n- *Shaping vs. Clipping are Fundamentally Different:*\n  - Policy shaping is not a mere numerical trick like clipping. Shaping re-weights gradients to emphasize low-probability yet pivotal tokens, enabling the model to internalize key reasoning steps from off-policy trajectories. In contrast, clipping is a constraint that limits updates for actions deviating too much from the current distribution. Importantly, our method does not use clipping on off-policy rollouts, as explicitly stated in Sec. 3.1.\n- *Clipping Alone is Insufficient (Figure 6):*\n  - We have tried mixed-policy GRPO with different clipping values, and all results were comparable to plain mixed-policy (Figure 6). This confirms that clipping cannot replicate the effect of shaping, which is designed to actively encourage exploration of critical low-probability actions.\n\n***Q2: The ablation study setup feels messy. The gains are questionable. The method is not purely off-policy; it uses a mix of off- and on-policy data. Most baselines use different rollout sources. In other words, compared to the RLVF baseline, this method uses extra distillation from DeepSeek-R1 (so the performance boost is somewhat expected). Compared to SFT, it also benefits from online rollouts, which could help with OOD robustness. It is not apples-to-apples.***\n\n- We respectfully disagree and would like to clarify:\n- *Ablation Study Highlights Non-Trivial Gains:*\n  - Beyond the main results, our ablation study (Table 4) carefully compares on-policy, mixed-policy, and full LUFFY. This setup demonstrates that mixed-policy alone does not yield the same improvements, and policy shaping is critical to leveraging off-policy data effectively. These controlled ablations underline that LUFFY’s improvements are not from superficial data mixing but from a well-designed algorithmic combination of mixed-policy GRPO and shaping.\n- *Comprehensive and Fair Baseline Coverage:*\n  - With all due respect, we believe our comparisons are thorough and fair, as noted by Reviewer 5T3H and VkPb. Since LUFFY is the first mixed-policy approach in RLVR, we compare against:\n    - Pure on-policy: Standard RLVR baselines and our replication of on-policy RL.\n    - Pure off-policy: SFT using the same DeepSeek-R1 traces.\n    - Mixed-policy: RL w/ SFT Loss and SFT+RL.\n  - These cover all three categories (on-policy, off-policy, mixed-policy). LUFFY achieves substantial gains both on in-domain and out-of-distribution benchmarks compared to these baselines, with comparable data and rollout sources.\n- *Fairness of Data Usage:*\n  - The comparison with SFT and SFT+RL is strictly fair — all methods use the same off-policy traces (from DeepSeek-R1) and online rollouts. LUFFY’s superior OOD robustness stems from its ability to balance exploration and imitation, which SFT and SFT+RL fail to achieve due to their tendency to overfit to fixed off-policy data.\n\n***Q3: More metrics like pass@k would help better reflect actual effectiveness, instead of just reporting average reward.***\n\n- Thanks for your advice. We have the results of pass@8 in Appendix F.2 due to the space limit. With increased temperature, the pass@8 of On-Policy and LUFFY go up and SFT drops, and LUFFY has a substantial improvement.\n- The results demonstrate that LUFFY can maintain better exploration with large potential solution space while SFT cannot. We will modify our main content to better reference these results. Also, we have conducted experiments with confidence interval to demonstrate the effectiveness of LUFFY is not affected by variance. Please refer to our response to Reviewer VkPb.\n\n***Q4: In Table 2, why do the data setups vary so much across different experiments?***\n\n- We are sorry for the confusion caused. Here, we aim for a fair comparison in terms of either data or GPU usage, to demonstrate the effectiveness of LUFFY.\n- The data usage represents the data generated and consumed during the RL phase. We split the usage into on-policy usage and off-policy usage to demonstrate how much off-policy data different methods rely on. LUFFY is our base setup and LUFFY+ is the setup we match GPU hours with that of SFT+RL. We will make that part clearer in our next version.\n\nPlease let us know if we have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript. **If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.**\n\n*[1] Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., & Huang, G. (2025). Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?. arXiv preprint arXiv:2504.13837.*\n\n*[2] Rajani, N., Gema, A. P., Goldfarb-Tarrant, S., & Titov, I. (2025). Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them. arXiv preprint arXiv:2507.10616.*"}]

Author rebuttal:

Meta-review: The idea is nice and after the author response it was as consensus accept with 2 enthusiastic reviews.
Decision: Accept (poster)