Title: Learning to Reason under Off-Policy Guidance
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning with verifiable rewards

off-policy guidance
Abstract: Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(RLVR). 
However, existing RLVR approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. 
To address this issue, we introduce LUFFY (Learning to reason Under oFF-policY guidance), a framework that augments RLVR with off-policy reasoning traces. 
LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. 
Specifically, LUFFY combines the Mixed-Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. 
Compared with previous RLVR methods, LUFFY achieves an over +6.4 average gain across six math benchmarks and an advantage of over +6.2 points in out-of-distribution tasks.
Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.


Peer reviews:
comments: SUMMARY:
This paper focuses on off-policy RL for LLMs. Specifically, the proposed method mixes in some off-policy rollouts during RL training (e.g., 7:1 on-policy to off-policy in some setups), and applies a shaping function x / (x + \gamma) to the importance sampling ratio in the policy objective. They show that with this setup, the model outperforms baselines on a math dataset after RL.

STRENGTHS AND WEAKNESSES:
### Pros:
- Clear and easy-to-follow writing
- Shows positive results empirically

### Cons:
- The method is pretty straightforward—more like a numerical trick. The actual effectiveness and necessity need more ablations. For instance, could the “policy shaping” part just be done with a well-tuned clipping strategy?
- The ablation study setup feels messy. The gains are questionable. The method is not purely off-policy; it uses a mix of off- and on-policy data. Most baselines use different rollout sources. In other words, compared to the RLVF baseline, this method uses extra distillation from DeepSeek-R1 (so the performance boost is somewhat expected). Compared to SFT, it also benefits from online rollouts, which could help with OOD robustness. It is not apples-to-apples.
- More metrics like pass@k would help better reflect actual effectiveness, instead of just reporting average reward.

QUESTIONS:
In Table 2, why do the data setups vary so much across different experiments?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This work presents a new algorithm that blends on-policy GRPO with off-policy policy optimization from expert demonstrations. This algorithm avoids pure on-policy RL training that can fail to learn novel behavior. Hence, the authors propose to use guidance from stronger models (off-policy reasoning traces) to improve performance. They empirically evaluate their method in Math reasoning tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is well written. The authors clearly motivate their algorithm design.
- The authors provide a set of comprehensive baselines to contextualize the performance of their method.
- The authors provide empirical evidence of the algorithms training dynamics which provide further insight of their method. 

The authors provide comprehensive evaluation of their method and introduce a practical shaping mechanism to avoid collapse of the policy’s entropy that their basic mixed-policy GRPO. Moreover, they show empirically that they are able to exploit the expert’s guidance to improve upon the SFT+RL pipeline that is common practice.

QUESTIONS:
- From Equation 6 to 7, the function f changes from being a function of the ratio \pi_theta / \pi_\phi to only depend on \pi_\theta. Why is that?
- Why does your on-policy RLVR replication have an overall improvement of performance with respect to the baselines? 
- Why is setting \pi_\theta = 1 a reasonable choice?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the idea of introducing policy guidance in the RLVR framework. The idea of using off policy demonstrations have been very effective in RL literature and this paper introduces a similar term in GRPO objective especially for hard datasets where online RL is not able to take off and does not provide any rewards. Introduction of this objective further leads to entropy collapse where the model focuses only on off policy trajectories. This is further addressed by introducing policy shaping in the objective. Experiments with open source Qwen and Llama models show significant performance improvements. The experimental setup looks solid with comparison with many baselines including alternatives to off policy guidance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This paper proposes the idea of using off policy trajectories with RLVR which is very relevant to the contemporary advances in LLMs for reasoning so it is very timely and addresses an important issue how to bootstrap RL methods when on policy trajectories fetch no rewards\n* The experimental setup is solid with comparison across a variety of methods including other methods which can use off policy trajectories including variations of using SFT and experiments on 6 different benchmarks.  The ablations are complete and comprehensive\n* The section to compare against other off policy learning methods is really interesting especially from GPU usage point of view and gives an interesting alternative to distillation methods\nWeaknesses:\n* There is some confusion in the presented theory. Equation 4 mentions mean and std in advantage is computed jointly over both off policy and on policy trajectory. Was the same used in equation 5 (2nd term). How is A_j computed in equation 5 ? Would it be better to clarify this aspect clearly with each term clearly explained for reproducibility ?\n* The paper presentation on theoretical parts is unclear although complete and could be improved significantly. For example, the jump from eq 7 to eq 8 could be described in more detail.\n* The limitations in datasets where there is performance loss should be discussed and pointed out clearly especially in table 1\n\nQUESTIONS:\nConfidence intervals are missing for all the results. It is difficult to ascertain gains without confidence intervals. How much is variance when sampling results especially performance @32\n\nLIMITATIONS:\nIt would be important to highlight and include discussions on dataset where Luffy didn’t perform that well compared to other methods e.g Arc-C or Minerva", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}