Squeeze the Soaked Sponge: Efficient Off-policy RFT for Large Language Model
Authors: Jing Liang
Jinyi Liu
Yi Ma
Hongyao Tang
YAN ZHENG
Shuyue Hu
LEI BAI
Jianye HAO
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Finetuning
Large Language Model
Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning (RL) has demonstrated its potential to improve the reasoning ability of Large Language Models (LLMs), yet most existing Reinforcement Finetuning (RFT) methods are inherently *on-policy* RL, failing to reuse historical data and thus preventing efficient scaling. In this work, we explore the potential of *off-policy* RL to leverage historical data for rollout-efficient RFT. Specifically, we propose **Re**incarnating **Mix**-policy Proximal Policy Gradient (**ReMix**), which enables on-policy RFT methods to leverage off-policy data. ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio that utilizes the data from both current and past policies for efficient training; (2) KL-Convex policy constraint that combines the KL constraints on the base and precedent model to balance stability and flexibility; (3) Policy reincarnation that replaces the base model with the mix-policy RFT model in the mid way of training and restarts on-policy training, to achieve a seamless transition from early efficiency to steady convergence. In our experiments, we train a series of ReMix models based on PPO, GRPO from 1.5B, 7B base models. On five math reasoning benchmarks (i.e., AIME'24, AMC'23, Minerva, OlympiadBench, and MATH500), ReMix achieves an average Pass@1 accuracy of **52.10%** (with **0.079M rollouts**) and **64.39%** (with **0.011M rollouts**) on 1.5B and 7B models, respectively. Compared with 15 recent advanced models, ReMix shows SOTA-level performance with an over **30x to 450x reduction in training cost in terms of rollout data volume**, demonstrating superior training efficiency. Additionally, our multifaceted analysis reveals insightful findings, including the implicit preference for shorter responses of off-policy RFT, the collapse mode of self-reflection under severe off-policyness, etc. The code and the trained models are available at https://anitaleungxx.github.io/ReMix/ .

Published as a conference paper at ICLR 2026
SQUEEZE THE SOAKED SPONGE: EFFICIENT OFF-POLICY RFT FOR LARGE LANGUAGE MODEL
Jing Liang")
Jinyi Liu Yi Ma Hongyao Tang
Yan Zheng
Shuyue Hu Lei Bai
Jianye Hao
School of New Media and Communication, Tianjin University
School of Computer Software, Tianjin University
School of Computer and Information Technology. Shanxi University
*Shanghai Artificial Intelligence Laboratory
ABSTRACT
Reinforcement Learning (RL) has demonstrated its potential to improve the reason- ing ability of Large Language Models (LLMs), yet most existing Reinforcement Finetuning (RFT) methods are inherently on-policy RI., failing to reuse historical data and thus preventing efficient scaling. In this work, we explore the potential of off-policy RL to leverage historical data for rollout-efficient RFT. Specifically, we propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix), which enables on-policy RFT methods to leverage off- off-policy data. ReMix consists three major components: (1) Mix-policy proximal policy gradient with increased Update-To- To-Data (UTD) ratio that utilizes the data from both current and past poli- cies for efficient training; (2) KL.-Convex policy constraint that combines the KI constraints on the hase and precedent model to balance stability and flexibility; (3) Policy reincarnation that replaces the base model with the mix-policy RFT model in the mid way of training and restarts on-policy training, to achieve a seam- less transition from early efficiency to steady convergence. In our experiments, we train series of ReMix models based on PPO, GRPO from 1.5B, 7B base models. On five math reasoning benchmarks (i.e., AIME'24, AMC'23, Minerva, OlympiadBench, and MATH500), ReMix achieves an average Pass @ accuracy of 52.10% % (with 0.079M rollouts) and 64.39% (with 0.011M rollouts) on 1.5B and 7B models, respectively. Compared with 15 recent advanced models, ReMix shows SOTA-level performance with an over 30x to 450x reduction in training cost in terms of rollout data volume, demonstrating superior training efficiency. Additionally, our multifaceted analysis reveals insightful findings, including the implicit preference for shorter responses of off- polico RFT, the collapse mode of self-reflection under severe off- off-policyness, etc. The code and the trained models are available https: // /t/iiaataallaaaaaaa htt///aiiaallaxxxx.aaa/a eungxx, gi thu.. i ReMix/.
INTRODUCTION
The emergence of Large Language Models (LLMs) has lifted artificial intelligence to a next level, with the milestone works like (OpenAI, 2022; Jaech et al., 2024; Bai et al., 2022a; Trung et al., 2024; Guo et al., 2025). Consistent efforts are being made to push forward the limits of LLMs in performing deeper thinking and solving more complex tasks (Li ct al., 2025b). Recently, Large Reasoning Models (LRMs) (Jaech et al., 2024; Guo et al., 2025; Kimi et al., 2025; Yang et al., 2025a) have taken the stage and attracted great attention, showing that significant improvement of problem-solving ability can be achieved by a long human-like reasoning process (i.e., slow thinking), especially in scenarios like Math, Coding, Scientific Q&A, etc. One of the central recipes of LRMs is Reinforcement Finetuning (RFT) (Trung et al., 2024). By treating the LLM as a policy model, the LLM can follow the philosophy of Reinforcement Learning (RL) (Sutton & Barto, 1998) and learn to
"quual contribution.
Corresponding authors: ji jinyee. aaane.hoott, wdu.ed..aa. en, yanzheng@t ju. edu. en, tanghongyaa tju. mu..edu.com

---

Published as a conference paper at ICLR 2026
1.5B Models
Models
56
Average Performance (Pass
52
ReMix-PPO
X
X
PPO
(500)
(200)
FastCuRL-V3
DeepScaleR
(30)
(75)
AceReason
4.
(200)
[500]
(200)
L1-Max
L1-Exact
ReMix-PPP
86
(109)
GRPO
64
GRPO
FastCuRL-Prewiew
AdaptThink
AdaptThink
36
Open R52
Skywork-DRt Prevvle
Open-RS1
Open-RS3
4.0
5.0
5.5
6.0
6.5
7.0
Rolliout Data Volume (Log10)
Rollout Data Volume (Log10)
Models Trained on R1-Distilled
Models Trined on DeepScaleR
dt
Ours
Figure 1: Efficiency-perfommmace comparison for 1.5B models (left) and 7B models (right) in terms of Rollout Data Volume (i.e., total number of responses generated during training) v.s., Average Pass@1 1 Accuracy on five math reasoning benchmarks. An ideal model should appear in the top-left.
reason and answer the queries according to the reward signals, e.g., either from a verifiable reward function (Guo al., 2025) or a learned reward model (Bai al., 2022a).
Although RFT opens another space for more powerful reasoning ability of LLMs, the longstanding and notorious shortcoming of RL sample inefficiency still exists. In another word, RFT usually needs significantly more computational cost (e.g., rollouts and training) than SFT due to its
nature. The inefficiency of RL poses a stringent bottleneck on time and cost, consequently preventing further scaling of model size and response length of LLMs. Currently, policy gradient algorithms like PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), RLOO (Ahmadian et al., 2024) are widely adopted for RFT of LLMs due to their stable learning performance and friendliness to engineering. However, they are all on-policy algorithms, which are known to be sample inefficient as the data generated by the online policy is dropped after each iteration. In the literature of RL, off-policy algorithms are naturally more sample efficient since they also learns from the data generated by historical policies (i.e., experience) (Sutton & Barto, 1998; Silver & Sutton, 2025). Following this direction, recent research has begun to incorporate off-policy data in RFT in different ways, including using nonuniform replay strategies (Li al., 2025a), learning from positive and negative signals asymmetrically (Roux et al., 2025; Arnal et al., 2025), proposing new learning objectives based on generation consistency (Tang et al., 2025; Cohen et al.. 2025). and learning from demonstrations of superior models (Yan et al., 2025), etc. Despite the efforts made by these works, off-policy RFT remains underexplored in two aspects: (1) None of these methods was compared with SOTA models on multiple mainstream math reasoning benchmarks, leaving training efficiency and final performance of these methods untested thoroughly; (2) The influence of off-policy learning on the learning process of reasoning ability remains unknown, which impedes essential understanding of off-policy learning for RFT and advancement of effective methodologies.
In this paper, we study off offlpo RL for post-training finetuning of LLMs, aiming achieve SOTA- level reasoning ability efficiently and unbox the effects of off off-policy learning for useful insights. We propose Reincarnating Mix-policy Proximal Policy Optimization (ReMix), a general approach to enable on-policy proximal policy gradient methods to leverage off-policy data efficiently. ReMix consists of three major components: (I) Mix-policy proximal policy gradient with an increased Spdate-To-Data (UTD) ratio (Chen et al., 2021) leverages the data generated by both current policy and past polices for efficient training: (2) KL-Convex policy constraint (Ma al., 2024b) combines the KL constraints on the base model and the precedent model to balance the trade-off between stability and flexibility during training: (3) Policy reincarnation (Agarwal et al., 2022) replaces the base model with the mix-policy RFT model in the mid way of training and restarts on-policy training. to achieve a seamless transition from efficient early-stage learning to steady asymptotic improvement. Under the synergy of the three components, ReMix is able to improve the reasoning ability of LLMs efficiently while retaining a stable and flexible training process.
In our experiments, we adopt PPO and GRPO representative on-policy methods and implement ReMix-PPO and ReMOx-GRPO. We use DeepSeek- R1 -ii---itil---O----- 5B and -7 (Guo et al., 2025) as the base models, and train our models based on et al., 2025). We conduct a range of comparative evaluations against 15 recent advanced models on five math reasoning benchmarks, including AIME'24, AMC'23, Minerva (Lewkowyez et 2022),
(Luo
2

---

Published as a conference paper at ICLR 2026
OlympiadBench (He et al., 2024), and MATH500 (Hendrycks et al., 2021). Figure 1 summarizes the experimental results in a view of
comparison. Our method achieves an average Pass accuracy of 52.10% % (for 1.5B model) with 0.079M response rollouts, 350 training steps and achieves 63.27%//4.33 % (for 7B model) with 0.007M/0.01 1M response rollouts, 50/75 training steps respectively, showing SOTA--lvvll performance and over 30x to 450x training cost reduction in terms of rollout data volume. Moreover, to gain better understanding of off ooolpy learning for RFT, we conduct multifaceted studies and analysis, revealing insightful findings including the implicit preference for shorter responses due to the Whipping Effect of off off-policy discrepancy, the collapse mode of self-reflection behavior under the presence of severe off- policyycss,ss the performance under response length constraint, the impact of prompt format, etc.
2 PRELIMINARIES
Reinforcement I Learning for LLM Fine-tuning Reinforcement Fine- Tuning (RFT) is a paradigm for adapting pre-trained LLMs to specific downstream tasks using RL (Trung al., 2024; Jaech et al., 2024). In this paradigm, text generation is modeled as a Markov Decision Process (MDP) M = (S, A. P. R, 7), where state s = (g. 31:11) E is the prompt with the output generated so far, and the action
E EAir is the next token selected from the vocabulary V. Hence, the transition P( (e: ar) is deterministic in this context. An episode start from prompt So (out of predefined set Do) and terminate at an end-of-sequence token by the maximum sequence length H.
The reward R( as) signal is issued by either a rule-based reward function or a learned reward model. In the scope of this paper, we consider the verifiable reward function. For any non-terminal timestep t < T - 1, [[; az) is 0; on completion, the terminal reward, denoted by R(T) for the whole sequence, equals 1 if 7 produces correct and policy (a 8e) in the MDP is the LLM itself, parameterized by 8 and it defines probability distribution of next-token generation. We use dF to denote the distribution of the output sequence generated by me and use d d for the state-action pairs (8, a) and the state respectively. The learning objective of an RL policy is to maximize the reward function, i.e., x² = arg max me J(To).
answer and 0 otherwise. The
Proximal Policy Gradient Methods for RFT Proximal Policy Optimization (PPO) (Schulman et al., 2017) is a canonical Policy Gradient (PG) method (Sutton & Barto, 1998) to maximize [me),, which offers stable training and implementation simplicity. PPO is further developed to be GRPO (Shao et al., 2024) with a group- based advantage estimator. The policy optimization objective of PPO is formulated as:
CLIP (0) = E s,and 2. min (ro(s, a) a)A(s, a), clip(ro(s, a), 1- c, -,,
(I)
Tew (als) represents the importance sampling ratio between the current policy me and the old policy Tew, (i.e., the policy before the update), Á(s, a) is an estimator of the advantage function with GAE (Schulman al., 2016) as popular choice, and the clip ratio defines the clipping range that determines the proximity of policy update, thereby enhancing stability. When applying RL for LLM, a KL-divergence penalty is often added to prevent the policy from deviating too far from reference model Thuse, e.g., the SFT model. The complete objective is:
where re a)
(als))
L PPO (0) = -E
CL + cH [T){ + 3 grud,
(DK((TT() 8) Tasee (- 8)]]
srnd
Lgu (0:mws)
where (s) is the entropy of the policy ng at state 8, DKL is the KL metric, and c, 3 are weighting coefficients. In this work, we view both PPO, GRPO, and other variants of PPO Proximal Policy Gradient (PPG) methods.
3 REINCARNATING MIX-POLICY PROXIMAL POLICY OPTIMIZATION
In this section, we introduce our method, Reincarnating Mix-policy Proximal Policy Optimization (ReMix), for efficient and stable RFT of LLMs. Specifically, ReMix consists of three synergistic innovations: (1) Mix-policy proximal policy gradient with an increased Update-To-Data To-Data (UTD) ratio for efficient training (Section 3.1); (2) KL-Convex policy constraint to balance stability and flexibility (Section 3.2): (3) Policy Reincarnation for a smooth transition from efficient early learning to stable asymptotic improvement (Section 3.3). We introduce the three components below.
3

---

Published as a conference paper at ICLR 2026
3.1 MIX-POLICY PROXIMAL POLICY GRADIENT WITH INCREASED UTD RATIO
While proximal policy gradient methods like PPO, GRPO deliver strong performance in RFT, the on-policy nature of these methods leads to significant bottleneck data utilization. To address this inefficiency, we trace back to the off-policy RL literature. To be specific, we revisit the generalized proximal gradient theory (Queeney et al., 2021), which allows proximal gradient methods to make use of historical trajectories generated during the past policy optimization process.
In this work, we launch the renaissance of off-policy RL for RFT and introduce an On-/Off-policy Mixed Proximal Policy Gradient method (Mix-PPG) that strategically leverages both off off-policy and on-policy data within a unified objective function. Formally, for policy at training step kt the mini- batch training data are sampled from mixture of sources: the trajectories generated by historical policies (i.e., Tk=; for ~ D), and the trajectories of the current policy (i.e., T&)). This hybrid sampling strategy balances two competing purposes: (1) Data Reuse: Exploiting past trajectories reduces the autoregressive rollout and inference overhead; (2) Distribution Aligmment: Maintaining sufficient on-policy samples prevents training instability and degradation due to the divergence from the current state-action distribution. The policy optimization objective function can be formalized as:
L Mix-PPG (0) -EEww E ((aa)aatt Tk- min (rá (s, ()AAA) (s,
a), Tx s)
E, Ta 8) TT-i(a s) + AT
clip rg (s,
(3)
Ta= 8)
where i ~ with i 10, N is a combined distribution over historical policy indices - and the current policy лy (i.e., when = 0), the importance sampling ratio r m==s(o)s)) Notably, we incorporate a sampling strategy to strike a balance between training stability and efficient data utilization by using a portion p of off-policy data drawn from Tk- and 1 p on-policy data drawn from TT with E [0, 1J. A theoretical discussion on V can be found in Appendix D.2. Now, we are ready to replace the on-policy policy optimization objective, e.g., the (() term in Eq. 2, with the Mix-PPG objective M.Mis-PPi(0 (0) for efficient data utilization. One thing to note is, we found that explicitly maintaining the portion of on-policy data at a sufficient level is critical to effective training, as much off- policyesssss data will lead to a degradation or even collapse (as depicted in Figure 2).
(a(a
To further improve sample efficiency, we leverage an increased Update-To- Tata (UTD) ratio (Chen et al., 2021) ), defined originally as the number of gradient updates per environment interaction step. Specifically, we use a UTD ratio m, i.e., performing repeated gradient updates on sampled data batches for m times, thereby further reducing fresh environment interaction demands.
3.2 KL-CONVER POLICY CONSTRAINT
Conventional RFT imposes a static Kl-conntraint regularization on deviations from the base pre- trained model buse- This rigid static constraint fails to accommodate evolving policy distributions, which could lead to suboptimal updates during the dynamic learning process.
Inspired by the recent study (Ma et al., 2024b), we propose to dynamically update the anchor objective to a convex combination of - and Those On the one hand, by constraining the policy within the support of Tasee we enforce behavioral consistency with foundational capabilities, thereby preventing catastrophic forgetting of core skills. On the other hand, the constraint imposed on m serves as dynamic adaptation to the policy's 's current knowledge frontier. It facilitates iterative refinement of the policy and enables the policy to continuously evolve and improve steadily. With this mechanism, the policy can leverage the strengths of both the pre-trained model and the iterative refinement process.
Specifically, we reconcile the KL-constraint in RFT via KL-convex policy constraint (KLC), which modifies the essential optimization objective described in Eq. 2 by replacing the conventional Lxx. (0; Thase) with the KL-convex constraint term as formulated below:
LKLC((: (0; Thaso k) = E, [DDRL (-o(-)s) ) || Tosse (-Is) + (1 A) (Te(-ss) (x-1(- (4)
where > E [o. 1] balances base-model alignment and behavioral consistency with recent policy Tk-1- This convex combination preserves foundational capabilities while enabling targeted adaptation, act- ing as a conservative regularizer against ver-specializztion. We provide more theoretical discussion on KLC loss in Appendix D.3. Empirically, we found that using a decaying À consistently beat
4

---

Published as a conference paper at ICLR 2026
Space Models
ReMix
Off-policy PPG
Mix-PPG w/ high UTD
Mix-PPG
Mix-PPG
Base Model
On-policy PPG
Figure 2: The conceptual illustration of RFT for LLMs with different proximal policy gradient (PPG) methods. Starting from a base model, (1) on-policy PPG methods (e.g., PPO, GRPO) train stably, yet uses data inefficiently. it leads to collapse. (3) To strike a balance, we introduce Mix-PPG, which boosts early-stage performance but still faces slow asymptotic improvement and even collapse when adopting high UTD ratio. (4) Therefore, we propose policy reincarnation and introduce ReMix, which achieves better efficiency at no compromise of final performance.
(2) Off- oof-po PPG is data-efficient. However, naively adopting
fixed linear decay schedule works well while other forms (e.g.. exponential decay) did not make a significant difference.
3.3 POLICY REINCATENATION
While the mix- mix-policy proximal PG method proposed above accelerates carly-stage training, the off- policy bias in it inevitably limit the asymptotic performance, as the empirical evidence later shown in Figure 4. Inspired by Reincarnating RI. (Agarwal et al., 2022), we propose Policy Reincarnation in the context of RFT for LLMs, seamlessly combine the advantage of both off off-policy RL and on-policy RL, thus being more efficient at no cost of asymptotic performance.
To be specific, the training process consists of the Mix-PPG stage and the reincarnating on-policy PPG stage. First, the initial policy model is trained for predetermined T steps of gradient update according the proposed Mix-PPG algorithm for quick improvement of policy performance. Thereafter, the reincarnation happens through two changes to the training setting: (1) reset the base model from the initial reference model Thase to the current policy model कy (which alters the conventional KL constraint term), and (2) switch Mix-PPG to a on-policy PPG method (e.g., PPO or GRPO).
Finally, by composing -PG (Eq. 3), KL-convex policy constraint (Eq. 4), and policy reincarna- tion, we arrive at the complete method proposed in this paper, i.e., Reincarnating Mix-policy Proximal Policy Optimization (ReMix), as follows:
Bagg MixPPF (+) + B. LKIC(0; Thase, 1) L (0) + (H) + + LkLe(0; wr,t)
IT otherwise.
R.ReMix (0)
(5)
E B
Note that & is the number of batch training steps and the two changes that occur upon policy reincarnation are highlighted in blue and red respectively. In Eq. 5, we use PPO as the on-policy PPG method for demonstration. For the case GRPO, one can replace the advantage estimation both L Mix.PPG and L PPO with the group- based estimation, as done in our experiments.
The efficacy of ReMix is two-fold. First, it leverages the advantages of Mix-PPCC and on on-policy PPG in boosting early-stage training and stable asymptotic improvement respectively, by establishing seamless transition between the two stages. Second, the KL-convex policy constraint and the reset of the base reference model for KI. constraint (i.e., Thase -> TT) upon policy reincarnation offers a dynamics and looser constraint compared to the conventional static KL constraint, allowing fast policy training and larger policy optimization space. For an intuitive understanding, provide conceptual illustration of RFT with different proximal PG methods in Figure 2.
EXPERIMENTS
In this section, we empirically evaluate the efficacy ReMix on a range commonly adopted Math reasoning benchmarks in terms of both accuracy and efficiency (Section 4.2), along with ablation
5

---

Published as a conference paper at ICLR 2026
studies (Section 4.3) and multifaceted analyses (Section 4.4 and Appendix L). In addition, we provide extensive evaluation regarding more evaluation metrics, code generation, base RFT algorithms, etc.
4.1 EXPERIMENTAL SETUP
Training We use base models in our experiments. For implementation, we adopt PPO and GRPO as two representative on-policy proximal PG methods in our experiments, resulting ReMix-PPO and ReMix-GRPO.
and -7B (Guo et al., 2025) as the
We use unique problem-answer pairs sourced from AIME (1984-20 2023),, AMC (prior to 2023), the Omni- MATH dataset (Gao et al., 2025), and the Still dataset (Min et al., 2024). We use the DeepScaleR's s prompt format by default, instructing the LLM to follow structured step-by-step reasoning and produce verifiable {boxed{) final answer. Full templates and instances are provided in Appendix G.
(Luo et al., 2025), which comprises approximately 40,000
Our experiments are conducted using the verl' framework and the codebase derived from tinyzero?. For ReMix, we use off- off-polliy data portion p 0.4, UTD ratio m 2, historical policy window size N = 2. and we set the policy reincarnation step point to T E (50, 100) for ReMix-PPO and T = 50 for GeMOX The KL-Convex coefficient À decays with training steps i as X(t) = max(1 1 0.1 |max(t 50, 0)/10], 0.5). We use these configurations by default, except in hyperparameter analysis. Prompts are truncated 766 tokens, and the maximum generation length is 8,192 tokens. The detailed hyperparameter choices are presented in Table 8.
Evaluation We evaluate the performance of different models on series of mathematical reasoning benchmarks, including AIME'24, AMC'23, Minerva (Lewkowyez al., 2022), OlympiadBench (He et al., 2024), and MATH500 ((Lightman et al., 2023)) (None of these datasets are contained in our train set). During evaluation, we feed the entire context into the evaluation function. The models in comparison use the same generation settings as in training, except the do sample parameter set to fa se, resulting in greedy decoding. For the evaluation of baseline methods, we use the officially released checkpoints from HuggingFace to ensure fair results; for our models, we use the best checkpoints obtained within a specific training step budget, e.g., ReMix-PPo (200 Steps).
In our experiments, we focus on the evaluation of ReMix in terms of both model performance and training efficiency. For model performance, we mainly use Pass@1 1 accuracy, and Avg@32 @3 for small datasets (AIME'24 and AMC'23) in Appendix J.1. For training efficiency, we evaluate the models mainly in terms of rollout data volume, defined as the total number of rollouts generated by the model during training, which is usually the dominant source of computational cost during training in practice. We also use training steps (i.e., the number of rollout prompt batches) and training duration (i.e., the actual elapsed wall-clock time) as additional aspects for efficiency evaluation.
The detailed introduction of the compared baselines is provided in Appendix F. We also provide the discussion on related off-policy methods that are infeasible to compare with in Appendix F.3. For other training details, please refer to Appendix I.
4.2 PERFORMANCE EVALCATION FOR MATH REASONING
The performance evaluation in terms of Pass accuracy on five math reasoning benchmarks are shown in Table and Table 2, our method ReMix achieves consistent and substantial improvements over the base 1.5B/7B model on all five benchmarks. For ReMix-PPO, it achieves an average performance gain of 14.52 points and 12.31 points over 1.5B and 7B base models respectively, achieving the second-best average score for 1.5B and the best for 7B among all the baselines. In addition, compared with PPO (900 Steps. 1.5B) and PPO (200 Steps, 7B), our model achieves higher average scores within 100 steps for 1.5B and 50 steps for 7B. Similarly, our model exceeds GRPO (100 Steps, 1.5B) and GRPO (200 Steps, 1.5B) within 50 and 200 training steps, respectively. This indicates that ReMix is able to achieve competitive reasoning ability efficiently with overall no compromise accuracy and even showing a higher accuracy.
More importantly, we move on to the evaluation in terms of training efficiency. This is shown in the last volume (i.e., Cost) of Table and 2, and notably, Figure 1 illustrates the efficieccy-accuracy
http://gtthobbccm////o/nnnnn//// Phttps://gg wttp. www../////aaapaaaaacc

---

Published as a conference paper at ICLR 2026
Table 1: Pass @1 accuracy (%) and training cost (in terms of Rollout Data Volume) of 1,5B models. Bolded and underlined values denote the highest and the second-highest scores in each dataset (i.e., column). denotes that not enough information was found. ReMix achieves better average scores than both the standard PPO and GRPO in significantly more efficient manner.
Model
AIME'24
AMC'23
MATHS00
Minerva
Olympiad
Avg.
Cost
L.Disiill-QQe--lll (Base)
33.33
43.37
67.40
16.54
27.26
37.58
N/A
Open-RSI
23.33
42.17
64.20
16.18
27.11
34,60
0.058M
Open-RS2
16.67
45.78
65.00
18.38
26.96
34.56
0.029M
Open-RS3
16.67
44.58
67.60
15.64
25.48
33.99
0.029M
Adap/Think
13.33
57.83
78.60
23.90
38.07
42.35
0.643M
I1-Thought
26.67
56.63
73.00
23.16
40.89
44.07
FASTCuRL-prevee
26.67
60.24
74.20
20.22
32.59
42.78
0.676M
FASTCuRL-V3
36.67
66.27
84,40
28.67
43.56
51.91
2.478M
LI-Exact*
23.33
71.08
84,00
29.41
44,59
50.48
3.953M
I.IMMa
20.00
69.88
83,00
29.04
46.37
49.66
2.764M
DeepScaleR
40.00
65.06
83.20
29.04
43.41
52.14
2.519M
GRPO (100 Steps)
30.00
56.63
75.80
25.37
38.22
45.20
0.205M
GRPO (200 Steps)
36.67
61.45
80.00
25.37
39,70
48.64
0,410M
ReMix-GRPO (50 Steps)
23.33
57.83
80.40
26.10
39.70
45.47
0.061M
ReMix-GRPO Steps)
23.33
62.65
47.27
0.163M
ReMix-GRPO (200 Steps)
33.33
65.06
84.60
26.10
43.55
50.53
0.368M
PPO (500 Steps)
36.67
62.65
82,60
25.73
40.14
49.56
0.128M
PPO (900 Steps)
30.00
69.88
84.00
25.74
43.41
50.61
0.230M
ReMix-PPO (100 Steps)
63.86
26.84
0.020M
ReMix-PPO (200 Steps)
46.67
62.65
82,20
26.10
40.74
51.67
0.041M
ReMix-PPO (350 Steps)
36.6773.34
69.88
82,00
30.15
41.78
52.10
0.079M
trade-off in terms of rollout data volume (logı0 scale) versus average Pass @ accuracy, where the scores are out of Table I and 2 (i.e., Avg. and Cost). In the ideal case, the model should appear in the top-left corner the plot. To ensure a fair comparison, the rollout data volume square- marked models (which means the models fine-tuned upon DeepScaleR) includes the data cost of training DeepScal 1 leR itself. For ReMix-GRPO and GRPO, we report results after 200 training steps due to computational resource constraints.
Specifically, for 1.5B models, ReMix-PPO matches DeepScaleR, the strongest competitor, with just 0.079M vs. 2.519M rollouts, over 30x reduction in rollout data volume. Also, ReMix-PPO reaches 50.61 after 0.020M rollouts, 10x fewer than PPO (50.61 at 0.230M), highlighting rapid early gains. For 7B models, ReMix PPO topping AceRcason-Nemotron with over 450x reduction in rollout data volume, and outperforming PPO with a 6x reduction. Notably, the average rollout response length of ReMix is lower than the baseline models (see Fig. 4), hence the exact efficiency should be higher. The corresponding detailed factors associated with computational cost for training all compared models above are shown Appendix I.
Takeaway 1. ReMix learn strong reasoning ability in highly efficient way.
ReMix achieves SOTA-level accuracies at 1.5B and 7B scales five math reasoning bench- marks, with an over 6x to 10x reduction in rollout data volume when outperforming PPO and an over 30x 450x reduction when performing on par with (or exceeding) the best baseline.
4.3 ABLATION STUDIES
To assess the contribution of each components in ReMix, we conduct ablation studies focusing on both training dynamics and final performance. We use ReMix-PPO for the ablation studies.
The results of the ablation studies regarding @ass@1 accuracy are presented Table First, when Mix-PPG, the core of ReMix, is ablated, the method degenerates to PPO since it does not make sense any longer to apply other components of ReMix. Removing any single component among increased UTD. KL-Convex, and policy reincarnation leads to final average lower than ReMix within 500 training steps, Dropping policy reincarnation hurt the most among the three ablations. Further, when Mix-PPG works solely, it leads to even lower score.
comparable to PPO but
This reflects the off-policy nature in Mix-PPCC although it significantly increases the training effi- ciency, the off off-pollitte off-colicyness bias may hinder the convergence performance. The superiority in efficiency brought by Mix-PPG can be observed by referring to the first subplot of Figure 4: Mix-PPG shows a somewhat surprising boost of Pass @ accuracy within the first 100 training steps, which an increased
7

---

Published as a conference paper at ICLR 2026
Table 2: Pass@1 I accuracy (%) and training cost (in terms of Rollout Data Volume) of 7B models. ReMix-PPO achieves the best average score within 75 training steps.
Model
A1ME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
Cost
R1-Distill--Q Bhsill-Qwee-7B (Base)
33.33
68.68
83.80
30.15
44.44
52.08
N/A
ReasonFlux-H FF
20.00
54.22
77.20
29.04
37.04
43,50
Light-R1
30.00
67
87.00
34.56
47.56
53.08
Skywork-OR1 1.it--------eeiew
43.33
63.86
84.40
29.41
46.22
53.44
> 8.11MM
Polaris
40.00
63.86
87.60
36.40
48.00
55.17
AdapeThink
46.67
75.90
87.60
33.46
50,22
58.77
0.307M
AccRcason- -acccssoon-eoooooonnnn
60.00
80.72
89.00
36.40
50.07
63.24
>3.584
ReMix-GRPO (75 Steps)
63.88
90.60
80,72
40.07
53.78
65.81
0.046M
ReMix-GRPO (200 Steps)
64.37
91.60
81,93
39.34
53,19
66.09
0,163M
PPO (50 Steps)
33.33
71.08
87.20
36,03
48,00
55.13
0.013M
PPO (100 Steps)
40.00
77.11
90.00
35.66
$1.56
58.87
0.026M
PPO (200 Steps)
53.33
78.31
87.00
34.19
48.88
60.34
0.05
ReMix-PPO (25 Steps)
36.67
78.31
89.00
38.24
50.22
58.49
0.003M
ReMix-PPO (50 Steps)
56.60 26.66
79.52
88.60
38.97
52.59
63.27
0.007M
ReMix-PPO (75 Steps)
63.33
78.311961
90.20
37,50
52.59
64.39 112.91
0.01 1M
Table 3: Ablation studies regarding Pass@1 accuracy. The three components of ReMix work in synergy for both efficiency and final performance.
Model
AIME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
RI-Distill- Sieill--wee-1.55 (Base Model)
33.33
43,37
67.40
16.54
27.26
37.58
PPO (500 Steps)
36.67
82.60
25.73
40.14
49.56
ReMix-PPO (350 Steps)
36.67
69.88
82.00
30.15
41.78
52.10
ReMix-PPO wlo UTD
36.67
62.65
82,20
28.68
42.96
50.63
ReMix-PPO KL-Convex
81.60
42.22
49.36
ReMix-PPO w/o Policy Reincarnation
20.00
62.42
82,00
26.84
40.00
47,26
ReMix-PPP PPO w/o UTD, KL- Lonvee Policy Reincarnation
40.00
57.83
80.40
25.74
39.55
48.70
UTD further enhances it; while the KL-Convex and policy reincarnation in ReMix contribute to the stcady asymptotic improvement.
Takeaway 2. The three components work in synergy for efficiency and final performance.
Mix-PPG with increased UTD boosts early-stage training significantly, while policy reincarnation plays a critical role to ensure asymptotic improvement.
Training Curves In addition to the efficiency evaluation in terms of rollout data volume, present the training curves for ReMix-PPO and PPO in Figure 3 on Olympiad regarding two more efficiency aspects, i.e., training steps and wall-clock time. Our method demonstrates superior training efficiency by achieving score above 40 on Olympiad with a 6x and 4x reduction in rollout data volume and wall-clock time. We provide more training curves for the other four benchmarks in Appendix K.
4.4 ANALYSIS ON THE EFFECTS OF OFF-POLICY RL LLM RFT
In this subsection, we present an empirical analysis to gain better understanding of the effects of off-policy RL on LLM RFT. For convenience, we use ReMix-PPO for the analysis in the following.
To delve into the influence of off-policy RL enabled by ReMix on the reasoning behaviors during the learning process of LLMs, we make use of two more metrics: relative response length (against the training dynamics of PPO), and self-reflection rate that is calculated according to the occurrence of reflection tokens (c.g.. 'verify', 'check', 'but', 'wait', etc.). Moreover, compare PPO, ReMix-PPO, MiX-PPG and Mix- with an increased UTD. The results are shown in Figure 4.
The vanilla PPO shows steady increase of Pass @ accuracy well as decrease in response length, while maintaining a self-reflection rate near 1. Mix-PPG accelerates early training but yields inferior asymptotic performance (see more in Figure 9), with clear drop in response length and self-reflection rate. When applying an increased UTD ratio, Mix-PPG speeds further yet shows a destructive degradation after 200 steps, accompanied by sharp decrease in response length and
8

---

Published as a conference paper at ICLR 2026
Rollout Data Volume
Training Steps
Training Duration
0.450
0.490
0.450
0.425
0.425
0.425
0.403
0.403
0.000
0.325
0.325
0.325
0.300
0.303
0.300
4x
0.25
0.275
0.75
20
60
60
80
1000
120
3.00
200
300
400
500
10
20 Timethours)
50
Rollout Data Vollum(())
Steps
Figure 3: Training efficiency comparison for ReMix-PPO and PPO (1.5B) on Olympiad. We evaluate training efficiency across three dimensions: rollout data volume, training steps, and training duration. ReMix achieves score above 40%, around 4x to 6x faster than PPO.
Math500 Accuracy
Response Length
Relative Response Length
Self-Refiection
10000
0.2
5300
4300
500
0.5
-1.000
3300
-1500
2200
0.0
1300
0.5
S00
Steps
Stops
Stsps
Steps
PPO
RoMix-PPO
PPG
MIx.PPG Increased
Figure 4: Training dynamics regarding accuracy, response length, self-reflection rate for on- policy v.s. off-policy training. ReMix shows merged learning behavior and perfectly combines the superior efficiency and the asymptotic improvement thanks to the policy reincarnation.
self-reflection rate. This pathology drives the model to generate a final answer without adequate intermediate deliberation, resulting in drop of accuracy. Please see Appendix M for detailed cases.
ReMix seems to perfectly combine the strly-stage efficiency of Mix-PPG with an increased UTD and the asymptotic improvement of PPO, thanks to policy reincarnation. Throughout training, ReMix first quickly decreases the response length and improve its accuracy the early stage; it then lengthens responses and uses more reflection for careful exploration and further improvement of accuracy.
Takeaway 3. A trade-off between efficiency and final performance needs to be balanced when incorporating off off-policy training in RFT.
More off-policy training leads to a faster carly-stage boost with a larger policy shift, leading to shorter responses and quicker unlearning of self-reflection, consequently damaging reasoning performance. ReMix well leverages off-policy efficiency at no compromise of final accuracy.
To take a further step on why off-policy learning leads the observed reasoning behaviors, we conduct a formal analysis on the learning dynamics when optimizing the Mix-PPG loss function Lix Mix-PPG(()) (shown in Eq. 3). Similarly in (Fatemi et al., 2025), the average loss of Mix-PPG can formulated below:
The Implicit Preference of Off-policy Learning for Shorter Responses
H
T. Mix PPG Avg
Σ A=
I.
PPG
x
1
Σ h=0
k
TK
H
H
simple derivation is provided in Appendix D.4. With the equation above, we can find: when the advantage estimate is negative, the model learns to minimize the loss by steering its policy to achieve a lower importance sampling ratio. As the policy loss is almost always positive (as in Figure 10), the advantage estimates negative most of the time in experiments. Also, the importance sampling ratio stays above one empirically (as in Figure 9), directly amplifies the loss term. Since the average loss is computed based the data of historical policy के there apparently exists Whipping Fffect:

---

Published as a conference paper at ICLR 2026
the longer the response is, the larger the distribution shift should be on later states. Consequently, the model tends to prefer shorter responses to reduce the loss associated with long rollout trajectories. This tendency is further amplified as the proportion of off-policy data increases.
4.5
MORE RESULTS ON PASS @ K METRICS, CODE GENERATION
Pass @ K Evaluation In the experiments above, we mainly use @ass@1 1 accuracy (as well as Avg @ 32) as the evaluation metric. As advocated in (Yue et al., 2025), we extend our evaluation by using Pass@ @/16 accuracy here. The results are summarized in Table 4 and 5. We can observe that ReMix-PPO effectively improves both the Pass@8 and Pass @ 16 accuracies of the base model.
Table 4: Pass@8 Pass @ 16 accuracy (% of 1.5B models on five reasoning benchmarks.
Model
AIME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
R1-Distill-Qwae 1.5B (Base, (ass()
20.00
44.58
73.00
19.12
31.70
37.68
ReMix-PPO (Pass @8)
30.00
68.67
84.60
28.68
46.67
51.72
R1-Distill-Qwen Saaill-wee--1.55 1.5B (Base, Pass 16)
20.00
48.19
75.40
18.75
32.59
38.99
ReMix-PPO (Pass116)
30.00
72.29
86.80
30.15
46.96
53.24
Table 5: P Pass
@ 16 accuracy (%) of 7B models on five reasoning benchmarks.
Model
AIME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
RI-Distill-Qwenn-B (Basc, @ass
63.33
83.13
93.80
47.79
55.11
68
ReMix-PPO (Pase(8)
76.67
92.77
95.80
50.74
63.70
75.94
D1-Distill-wwen-1 (Base, Pass 16)
73.33
85.54
94. 80
51.10
57.19
72.39
ReMix-PPO (Pass@16)
80.00
92.77
95.40
53.31
65.33
77.36
ReMix for Code Generation Beyond Math reasoning, we move on to code generation to evaluate the domain generalization of ReMix. We use Skywork-ORI-RL-DE
Table Performance evaluation of ReMix for code generation. Pass@1 accuracy (%) on
LiveCoteBenncn8/11222/1122
(He et al., 2025) for training. We set the max prompt length to 1600 because coding prompts are longer than math tasks. We evaluate our models on LiveCoddeBencc(//11/22211
Model
LiveCodeBench
Cost
Distill-Qwen-I... (Base)
12.09
N/A
PPO (375 Steps)
19.35
0.096M
ReMix-PPO (300 Steps)
20.07
0.066M
(Jain et al., 2025) with sampling temperature of 0.7. All other experimental settings follow those in Sec- tion 4.1. The results are shown in Table For both 1.5B and 7B scales, ReMix-PPO improves PPO while using much less cost, which aligns with our observation in the Math reasoning domain. This demonstrates the generalization ability of our method across both domains.
ReMix-PPO (475 Steps)
24.01
0.111M
I1-Disiil-Qwen--- (Base)
26.16
N/A
PPO (200 Steps)
33.33
0.051M
ReMix-PPO (100 Steps)
34.41
0.015M
Other Evaluation and Analysis
We present supplementary performance evaluations to demonstrate the robustness ReMix, encompassing experiments that use GRPG as baseline Appendix J.2 and llama-series model the base model in Appendix J.3. Additionally, we provide more analysis on the performance under response length constraint, the impact of prompt format, etc. Please refer to Appendix L for complete analysis results.
5 CONCLUSION
In this paper, we aim to address the notorious drawback of on-policy RFT methods (e.g.. PPO and GRPO) on training inefficiency and prohibitive computational cost. We launch the renaissance of off-policy RL and propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix), general approach to enable on-policy RFT methods like PPO and GRPO to leverage off-policy data. In our experiments, we implement ReMix upon PPO, GRPO, and I. 1.5B-, 7B-scale base models. Through evaluating the reasoning accuracy and training efficiency of ReMix on five math reasoning benchmarks against 15 recent advanced baseline models, we demonstrate the superiority of ReMix in improving training efficiency and achieving SOTA-level reasoning performance with a great reduction training cost. Due to the space constraint, we provide the discussion limitations in Appendix C.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605. 62533021, 62541610, 92370132, 62506186), the Fundamental Research Program of Shanxi Province (No. 202503021212091) the National Key Research and Development Program of China (Grant No. (024YFE0210900) the Shanghai Municipal Science and Technology Major Project, and the Shanghai Artificial Intelligence Laboratory.
ETHICS STATEMENT
We adhere to the code of ethics and the general principles. Our study fine-tunes publicly released LLM using reinforcement learning to improve mathematical reasoning, which involves no human subjects, studies, personal data. All datasets used for training and evaluation publicly available. These resources contain no personally identifiable information or otherwise harmful information.
To mitigate potential risk, our release will be research-only. The authors declare no conflicts of interest and no sponsorship that would unduly influence the research.
REPRODUCIBILITY STATEMENT
The code is provided in the following repository nttpp:///itthu com/ An taLeungxx ReMix Off Peee-oolicyRRRe RFT. All experimental settings, hyperparameters and datasets are listed in Subsection 4.1 and Appendix I; descriptions of compared baselines can be seen in Appendix Any experiment-specificc configurations are introduced in their respective sections. All datasets, base models and baselines we used are publicly available HuggingFace.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Bellemare. Reincarnating reinforcement learning: Reusing prior computation accelerate progress. In NeurlPS, 2022.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.0440 2025.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker, Back to basics: Revisiting reinforce- optimization for learning from human feedback in llms. In ACI., pp. 12248-1 12267, 2024.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL https: //te:///hkul.... github. i 1c/blog/2025 /
Charles Arnal, GaÄTian Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, Remi Munos. Asymmetric reinforce for off-policy reinforcement learning: Balancing positive and negative rewards, arXiv preprint arXiv:2506.20522 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05822 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022b.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double qelearning Learning fast without model. In ICLR, 2021.

---

Published as a conference paper at ICLR 2026
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron. nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv:2505.164400 2025.
Taco Cohen, David W Zhang, Kunhao Zheng, Yunhao Tang, Remi Munos, and Gabriel Synnaeve. Soft policy optimization: Online off off-policy rl for sequence models. arXiv preprint urXiv:2503.05455 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Ilms: What works and what doesn't t. arXiv preprint arXiv:2503. 2025.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. JCML, volume 80, pp. 1406-1415, 2018.
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint SkKv:2554405/88. 2025.
S. Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, volume 80, pp. 1591, 2018
Bofci Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models. In ICLR, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945. 2025.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICMI., pp. 1861-1870.
Pmlr, 2018.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. OlympiadBench: challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In ACL, pp. 3828-3850, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng. Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An. Yang Liu, and Yahui Zhou. Skywork open reasoner I technical report. arXiv preprint arXiv:2505.22312. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS, 2021.
Mattco Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), AAAI, pp. 3215-3222, 2018
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint a 2021.
Intelligent-Internet li-thought.
2025.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 267. 2024.
12

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In ICLR, 2025.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating hallucination in large language models via self-reflection. arXiv preprint arXiv:2310.06271, 2023.
Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. NeurIPS, 35:3843-3857, 2022.
Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09140, 2025a.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, al. From system system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. /7419, 2025b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 0rXiv:2305.20050 2023.
Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, and Yan Zheng. Enhancing robotic manipulation with ai feedback from multimodal large language models. arXiv preprini arXiv:2402.1 14245, 2024.
Jinyi Liu, Yan Zheng, Rong Cheng, Qiyu Wu, Wei Guo, Fei Ni, Hebin Liang, Yifu Yuan, Hangyu Mao, Fuzheng Zhang, et al. From chaos to order: The atomic reasoner framework for fine-grained reasoning in large language models. arXiv preprini arXiv:2503. 15944, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol o1-preview with a 1 1.5b model by scaling rl, 2025.
Yi Ma, Hongyao Tang, Dong Li, and Zhaopeng Meng. Reining generalization in offline reinforcement learning via representation distinction. NeurIPS. 66:40773-40783 2023.
Yi Ma, Jianye Hao, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. In NeurlPS, 2024a.
Yi Ma. Jianye Hao, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. NeurlPS, 37:56215-5660 2024b.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Imitate, explore, and self-improve; A reproduction report on slow-thinking reasoning systems, 2024.
OpenAl. Gpt-3.5. Technical report, OpenAI, 2022. URL https::s:// www.errmmpppeai..commmmm com/ docs/mone foccs/modess/ g9pt-
James Queeney, Yannis Paschalidis, and Christos G. Cassandras. Generalized proximal policy optimization with sample reuse. In NeurlPS, pp. 11909-11919, 2021.
Carlo Romeo, Girolamo Macaluso, Alessandro Sestini, and Andrew D Bagdanov. Speq: Offline stabilization phases for efficient gelearning in high update-to-datt ratio reinforcement learning. In RICC, 2021.
13

---

Published as a conference paper at ICLR 2026
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Amaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufee Sándor Toth, and Sam Work. Ta- pered off- off-policy reinforce: Stable and efficient reinforcement learning for llms. arXiv preprint arXiv:2503. 14286, 2025.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICMI, volume 37, pp. 1889-1897, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In ICLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv 2017.
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning, arXiv preprint arXhv:2504.13911 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
David Silver and Richard S. Sutton. Welcome to the era of experience, 2025.
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang. Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with progressive context extension for efficient training rl-like reasoning models. arXiv preprint arXiv:2503.17287, 2025.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 178-0-222-19398-6
Yunhao Tang, Taco Cohen, David W Zhang, Michal Valko, and Rémi Munos. Rl-finetuning lims from on-and off-policy data with single algorithm. arXiv preprint arXiv:2503. 2025.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. ACI., pp. 7601-7614, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprini ar arXiv:2203. 171, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. NeurlPS, 35: 14824-24837, 2022.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du Junchen Liu, Lifu Tang, Xiaowei Lx, Haosheng Zou, Yongchao Deng. Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, DPO and RI. for long COT from scratch and beyond. arXiv preprint a5Xiv::2503. 104.0 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- off-policy guidance. arXiv preprint arXiv:2504. /4945, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang. Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprini 2505 09388, 250500988, 2025a.
14

---

Published as a conference paper at ICLR 2026
Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical Ilm reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772, 2025b.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Yang Yue, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrapping reasoning with reasoning. In NeurIPS, volume 1126, 2024.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Lim self-training via process reward guided tree search. NeurlPS, 37:64735-64772. 2024.
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505. 2025a.
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, al. Srpo: A cross-domain implementation of large-scale reinforcement learning on Ilm. arXiv preprint urXiv:2504. 14286, 2025b.
15

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
The Use of Large Language Models (LLMs)
17
Related Work
17
C
Limitations
18
D Algorithm and Related Discussion
18
D.I The pseudocode of ReMix
18
D.2 Theoretical Discussion on The Historical Policy Distribution
18
D.3 Theoretical Discussion The KL-convex Loss
19
D.4 The Derivation of Equation 6
19
D.5 Sensitivity Analysis of The Choices of Policy Reincarnation Trigger Step T
20
Advantage Estimation
20
A Brief Overview Baseline Models
F.1
1.5B Models
21
F22 Models
F3 Exclusion Rationale for Off-Policy Baselines
22
G System Prompt
22
H Key Observations from Figure 1: Efficiency-Accurayy
22
I Training Details
23
J Supplementary Performance Evaluation
23
J.1 Extended Evaluation of AIME'24 and AMC'23
23
J.2 ReMix for Dr.GRPO
................................
25
J.3 ReMix for Llama-series Base Model
26
K More Training Curves
26
Various Analysis
28
L.1 The Performance under Constrained Maximum Response Length
28
L.2 The Impact Guide Tokens Prompt Template
M Case Study
30
16

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs only for coding and writing assistance. During experimentation, we consulted the LLM for code debugging. All algorithmic designs, implementations, and results were produced and verified by the authors, and LLM suggestions were reviewed and tested. After completing the manuscript, we used L.MS solely to polish the language (grammar and phrasing) without generating new scientific content. The authors remain fully responsible for the paper's 's contents.
B RELATED WORK
Post-training enhancement of LLM reasoning capabilities predominantly follows two paradigms (Li et al., 2025b). The first, inference-time optimization, improves reasoning without updating model parameters through techniques like Chain-of- Chain-of-Thought (CoTT) prompting (Wei et al., 2022), parellel reasoning and itegration (Wang et al., 2022), self-reflection (Ji et al., 2023), tree-based search (Zhang et al., 2024), and macro-action-guider cognitive reasoning (Liu et al., 2025a). Despite their effec- tiveness, the performance of these methods is fundamentally constrained by the model's inherent capabilities. The second paradigm, parameter fine-tuning, aims to enhance these intrinsic abilities into LLM. While SFT on high-qualit, reasoning data is a common approach, its effectiveness is often limited by data availability and scalability (Zelikman et al., 2024). Consequently, RLVR has emerged as a powerful alternative, learning directly from reward signals to unlock superior performance, as demonstrated by models like DeepSeek-R (Guo et al., 2025). Notably, this differs from preference- based RL which learns from reward model trained on human/AI feedback (Bai et al., 2022a;b; Liu et al., 2024), as the RLVR here utilizes direct, verifiable reward signals. Our work is situated within the RFT paradigm, especially under varifirable reward.
The majority of existing RFT research has relied on on-policy RL algorithms prized for their training stability, such as PPO (Schulman et al., 2017). Some recent approaches have sought to improve efficiency by modifying the RL architecture (e.g., GRPO (Shao et al., 2024)) or relaxing optimization constraints (Seed al., 2025). However, these on-policy RL methods exhibit severe sample inefficiency, as they require fresh samples for each iteration of gradient updates. To alleviate this, recent research has begun to incorporate off-policy data in RL training. Tang al. (2025) propose AGRO for unified algorithm to leverage generationn data, encompassing both on- and off-policy samples. However, their experimental results show that off-policy training is inferior to on-policy training, underscoring the non-trivial challenge of achieving stable and effective off-policy training for L.LMs. TTapered Off-Policy REINFORCE (Roux et al., 2025) introduces a novel variant of importance sampling to downweight negative trajectories that are not likely under the current policy, while allowing positive trajectories to be upweighted. This enables the utilization of both off-policy and on-policy rollout trajectories. The method is trained and evaluated on GSM8K and MATH, leaving its efficacy on broader reasoning tasks unknown.
Recently, concurrent to our work, Based REINFORCE, AsymRE (Arnal et al., 2025) is proposed to leverage both off- ool-co and on-policy data by introducing a tunable baseline. An asymmetry is presented that while on-policy updates safely leverage both positive and negative signals, off-poliey updates benefit more from positive rewards, which to some extent echoes the idea proposed in (Roux et al., 2025). AsymRE is trained and evaluated on MATH. SRPO (Zhang et al., 2025b) builds on GRPO with a two-stage curriculum that first trains on math and then on code. Besides, SRPO adopts historical resampling discards groups with uniform rewards to avoid zero gradients and retains hard samples for later replay. Using the same base model, SRPO outperforms DeepSeek-PR R1 Zero--Qeeer 32B while using only one-tenth of the training steps. Similarly, RePO (Li et al., 2025a) also exploits historical data. SRPO's resampling chiefly targets the quality of samples, while RePO emphasizes efficiency and systematically analyzes the impact of replay strategies. RePO proposed upon GRPO to replay both historical off-policy data and on- on-policy data together during typical GRPO training. Different off- policy data replay strategies are studied, among which recency-base based and reward- strategies show improved performance. The RePO models trained with a maximum response length of 1,024, thus showing limited performance on math reasoning benchmarks.
By following the principle of Soft RL, SPO (Cohen et al., 2025)) is proposed to leverage both off- policy and polloy data based on Cumulative Q- P-Paramtterization SPO is trained and evaluated for code contests and demonstrates superior performance the standard PPO. In contrast, LUFFY (Yan
17

---

Published as a conference paper at ICLR 2026
al., 2025) uses off-po samples from superior models (e.g.. DeepSeek-RR R1) and employing policy shaping. However, in essence, this is more akin to learning from demonstrations rather than the canonical off-policy RL where the behavior policy is often one of the historical policies separate inferior policy. Moreover, the idea of off-policy guidance is orthogonal to our method.
While early efforts have conducted first-step explorations realizing off-policy learning for RFT, they have primarily focused on adapting existing on-policy methods (e.g., PPO, GRPO, REINFORCE) off-policy data from the angles of modifying importance sampling, leveraging data or trajectories asymmetrically, etc. These initial steps have not investigated the essential effects of off-policy learning on reasoning behaviors, while leaving the potential of existing off policy RL techniques unexplored. In the broader field of RL, methods such as Rainbow (Hessel et al., 2018), TD3 (Fujimoto et al., 2018), and SAC (Haarnoja et al., 2018) have set a precedent for leveraging historical data to improve sample efficiency. Building on this, advanced research has pursued maximizing data utilization through high UTD ratios, managing the resultant estimation errors with techniques like ensemble learning, as seen in REDQ (Chen et al., 2021), DroQ (Hiraoka et al., 2021), and SPEQ (Romeo et al., 2021). Concurrently, novel approaches have emerged, including hybrid methods that seek an optimal balance between the stability of on on-policy learning and the efficiency of off off-policy methods (Queeney et al., 202 I), as well as fully offline algorithms designed to mitigate extrapolation errors from static datasets (Ma et al., 2024b; 2023). The value of ReMix lies in its departure from simply implementing off-policy RL in the context of RFT. Instead, by drawing inspiration from rich RL literature, our research aims to conduct an in-depth investigation of different off-policy RL techniques and integrate them to improve the RFT process effectively, thereby significantly enhancing the efficiency and performance of LLM fine-tuning.
C LIMITATIONS
Due to resource constraints, our experiments were limited to models up to 7B. While this provides a strong proof-of- performance on larger-scale models is yet to be explored. To support this future work, provide open-source code and models for community validation. We also note that the principle of using off-policy data to improve sample efficiency is general and not inherently tied model scale. For the utilization of off-policy data, we fixed proportions in this work, although we believe an adaptive control on the proportion of off-policy data should be possible and favorable. Moreover, our method is orthogonal to many of the advanced RFT methods considered and not considered in our experiments, while we do not explore the combination of them. We believe that integrating off- policoliy learning and other advanced techniques is promising to realize new LLM models that are more efficient and powerful at the same time. We leave these potential angles for the future.
D ALGORITHM AND RELATED DISCUSSION
D.1 THE PSEUDOCODE OF REMIX
The pseudocode of ReMix is presented in Algorithm 1.
D.2 THEORETICAL DISCUSSION ON THE HISTORICAL POLICY DISTRIBUTION
The distribution of policy index i, where i E {0., 10. N). is denoted by v. The theoretical explanation for the influence of different choices of v should trace back to how Trust-Region Policy Optimization (TRPO) (Schulman et al., 2015) approximates the trust region.
For i = 0, i.e., the standard PPO (on-policy case), the trust region a² |max, (T)) or its upper bound ( т) (this is can be found Theorem 1, Eq.8 in TRPO paper) is approximated by using the expectation regarding T to replace the maximum case. In turn, the approximate trust region is d² = [E TV (Tk, (s)]? In the practices of PPO and TRPO, this approximation works well in many problems. Intuitively, this is because the new policy (post-update) should not differ a lot with the current policy Te (prior-update), the distribution dt works as an effective surrogate.
18

---

Published as a conference paper at ICLR 2026
Algorithm 1 Reincarrating Mix-Policy Proximal Policy Gradient Method (ReMix)
1: [Input]: Base model Mhsse and on on-policy proximal PG method & (e.g.. PPO, GRPO)
Set training batch size off-policy data portion p. UTD ratio m. historical policy window size N. policy
reincamation step point
3: Init the model T. Whsse and the historical policy set F =
4: # Stage I: Mix-policy Proximal PG Training
5: for step
2, 3,... T do
Sample a batch questions y ~ Do with size (1 - p) and generate fresh responses according to ms
and A
7: Reuse historical responses from H with size pB and form the mixed training batch
8: Save H with its responses and logprob data, drop the oldest policy if [H] > N
Repeatedly update ITe with the mixed training batch according Mix-PPG and (the first row, Eq. 5)
for m times
10: end for
11: # Stage Transition: Policy Reincarnation
12: Reset the base reference model from These ny, and drop the historical policy set HH Stage 2: Reincarnating On-policy Proximal PG Training 14: for step / = 7 + 1, + 2, + 3,... do
15: Sample a batch of questions Du with size B and generate responses according to ma and A
16: Construct training batch with the fresh responses, and update according to A (the second row, Eq. 5)
end for
When it moves on to the off-policy case where 0 (Queeney al., 2021), the approximation for the trust region could be no longer effective for stable policy optimization due to the increasing discrepancy between d and d the increase of (i.e., for older historical policies). our work, empirically found that using too old historical policies can introduce large off-policyness which makes the training unstable.
Our empirical observation aligns with our discussion the theoretical explanation of the approxima-
tion of trust region above. Therefore, in practice, we use N = 2, p = 0.4 and a uniform distribution for off- aff-policy data. This is equivalent to the policy index distribution that one important future direction is to study how to replay off-policy data better instead of replaying in uniform manner.
[0.6, 0.2, 0.2]. We believe
D.3 THEORETICAL DISCUSSION ON THE KL-CONVEX LOSS
The KL-convex loss is theoretically grounded in Conservative Policy Iteration (CPI) (Ma et al., 2024a). The CPI paper proves that iteratively refining the reference policy guarantees monotonic improvement (Proposition 1: V (s) V (s)) and support preservation (#* (a|s)
wherever ((als) 0). Our convex combination LKLC ADR1. (Tal Thase) + (1 DKI. (T T 1) directly implements this where the π&-1 term acts as CPI's dynamic reference policy prevent OOD queries, while the Tbase term extends the framework to preserve foundational capabilities.
Furthermore, CPI's Theorem identifies that multi-step actor-critic implementations suffer from high variance, and the authors explicitly recommend adding behavior regularization (their Eq. 6) to constrain policies to data support. Our decaying X(t) schedule dynamically balances this off shifting from conservative exploration to aggressive refinement. Thus, KLC inherits CPI's theoretical guarantees of improvement and convergence while addressing practical stability challenges in iterative LLM fine-tuning.
D.4 THE DERIVATION OF EQUATION
We conduct the formal analysis of off-policy PPG in Section 4.4 by using a similar form of average loss in (Fatemi et al., 2025). Here, we provide the complete derivation of Equation 6 below.
Starting from Equation 3, that is the definition of the loss function for Mix-PPG, we rewrite the loss function from the original per-sample expectation (i.e., (s, a) d=+) form the per-trajectory
19

---

Published as a conference paper at ICLR 2026
expectation form (i.e., d= This derives:
L Avg
(ix) Biny En 1 -H-
Hr hh= min (10 (s), ag) aA)Aat (sh,
(7)
clip r0 (sb: un):
(an sh))
- (an 8%) +e) (sh, an))]:
(an 8h)
T 18))
where H, is response length for trajectory т. By dropping the clipping range (i.c., ignoring the out- of-clipping range cases which have gradient) and simplifying the expression by omitting subscript notations, expectation notations, etc., we only keep the proportional relationship for the analysis. This then leads to: L Mix-PPG 1 I/l k AA* which is the exact form used in Equation 6.
Avg
o
D.5 SENSITIVITY ANALYSIS OF THE CHOICES OF POLICY REINCARNATION TRIGGER STEP T
our pipeline, T the hand-over step that switches from the off-policy Mix-PPG phase to the on-policy phase with reference model changed too. We use it to harvest early data-efficiency from replay historical data and then let on on-policy optimization continue improving stably. To address the reviewer's s concern about how the trigger step T chosen and how sensitive performance is to this choice, we provide a controlled sweep T E {25, 50, 100, 200} with 500-step runs, whose experimental settings
all the same those of the main text. The results shown in the table below. For (T' = 25), we report two checkpoints per run, the first time the macro average reaches 49 ("early lift") and the final plateau
The pattern is consistent: a very small T (25) slows down performance improvement; a very large T (200) lifts early but later stalls due amplified off-policyness; a moderate T (50-100) yields both strong early lift and the best final averages. We also find this choice to broadly work well across our experimental cases, achieving favorable
trade-off
Table 7: Pass @ accuracy (%) of 1.5 model under different policy reincarnation trigger step T.
Model
AIME'24
AMC'23
MATH1500
Minerva
Olympiad
Avg.
T 25 (at 350 steps)
33.33
65.06
81.20
27.57
39.26
49.29
T = 50 (at 75 steps)
43.33
63.86
79.60
26.84
39.41
50.61
T = 50 (at 500 steps)
40.00
63.86
83.00
26.84
43.70
$1.52
T 100 (at 225 steps)
30.00
69.88
81.80
24.63
41.78
49.62
T 100 (at 325 steps)
36.67
69.88
82.00
30.15
41.78
52.09
200 100 steps)
30.00
66.27
81.00
29.41
43.26
200 (at steps)
16.67
49.40
77.00
16.67
38.67
40.76
E ADVANTAGE ESTIMATION
To enable stable off-policy training, adopt V-trace (Espeholt et al., 2018) formulation for generalized advantage estimation (GAE) (Schulman et al., 2016), which incorporates truncated importance sampling ratios correct for policy mismatch. We first compute the temporal-difference enror(TD-error)) at each time step / as
Ry r(s at) 7V(s++1) V(st,,
8
and define the truncated importance sampling weight e = min mu(als) where is a clipping threshold limit the variance of the correction, we use I in our implement.
x2-1(ols)
The advantage at step is estimated recursively using the V-trace correction as
A 5.¹ +Ac4A4+1 and the return-to-ge to-go is computed by combining the advantage estimate with the baseline value:
RTG, A, + V(s)..
(10)
This V-trace corrected GAE formulation ensures that the estimated advantages remain stable and consistent under significant off off-policy drift, which is critical in our training regime involving long- horizon trajectories and evolving policies.
20

---

Published as a conference paper at ICLR 2026
F A BRIEF OVERVIEW OF BASELINE MODELS
F.1 1.5B MODELS
Open-RS Series (Dang & Ngo, 2025): The Open-RS series employs the GRPO algorithm to train language models, using datasets constructed by filtering and combining existing corpora. Specifically, Open-RS1 utilizes dataset with 18,615 samples with accuracy and format rewards, Open-RS2 incorporates dataset with 7,000 samples and shorter maximum response length while retaining the same reward functions. Compared to Open-RS2, Open- RS3 replaces the accuracy reward with cosine reward and adds an English-only instruction to the system prompt.
DeepScaleR (Luo et al., 2025): DeepScaleR is obtained via two-phase training process with the GRPO algorithm: starting with 8k context for efficient reasoning, then scaling up to 16k and 24k contexts address more challenging problems.
II-Thought (Inteliigent-nntemee) 2025): Based on systematic analysis of existing public datasets, the authors constructed a large-scale, high-quality dataset comprising over 300,000 reasoning problems across multiple domains. Each sample was rigorously filtered and deduplicated. Subsequently, the models were trained on this curated dataset, using the GRPO algorithm.
FastCuRL Series (Song et al., 2025): The FastCuRL Series adopts multi- stage training process where both context length and data complexity (defined by input prompt length) are progressively increased. Training starts with short-context and low-complexity data, then moves to longer contexts with medium and high-complexity datasets.
L1 Series (Aggarwal & Welleck, 2025): The L1 Series trains models using Length- Controlled Policy Optimization (LCPO), method that encourages correct answers while matching target output length specified in the prompt (measured by input prompt length). L1-Exact enforces exact-length generation by penalizing deviation from the target length, while L1-Max applies a soft maximum-length constraint, allowing shorter outputs when appropriate but discouraging overruns.
Adap'Think (Zhang et al., 2025a): AdaptThink is an RFT method that trains reasoning models choose between two modes Thinking and NoThinking based on problem difficulty. It uses a constrained optimization objective to encourage NoThinking while maintaining performance, and an importance sampling strategy to balance both modes during training.
F.2 7B MODELS
ReasonFlux-Fi (Yang et al., 2025b): ReasonFlux-F1 is an SFT model obtained by fine- tuning R1 RI-Dittill model based cooplate-aumenter reasoning trajectories collected by ReesonFFlu--l.. These trajectories are first enhanced with structured templates, then transformed into a long chain-of-thought format.
Light-R1 (Wen et al., 2025): Light-RI is multi-stage post-training framework. It be- gins with curriculum-based supervised fine-tuning (SFT) using progressively harder data, followed by Direct Preference Optimization (DPO) and an RFT process with GRPO filtered dataset. Light -ant--R1-- R1-7B-DS is trained only in the second SFT stage of the frame- work. Thus, the Light-R1 7B baseline model used in our experiments is an SFT model rather than an RFT model.
Skywork Sawwork-RR-PEE ORI-Preview (He et al., 2025): Skywork-OR! -reeview is trained on a curated dataset of math and coding problems. selected through model-aware difficulty estimation and quality filtering. The training process modifies GRPO by incorporating both offline and online difficulty-basee filtering, rejection sampling, and multi-stage curriculum with adaptive entropy control.
Polaris (An et al., 2025): Polaris adopts multi multi-stage RL training approach with careful data difficulty control, using a data distribution with a slight bias toward challenging problems and
Phtpps:/ / thub. com Vereenneeeeeeeeeeeeeeeeee Reaeee/Reeonn" ux /bx/bllo /mai RReasson F1 Plux uuu___ P1 /README.
md
21

---

Published as a conference paper at ICLR 2026
dynamically adjusting question difficulty during training. It initializes sampling temperature based on rollout diversity and gradually increases during training. It employs length extrapolation techniques, enabling longer CoT generation at inference while keeping training rollouts short.
Adap'Think (Zhang et al., 2025a): The methodology for the AdaptThink 7B model identical to that of the AdaptThink 1.5B model, previously described.
adopts the GRPO al- gorithm without KI. divergence and avoids entropy collapse through controlled updates. The model is first trained on math-only prompts, then on code-only prompts, following curriculum with progressively increasing response lengths.
AceRcason-Nemotrro (Chen et al., 2025): AceReason-Nemmnn
F.3 EXCLUSION RATIONALE FOR OFF-POLICY BASELINES
For existing off-policy RFT methods, we do not include RePO (Li et al., 2025a) because their models are trained under a maximum response length of 1,024 tokens, thus showing limited performance on math reasoning tasks. We do not include LUFFY (Yan et al., 2025) since the usage of off- policy guidance from a superior model (e.g.. DeepSeek-R1) is orthogonal to ReMix, which is also viewed different setting where extrinsic guidance demonstrations are accessible. We exclude SRPO (Zhang et al., 2025b) since its publicly released model is not at the same scale as ours. In addition, we did not find public checkpoints for SPO (Cohen et al., 2025) (which is also trained for code contests), AGRO (Tang et al., 2025), AsymRE (Arnal et al., 2025) and Tapered Off-policy REINFORCE (Roux al., 2025), thus, we do not include them in our experiments. Please refer to Section B for detailed discussions on related off-policc RFT methods.
G SYSTEM PROMPT
Following the standard DeepScaler data processing approach, each prompt in the training set was prefixed with "< "<[User|>> and suffieed with the instruction "Lett' think step by step and output the final answer within {boxed < Assistant/><think>>
This structure encourages the model engage in step-by-step reasoning and produce final answers encapsulated within LaTeX boxed expressions. One example of the DeepScaler prompt format is shown below. The blue text indicates the fixed template used during inference, while the black text represents the uistace-speeiffe question inserted into the prompt.
System Prompt (Standard)
< of >> < User] Xenia and Sergey play the following game. Xenia thinks of a positive integer N not exceeding 5000. Then she fixes 20 distinct positive integers a1 .2...., O2n such that, for each & = 1,2,... ,22...,,0, 20, the numbers N and ag are congruent modulo k. By move, Sergey tells Xenia and she tells him back the set {ax k S} without spelling out which number corresponds which index. How many moves does Sergey need to determine for sure the number Xenia thought of? Let's think step by step and output the final answer within {boxed >
S of positive integers not exceeding 20,
H
KEY OBSERVATIONS FROM FIGURE 1: B#FICIENCY-ACCURGE
For clarity, summarize the major observations in Figure 1 below:
(1.5B) ReMix-PPO v.s., DeepScaleR: DeepScaleR, the strongest 1.5B competitor, requires around 2.519M rollouts to reach its final score (i.e., 52.14), whereas ReMix-PPO (350 Steps) achieves comparable score (i.e., 52.10) with 0.079M rollouts over 30x reduction in rollout data volume.
(1.5B) ReMix-PPO v.s., PPO: We trace the performance of ReMix-PPO at 100, 200, and 350 training steps (denoted by the yellow curve in Figure 1), corresponding to rollout data volumes of roughly 0.020M, 0.041M, and 0.079M, respectively. Even after generating just
22

---

Published as a conference paper at ICLR 2026
0.020M rollout samples, ReMix- PPO achieves score of 50.61, which has already surpasses most baselines. Compared to PPO (900 Steps), which achieves an average score of 50.61 with 0.230M rollouts, our model shows a 10x reduction in rollout data volume.
(1.5B) ReMix-GRPO v.S., GRPO: We also trace the performance of ReMix-GRPO at 50, 100, and 200 training steps (denoted by the cyan curve in Figure 1), corresponding to rollout data volumes of roughly 0.061M, 0.163M, and 0.368M, respectively. After generating 0.061M rollout samples, our model achieves the score 45.47 that exceeds the score 45.20 of standard GRPO trained for 100 steps with 0.205M rollout samples. Compared GRPO (200 Steps), which achieves an average score of 48.64 with 0.410M rollouts, ReMix-GRPP achieves a much higher score of 50.53 within 200 training steps, i.e., 0.368M rollouts, showing superior final performance with less computational cost.
(7B) ReMix-PPO v.S., AceReason-Nemotte baseline method our comparison, requires over 3.584M rollouts to reach its final score (i.e., 63.24)*, whereas ReMix-PPO (50, 75 Steps) achieves slightly higher accuracy (i.e., 63.27, 64.39) with 0.007M, 0.011M rollouts - over a 450x reduction in rollout data volume.
AceReason -enisonnne the strongest 7B
(7B) ReMix-PPO v.s., AdaptThink: Adap(Think, the second strongest 7B baseline method, requires around 0.307M rollouts to reach its final score (i.e., 58.77), whereas ReMix-PPP -PO (25 Steps) achieves a comparable accuracy (i.e., 58.49) with 0.003M rollouts over 80x reduction in rollout data volume.
(7B) ReMix-PPO v.s., PPO: Compared to PPO (200 Steps) that achieves an average score of 60.34 with 0.051M rollouts, ReMix-PPO achieves higher score of 63.27 within 50 training steps, i.e., 0.007M rollouts, showing a 6x reduction in rollout data volume.
Besides, when amppam 1e is set to tuue, Open-RS series models (i.e., RSI,, RSS2, -RS3) show better scores 40.62, 40.08, 39.31 respectively, and II TIoohhhht can achieve score 51.474. For other models, we found similar scores in our experiments, which do not change the conclusions.
I TRAINING DETAILS
Hyperparmeters
The major hyperparameter choices are shown in Table 8.
Compute Resource The 1.5B model was trained for 50 hours on 2 NVIDIA A800-SXM4-80GB GPUs, while the 7B model required 75 hours on such GPUs. The evaluation of each model was also conducted using the same number of GPUs in their respective training setups.
Comparison of Training Detail Computational Cost for 1.5B Models The corresponding detailed factors associated with computational cost for training the 1.5B models in the comparison above are shown in Table 9. Compared most baselines,
method uses nearly half the number of training steps (500 v.s. 2 860) while delivering superior performance. Furthermore, our entire training run is executed on single node with just two A800 GPUs over 52 hours, amounting to 104 A800 GPU hours. This finding shows that state-of-the-ar gains can be achieved with markedly reduced compute requirements.
Comparison of Training Detail on Computational Cost for 7B Models Table 10 shows the training details of 7B models. However, we failed to find complete training details for all the 7B models, so we did not plot the peffciency- trade-off for the 7B models due to missing information.
J SUPPLEMENTARY PERFORMANCE EVALUATION
J.1 EXTENDED EVALUATION OF AIME'24 AND AMC'23
For two small and hirinnariance benchmarks (i.e. AIME'24 and AMC"23), we report Pass@1 as the
average over 32 independent runs (Avg@ 32). Here we use use stochastic decoding (temperature
'The score of MecRReason-Nemotte is obtained by evaluating the official checkpoint, and the rollout data volume is estimated according the text and Figure in (Chen et al., 2025).
23

---

cost and the resources reported in Published as a conference paper at ICLR 2026
Table Hyperparameter setups for PPO, GRPO and ReMix trainer.
Parameter
Value
Training Configuration
temperature
1.0
top-p
1.0
top-k
-1
critic_warmup
0
learning
le-6
clip_ratio
0.2
lam
1
tau
0.95
entropy coeff
0.001
clipping gradient
true
do_sample
true
test.freq
Training Configuration for ReMix-GRPO and GRPO
kl loss loss.coef
0.001
kl loss type
low_var wwrvaa_.. kl
(gen per prompt)
8
Table 9: RFT training details associated with computational cost for 1.5B models. All the models are trained upon DeepSeek- -ieek-R---a faistiil-Qeen.. base model, except for LI series models, which are fine-tuned on top of DeepScaleR (denoted by superscript *). Accordingly, their total training cost should be considered as the sum of DeepScalet this table. Italicized entries indicate values not directly reported in the original papers, but instead retrieved from associated official training scripts. The underlined values denote the fresh on nolpollcc rollout in addition off-policy data reuse in ReMix.
Model
Training Steps
Rollout Batch Size
Gen per Prompt
Responses Length
Number GPUs
DeepScaleR
1750 steps
128, 128, 128
8616,16
16k.24k
8,32,32
FASTCuRL--pevicn
860 steps
128.64.64.64
8,8,8.16
1k.16k.244.166
8
FASTCuRL-v3
2620 steps
128,64,64,64464
8,8,8,6,116
16k,
I1-Thought
1024
5
32k
adapt think
314 steps
128
16k
Open RSI
100
96
4k
Open-RS2
50 steps
96
4k
Open-RS3
steps
4k
1.1-Exact*
700 steps
128
16
4k
8
L1-Max*
120 steps
128
4k
ReMix-PPO
500 steps
152,256
8k
ReMix-GRPO
200
152,256
8k
Table 10: RFT training details associated with computational cost for 7B models. All methods are trained upon wiw.seeekk----ssiiiii iilisttl1--ee-e base model. Italicized entries indicate values not directly reported in the original papers, but instead retrieved from associated official training scripts. The underlined values denote the fresh on-policy rollout in addition to off-policy data reuse in ReMix. Note that ReasonFlux-FF and Light-RI (7B) are SFT models as detailed in Appendix F.2, hence do include them in this table.
Model
Traing Steps
Rollout Batch Size
Gen per Prompt
Maximum Responses Length
Number of GPUs
Skywork ORI-Preview
2000 steps
256
Sk 16.
AccReason-Nemoolee
2000 steps
128
1,16,16,16
84k16k.24k.32k
128
Adap(Think
steps
128
16k
Polaris
1400 steps
16k.24k_32k
ReMix-PPO
steps
152.256
24

---

Published as a conference paper at ICLR 2026
0.1) for evaluation. We include these results in Table 11 and Table 12 provide robust estimate small datasets. ReMix-PPOO maintains the similar performance observed Table and Table under greedy decoding, showing consistent gains under the Avg @ 32 protocol.
Table 11: @ accuracy (%) on AIME' 24 and A3 (Avg @ 32) of 1.5B models. ReMix- PPO shows performance consistent with the greedy-decoding results (Table 1). Bolded and underlined values denote the highest and the second-highed scores in each dataset (i.c., column).
Model
AIME'24
AMC'23
RI-Distill-Qwen- 1.5B (Base)
17.92
44.58
Open-RS1
17.60
45.44
Open-RS2
17.62
44.92
Open-RS3
18.65
43.98
AdaptThink
19.06
56.81
I--Thought
28.65
59.72
-ASTCuuR--rreview
25.94
54.27
FASTCuRL-V3
33.44
63.63
L1-Exact*
25.1
66.57
L1-Max*
23.13
66.79
DeepScaleR
31.96
63.58
ReMix-GRPO (75 Steps)
25.10
60.17
ReMix-PPO (350 Steps)
29.08
64.04
Table 12: @ accuracy (%) on AIME'24 and AMC'23 (Avg@33) (2) of 7B models. ReMix-PPP reach SOTA-level on both benchmarks.
Model
AIME'24
AMC²23
R1 R1-Distill-Qwen-7B (Base)
37.53
66.55
ReasonFlux-F1
20.19
53.07
Light-R1
40.00
66.73
Kkywork-ORI-Prees
36.31
61.60
Polaris
39.71
67.40
AdaptThink
47.62
74.20
AceReason-Nemorte
50.00
77.48
ReMix-GRPO (200 Steps)
49.48
82.12
ReMix-PPO (75 Steps)
50.31
77.78
J.2 REMIX FOR GR.G
To further examine generalization ReMix regarding different base algorithms, we evaluate effect of ReMix based on GRPO (Liu et al., 2025b). For the convenience of experimentation. use Deepsea--ll-Distt11----
5B as the base model and set the max response length to 4096. We evaluate the models with sampling temperature of 0.7. All other experimental settings follow those in Section 4.1. The results show that ReMix-Dr.GRPP delivers consistent improvements DR.G with higher efficiency, mirroring the finding in main evaluation above. This demonstrates the generality of ReMix as an effective drop-in method.
Table 13: Performance evaluation of ReMix based on Dr.GRPO. Pass @ I accuracy (%) on five benchmarks (Avg@32 for A1ME The max response length is 4096.
Model
AIME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
Cost
RI-Distill 11-Disill-QQeen-155D (Base Model)
9.38
32.42
63.40
18.75
22.67
29.32
N/A
Dr.GRPO (350 stops)
21.25
63.10
79.40
27.21
41.33
46.48
0.358 M
Dr.GRPO (400 steps)
26.25
59.94
80.60
26.57
41.03
46.86
0.409
ReMix-DE:GRPO (325 steps)
26.25
63.25
81.60
27.21
41.63
0.291
ReMix-D.GRPO (400 steps)
28.75
62.65
82.00
27.74
42.96
48.42
0.368 M
25

---

Published as a conference paper at ICLR 2026
J.3 REMIX FOR LLAMA-SERIES BASE MODEL
In addition to the Qwen-series base model used above, we also evaluate the effect of ReMix based on DeepSeek- -i----1-- -D--sssti sisssilll 1ll-lamma 8B in Table 14 in the appendix. Similarly, found that ReMix-PPO improves the performance of the base model and outperforms PPO in terms of both score and efficiency.
Table 14: Pass @ accuracy (%)
RI-Distil--LlamaB
Model
24ME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
Cost
R Dl-Distil-Llama--B (Base Model)
26.46
62.69
82.00
25.37
42.67
47.84
PPO (50 steps)
34.58
70.63
85.40
26.10
45.18
$2.38
0.013
ReMix-PPO (25 steps)
39.48
88.00
87.04
48.74
55.99
0.004
K
MORE TRAINING CURVES
Training Curves for Efficiency Comparison In addition to the efficiency comparison between ReMix-PPe and PPO for Olympiad Figure 3. the remaining curves for the other four math reasoning benchmarks are presented in Figure 5, 6, 7,
Rollout Data Volume
Training Steps
Training Duration
Pass-21
0.26
0:26
0.26
0.20
0.20
0.20
0.28
600
0:18
2...8-...
0.18
3x
20
40
1000
120
202
300
400
500
10
20
40
50
Rollout Data Valumee()
Steps
Time(hours)
RoMixxPPO
PPO
Figure 5: Training efficiency comparison for ReMix-PPO and PPO (1.5B) on Minerva.ReMix achieves a score above 26%, around 3x 6x faster than PPO.
Training Steps
Training Duration
0.65
0.65
M
0.60
0.60
0.50
Posg03
0.55
0:55
0.55
0.30
0.30
0.30
0.45
0.45
0.45
0
20
40
60
RG
150
120
0
1.0
203
300
400
500
10
20
4.
50
Rollout Data Volume(K)
Stops
Time(hours)
ReMix-PFO
Figure 6: Training efficiency comparison for ReMix-PPO and PPO (1.5B) on AMC'23, ReMix achieves a score above 60%, around 3x to 6x faster than PPO.
Training Curves under Varying Proportions of Off-policy Data We vary the off-policy propor- tion E {0.1, 0.2, 0.3, 0.4, 0.5} (the UTD ratio is set to I here for isolation) to isolate the effect of historical data reuse. In addition the Pass@ I accuracy, we use three more metries: the importance sampling ratio rg k-i ratio =e_ that quantifies the distributional shift between current and historical policies, and the response length that reflects the reasoning behavior of the model.
For importance sampling ratio, we observe that larger yields slightly wilder importance-sampe swings. Notably, the consistently slight decrease of the importance sampling Figure can also be explained by the shortening of response length, as the whipping effect (detailed in Subsection 4.4) gradually diminishes.
Training Curves for Policy Loss Figure 10 shows that during the training process, the policy loss predominantly remains positive, which means a larger importance ratio will lead to larger policy
loss.
26

---

Published as conference paper at ICLR 2026
Rollout
Training Steps
Training Duration
0.35
0.35
Pasg01
0.30
0.30
0.30
0.25
0.25
3xx
20
1.00
120
1.00
200
300
500
Rollout Data Valumne())
Steps
Timeihhours)
ReMMxxPPO
Figure 7: Training efficiency comparison for ReMix-PPO and PPO (1.5B) on AIME'24, ReMix achieves a score above 35%, around 1.2x to 1.6x faster than PPO.
Volume
Training Steps
Training Duration
0.825
0.925
0.800
0.800
0.800
Pass01
0.775
0.775
0.750
0.725
0.725
0.725
0.575
0.675
0.675
40
6.0
130
500
10
e(K)
Stops
Timelhours)
ReMix-PPO
PPO
Figure 8: Training efficiency comparison for ReMix-PPO and PPO (1.5B) on MATH500. ReMix achieves a score above 80%, around 3x to 5x faster than PPO.
Importance Sampling
Minerva Accuracy
Response Length
10008
0.28
5000
1.0006
0.25
5000
1.0004
Takers
1.0002
0.22
6000
1.0002
1.0000
0.29
3000
1.0000
0.16
300
200
200
500
1.00
400
Steps
Stops
Stops
Steps
Figure Training dynamics regarding importance sampling ratio, accuracy, and response length under varying proportions of off-policy data p for Mix-PPG. Leveraging more off-policy data leads to a larger policy distribution shift, a faster early boost in accuracy yet worse later-stage performance, and shorter response length.
Policy Loss
0.3
0.2
0.1
0.2
0.3
0.4
-0.3
0.5
Steps
Figure 10: Policy loss under varying proportions of off--policy data for Mix-PPG. Leveraging more off-policy data leads to larger policy loss.
27

---

Published as a conference paper at ICLR 2026
L VARIOUS ANALYSIS
In this section, we further analysis the effects of ReMix, including the performance under response length constraint, the impact of prompt format.
L.1 THE PERFORMANCE UNDER CONSTRAINED MAXIMUM RESPONSE LENGTH
Since ReMix shows a feature generating more concise responses as discovered above, we conduct an additional experiment to evaluate the performance of our model when the maximum response length is constrained. Different from the default evaluation setting of 8,192 maximum response length, we halve the maximum response length 4,196 tokens for ReMix during evaluation. For comparison, we evaluate ReMix-PPO (1.5B) with the base model, DeepScaleR, and PPO under the halved maximum response length. The results summarized in Table 15.
Table 15:
Performance evaluation of 1,5B models with 4k maximum response length. The arrow denotes the accuracy degradation compared the results with 8k maximum response length (referring to the results Table 1). All the models are negatively influenced by the halved maximum response length. Compared with DeepScaleR, ReMix-PPO exhibits the smallest decrease in model performance and performs the best, thanks to its concise and shorter reasoning behaviors.
Model
AIME'24
AMC'23
MATH500
Minerva
Olympiad
Avg.
Maximum response length: 4096 tokens
--Distill-Qwen--.55 1,5B (Base Model)
20.00
37.35
60.40
13.24
22.37
30.67
DeepScaleR
10.00
49.4
75.00
21.32
34.22
37.99.14.15
PPO (500 Steps)
20.00
48.19
77.60
25.00
38.96
41.951761
ReMix-PPO (350 Steps)
23.33
59.04
79.00
27.57
39.11
45.61
All the models are negatively influenced by the halved maximum response length, which matches the intuition. Notably, DeepScaleR, the best 1.5B baseline model used in our work, suffers significant performance drop when the maximum response length is limited 4,192 tokens. In contrast, ReMix- PPO exhibits the smallest decrease in model performance and performs the best in this constrained setting. This finding underscores the resilience the preference for a concise and shorter reasoning process learned via off-polic, training of ReMix in handling the constraints on response length.
Takeaway 4. ReMix favors concise reasoning and is resilient to constraints on response length.
The constraint on maximum response length greatly degrades the test-time reasoning per- formance of LLM models, while ReMix suffers less thanks to its preference for a concise reasoning process.
L.2 THE IMPACT OF GUIDE TOKENS IN PROMPT TEMPLATE
In addition, we investigate the critical role of the prompt template for response generation used during training and evaluation. To establish comparison, we make use of prompt template without guide tokens (as shown below). Recall the standard prompt template we presented in Section 4.1, the difference is that the prompt template without guide tokens does not contain the guide tokens that appear as the prefix (i.e., <begin of sentence>< User (>) and the suffix (i.e., < Assí stant ><think>>.
28

---

Published as a conference paper at ICLR 2026
System Prompt (Without Guide Tokens)
<thegin of sentenee < Xenia and Sergey play the following game. Xenia thinks of a positive integer N not exceeding 5000. Then she fixes 20 distinct positive integers a1, ........
such that, for each k = ..2, 20, the numbers N and a are congruent modulo k. By move, Sergey tells Xenia and she tells him back the set {ax k 8} without spelling out which number corresponds to which index. How many moves does Sergey need to determine for sure the number Xenia thought of? Let's think step by step and output the final answer within (boxed < <think?
S of positive integers not exceeding 20,
To investigate the impact of different prompt templates, we define a response as format-correct if includes content enclosed within paired <think> < /tthink,, Parallel the 1.5B base model and ReMix-PPO, we consider variant of ReMix-PPO that is trained without guide tokens, denoted as ReMix-PPO w/o Guide Tokens. We evaluate the performance of the candidate models in terms of Pass @ accuracy' and format correctness MATH500, when using the standard template (i.e., with guide tokens) and the modified template without guide tokens. The purpose of this experiment to answer two questions: (I) whether the models trained with guide tokens (i.e., the base model, Re.Mix-PPO) can also perform well when the guide tokens are not prompted during evaluation; (2) whether the model trained without guide tokens also obey the format and output the solution. The results are summarized in Table 16.
Table 16: Performance evaluation of 1.5B models with and without guide okens on MATH500. Both the base model and ReMix-PPO show a 0 format correctness when the guide tokens are missing during evaluation, while ReMix-PPO exhibits a smaller drop in the accuracy. For the variant of ReMix trained without the guide tokens, it performs well under both the two template settings. means higher is better and J. means lower is better.
Model
Eval w/ Staudard Temp.
Eval w/o Guide Tokens (f)
Relative Decrease (L)
Pass@1
Format Cor.
Pass
Format Cor.
Pass@1
Format Cor.
R1- Dis-Ditill--ee Qwen- 1.5B (Base Model)
67.40
70.00
52.00
15.40
70.00
ReMix PPO (350 Steps)
82.00
93.50
71.00
11.00
93.60
ReMix-PPO w/o Guide Tokens (500 Steps)
82.00
92.20
77.60
91.20
4.40
1.00
The results show that the base model yields a format correctness of O when evaluated without the guide tokens, accompanied by a decrease of 15.40 points in Pass @ accuracy. Similarly, ReMix-PPO also exhibits 0 format correctness yet a smaller decrease of 11 1.00 in the accuracy. This indicates that the presence of the <think> token the prompt helps the model to autonomously generate a closing </think> tag, maintaining format consistency. Thus, it delivers negative answer to the first question above, while ReMix shows a better robustness to the absence of the guide tokens.
In contrast, the variant trained without the guide tokens also performs well when using the standard template, and achieves an increase from 77.60 to 82.00, reaching the same performance as ReMix that is trained with the guide tokens explicitly. It also maintains consistently high format correctness. This shows a good robustness to prompt change. We found similar results for the other four math reasoning tasks well.
We hypothesize that removing the guide tokens during the training of ReMix allows the model to explore a broader distribution, rather than overfitting to the explicit guide tokens in the standard template. Such flexibility encourages the model to internalize reasoning behavior in a robust and general manner, instead of relying on external structural cues too much. As result, it becomes more robust to prompt variation at inference time. The smaller relative degradation observed in both accuracy and format correctness supports this view.
Note that the correct answer with wrong format still counted correct for Pass@ accuracy here.
29

---

Published as a conference paper at ICLR 2026
Takeaway S. ReMix more robust to the variation of prompt template.
Removing explicit guide tokens in the standard template significantly cripples the performance of the base model, while ReMix exhibits better robustness and compatability to the absence of the guide tokens during both training and evaluation.
M CASE STUDY
To better understand model's reasoning behavior, we present a case study centered on representative example that the base model is able to solve correctly. Figure compares the responses produced by three variants trained under distinct strategies: PPO, Mix-PPG, and Mix-PPG with an Increased UTD ratio. Notably, the three outputs differ significantly in length, with the PPO-trained model producing the longest response, followed by Mix-PPG, and Mix-PPE PPG with an Increased UTD ratio yielding shortest.
With a sufficiently long response window, the model engages explicit self-reflection and follows structured step-by-step reasoning process to arrive at the correct answer. In contrast, the UTD-2 model, exhibits minimal or no reflective behavior and tends to bypass intermediate reasoning steps, leading more direct but less interpretable answer. These observations suggest that adequate response length plays a critical role in enabling reflective, multi-step reasoning.
Question
Prompt
rive points SAG, 385, sCs, $DS, 605 on flat field. 9A6 diractly north $os, $RS 15 rectly west of $08, $C8 1s directly south of $06, and ans 1a directly east su8. The distance between 963 and $D9 is 140 m. A hot-air balleon is positioned in the air SHE rect' sucs, and $IID.. Rope SHCS has length 150 and rope SHDS has length 130 m. [say] size (250)
sng. The balloon is held in place by ropes SHAS, SHRS,
pair A, B, C, D, O, M, 8, 21
(0;
8- (1,
De (1.5,,-.333
3=11,5,,,3;
C-( 1, 1);
g= (5/3) (2+0);
X= (5/3) (2+R);
Y-
7= (-()+ (X1)
(0-- 10----------
draw (A-
(3--11
draw (0--H,
1inewlitth
draw (R--E, dashed];
dzaw [3 E, dashed);
draw (C--F, dashed);
dashed);
dot (A);
dot (B): dot (S);
dot (D);
(o);
dot [H)
(*a", NEI;
label H, SW]
labal (*c", 5, SE1;
label (*D", D, NEI: label ("0", 0, SE);
label ("H", [/asy]
length of zope rope 8BCS rope $IDS are replaced by single rope SHPş where SP9 is paint on the atraight line between SC$ and sD6. 1The loon remaina at the same position above $08 deser hed above.) netermine the greatest length of that be saved.
reduce
Correct Answer
160
30

---

Published as a conference paper at ICLR 2026
Case 1: PPO (1/5)
## Stage 1 Question Rehcarsal
Okay, se 1 have this problem with łive points on a flat field: & B, C, D. and o. A is directly north of o, east of o. The distance between C and D 140 meters. There's het-air bellsen at point a directly above o. The ballocn is held by four ropes: HA, EE, HC, and HD.
direstly west
directly south O, and is directly
Now, the problem says that rope HC 1a 150 meters long and rope HD is 130 metera long, We need reduce
length roрe used by replacing IIC where P is point on the straight line between C and D. The balloon stays at the same the greatest langth rope that be
with single zope
position в above o. need
## Stage 2 Spatial Configuration and Equation Derivation
rirst, nead visualize the setup. Points A, center. la north, R is west, 5 la aouth, D la east. So, if imagine a coordinate system with
all around which the
(0,31, then (d, 0). But wait, actually, since A is directly north of o, B la directly west, C la directly south, and 5 is directly east, the coordinates can aet as follows:
(-b,
at 10, -c),
Let's assume is at (0, Then, A is at (0, hJ, is at (-k, 0), is at 10, -r), and u Since
31 But actually, the distances between is directly south of 0 and is directly east of O, the distance between and D would be hypeteause which 1s and on is the distance from 0 to D, which is Therefore, the diszance between and le aqri la? m²! 140 melers.
is given 140
triangle formed by 0C
sut we don't know m and n yet. However, we know the lengtha of HC and HD. Let mo recall that is di rectly y 150 meters, and the diatance fzom Il to is 130 nete:s.
0,
cooréinates
ht. The distance from R ta
## Stage 3- Step-by-Step Computation
8o, l write the distance tormulaa:
Distance uc: eqet (10 0) (h (-m)) sgrt ((h m) 1h ml 150. Since and m positive distances,
150.
similarly, 130.
8D: sgrt (0 n) (h 0) squt (n? hi) So, sgst h²)
also know that the distance between and is 140 meters, which agrt In? 140.
5o, have three equationa:
2. (n" + h2) = 130
sqrt (n? m²) 140
We con square the third equation to get mi 1403 19600.
From the first equation, h 150
Substitute h into the second equations
(2² (150 m) 130
Square aides:
(150 139 16900
31

---

Published as a conference paper at ICLR 2026
Case 2: PPO (2/5)
a1so have nt m² 19600.
subtract second equation First:
(n? 1150 m1
(n? (2)) 16900 13600
expanding (150 m) t:
a² 22500 300m. m²
n2 m² 2700
simpliff ify:
22500 300m -2700
So, 22500 2700
Divide both sides by 300:
m 25200 300 84
So, 84 neters.
Then, from the ft ret aquat h 150
130
84 66 meters.
from the third equation, n m² 19600
19600
Colculate
7056
So, n² 19600 7056 12541
ThezeCore, sqrt (12544) 112 meters.
So, we have:
84 metera icistance From 0 to 0)
112 meters (distance from 0 to D) metera theight of
## Stage 4- Optimization Strategy
need replace ropes and with single rope where F point Me need to find the soa soaition of 2 such that the total length of HP is minimized, thus meximining the length seved.
So, the original total length is IIC II 150
130 280 meters.
## Stage - Critical Point Analysis
need find the minimal. possible length of where CD. Then, the greazest length saved would 6 be 280 minimal HP.
To find the minimal length of HP, can model this as reflection problem. In inisatiioo problems where you have find the shortest path that reflects off line, you use the nethod of reflection.
think of point and paint such that the distance HP 18 minimized.
this
need find the point
32

---

Published as a conference paper at ICLR 2026
Case 3: PPO (3/5)
## Stage Step-by-Step Computation
alternstively, we can model. this as reflection of a accoss CD, and then the minimal distance fzom B Lo to would be the straight Line distance Irom to the reflection D across CD.
wait, sctually, since is the minimal diatance Irom H to P to is the same as the distance from a to D. But since things
reflact
point and then
CD,
is
line CD, might mixing
Wait, perhaps better approsch is parametrize point CD and then find the minimal distonce H2.
Let's set up coordinate system.
Let's place 0 at (0, 31. Then, is at 10, -84), D is at (112, D1. a is (0, 86).
need find point Parametrize
such that minimized.
-84) (112, 01, Letra let the parameter
wheze L-0 is and t-2 is D.
So, the coordinates of P can be expressed ag:
112t
y -34
So, (112t, 84t)
Now, the distance EP la the distance between 60, and 1112t, -84 846).
5o, distance squared 18:
(112 - 01 (-24 + ad: 66) '2
Simplity
(11251"2 1-150 84t)
compute this:
(12544t²) (22500 2=150+84t 70566*)
Compute each term:
12544L 22500
25200L 7056L7
Combine like terns:
(12544 73 25200L
22500
Calculate 12544 7056:
19600
So, distance squared La 19600E> 25200t 22500
To find the minimum, take derivative with respect to and set to zero.
33

---

Published as a conference paper at ICLR 2026
Case 4: PPO (4/5)
Letta denote ((t) 19600t 25200t 22.000
33200t
Set r" IL) 0:
39200t 25200
39200L 25200
25200 39200
Simplify:
Divide numerator and denominator by 2800:
39200 / 2830 14
So, t 9/14
Therefore, the point P. is at:
x 112 (9/14) 112A (8) /14 8.9 = 72
-94 84+ (9/14) -84 /14 -84 (6+9) -84 -30
So, (72,
Now, compute the
Distance (122
0) (:30 66) 21 sqzt (727 (6))
Conpute 72": 5184
Compute (-96) '2: 8216
Sum: 5184 9216 14400
30, distance HP sqrt
Therefore, the minima: length 120 meters.
Originally, 150 130 meters.
So, the greatest length of cope that cen be saved is 280 120 160 meters.
34

---

Published as a conference paper at ICLR 2026
Case 5: PPO (5/5)
## Stage 7 Final Result
++Binal Answerve
The greatest length zoge that can saved is \boxec meters. </think>
flat field, whera 1AM directly north of (OV), \BB ia directly wes: of VOV,, VCV is directly south of \oV, of (0V) distance between MCV
Given five points (AV, \BV, VICU, MBN, and (OO)
(DV) is directly A hot-sir balloon is positioned at (AY directly above VOV) the balleon is held by four ropes (EAN) (HBBU, VHEM,, 150 neters and 130 meters, respectively. wе need determine the greatest length of that con be aaved by replacing (HCC and (8D)) with a single rope ((PP whare MPV is the straight line between AIC and AIDA)
V(D
mecers.
(HDA). The lengths sopes M/HCM
coordinates - V22 is at ((I,, D1V.
First, we
points:
(AY A/1,, B1V, (RR at M(Hk, 01,, MEV M110 -m) and MOD ((n, 0/3.
Given:
The distance between (CC) and (DV is 140 meters, Adlaget/a"2 i m²2) 1401).
discance (BCVV) 150 meters, The distance (HDD)) is 130 maters, (n'grrinn2 h 1305).
m 150V).
Solving these equations, find: V/h 6611 meters, (mm 84%) meters, and Vnn 1121) meters.
To minimize the length of rope \HPA, we parametrize point (PP) on \ VICD)) and find the point (PM that minimizes the parameter Vit = (114)19) leading to the coordinates of (FF) as MO22, -30)
(SP)) Using calculus, se
optiral
The minimal length of rope MBPA) calculated 120 meters. The original total length of ropea \ (HCA) and (HD)) is 280 meters, so the greatest Length of rope that can be saved
\boxed(166
of
(3012 tokens)
35

---

Published as a conference paper at ICLR 2026
Case 2: Mix-PPG(I/4)
## Stage - Question Rehcarsal
Okay, se 1 have this problem with five points: A, B, e, D, and o on flat field. A is north of 0, B weat of O, is south of O, and D east of o, The distance between C and is 140 meters. There's hot-air balloen at point directly above o, The balloon held by four ropes: HA, HE, BC, and HD.
Rope N0 1a 150 meters long, and HD 1a 130 meters long, The proplem says we need to reduce with single cope
total length cope by replacing somewhere on the line between and U. need to find the greatest length of rope that con be saved.
## Stage 2 Spatial Configuration and Equation Derivation
rirst, let visualize this. Foints is di rectly above o, so it's like a vertical pole, The ropes HA, HR, FC, HD are all connected
form restangle with the center.
go
of
points.
Given that 130 280 After replecement, it's just one rope HP. So the maximum length 280 length of нР. So need to find the minimum possible length of HP, which would give the maximum saving.
150 n and HD is 130 m. the total Length before zeplacement 180 +
So, mininize small possible. Since P is on ch, which is atraight l're between and D, which are 140 meters apart.
point should chosen
distance from
Let ne assign coordinates to make this casier. Let's place o at 00,01. Then, since A is similarly, is west
north
(0, for C 1s south of 0, ao c is (0, -et, and D ls eas: of 0, so 5 is ld, 0)
1 (-b, 0) for aome
## Stage 3 Step-by-Step Computation
Given that the distance between and distance between them is sqet 01 e'2 140"2 19600.
m. Since Le (0, -c) and 1s (d, 01, the
2)) sqrt Id c72) 140. so d'2
Now, the balloon is at H, which is directly above o, so a is (0, h) for some h.
Given that HC is 150 30 the distance from B to 1s sqrt (10 01 (-c h2 I -
lo hl 150. So e + 150, Similarly, HD is 130 m. the discance Irom A to is ((d 21 (0 21 (d"2 h²2) 230. So d'2 h²2 130"2
16900.
So now, we have ewo equations:
150
d"2 hr2 16500
2"2 d'2 19600
From equation
550
this into equation
(150 h) 2
Expand (150 h) 2: 22500
300h h72
So 22500 300h h*2 d'2 19600
36

---

Published as a conference paper at ICLR 2026
Case 2: Mix-PPG(2/4)
Dut equation 16300 h'2. Substicute in:
h22)
Simplity:
22500 30th h 16900 h-2 19600
The h12 terns cancel;
22500 16900 300h 19600
Compute 22500 16900: 39400
19600
Subtract 19600 from both sidea:
39400 13600 300h
19800 303h 5
30th 19800
19800 300 66
Then 150
150
66 84
equation 2, d'2 h:2 16900
4"2 66"2 16900
66*2 is 4356, so d12 16900 4356 12544
8o d aqrt (12544) :12 meters.
have 66, 84, 112.
## Stage 4 Optimization Strategy
Now, point P is somewhere CD. tet's parameterize point P. Since on is from 10, -84)
D (112, 01. So the line CD can be parameterized, as (112t, -84 U)) wheze is Irom
point (1126,
need minimized.
such that the distance 10, 661 to (112t, 18411 t))
## Stage Step-by-Step Computation
Corpute the disttnnce aquarad:
(112t 01 (1 E) 661 2
(1126) '2 (-94 (1 t) (66 1
37

---

Published as a conference paper at ICLR 2026
Case 2: Mix-PPG (3/4)
Corpute each port::
(1126)
Second Lexm: 84 B4L 58) 150 84L). 30 equazed: 150 846) "2
Compute (-150 84t) '2 150 2+150.84t (84t) 22500
25200t 7056 t²2
30 Lotal distance squared Le:
12544 t 22500 25200t 7056 +
Combine like
(12544
22500
Compute 7056: 19600
So distance squared 19600 t12 25200t 22500
To find the minimum, take derivativo ve with respect to and set to zero.
Let E(t) 19600 t12 25200t 22500
(t) = 39200 25200
Set
25200
25200
L 25200 39230
Bimplity: divide numerato: and denominator by 2800: 25200 / 2800 9, 39200 2800 14
So 8/14
is 9/14. 8o point P 1s (112+ (9/141, -14+ 9/14)1
Corpute (9/14); divided
8-5-72. x-ccordinate
Compute -30.
9/14):
5/14. (54x1) -645 -30. y-cocrdinate
point P 1s (72, 30).
Now, compute the distance
Cocrdinates
6E). coordinates of (72, -30)
Distance squared: 172 0) 7 (-30 66) 2 7272 + (-96)
Compute 72"2: 5184
Compute (-96) '2: 9216
38

---

Published as a conference paper at ICLR 2026
Case 2: Mix-PPG(4/4)
distance squared: distance sqrt (14400) meters.
14400
the length meters.
120 meters. Therefore, the total length saved is 280 120 160
## Stage 6- Final Answer
*+Final Answer.*
The greatest length of rope that can be saved is \boxec (160) metera.
field, where MAM rerectly north of NOW,, AIR ia directly west of VOV,, XCV is direct south of (OV, and (DM) is directly eest of (OV). The distance between MCV and MDV is 140 maters. hot-pir balloon 1s posit' oned (EV
five peints (AM), \BB,, VICU, MBM, (OO)
directly above 110V, The balloon is held by four roses VIBA, VHBB,, \GEVV, and AIDDY) Rope (HCV)
zope (HDV) 130 meters. need determine greatest length of rope that can be seved by replacing V/HCV and MADD with a single rope VHEAA) batween V/C
meters
(PV) is a point
straight
(DD).
First, we assign coordinatas to the points. Lot MOV be at ((100)) Points VAV, MBV), MIM,, and (DV) The distance between VOV)) and (DD) is 140 mecers, giving us the equation (dd2 196003).
MIC,
Mi-b, 0 MOO, -o1 and Mr,, 013) respectively.
the distances /HC 150 metera MRD 1301) meters, have:
Vi h 153V)
2. Aid22 + h-2 16930V)
(o72 d"2
Solving
equations, find: - Mc = 84%) meters
V/h 65%) meters
Md 112% meters
Point (PV) VIDD) pacameterized (1(122, B4 DJA, distance Irom AII)) to (22) is mininized by finding the value of Vit)) that minimizes the distance squared cccurs
function.
(144)(9) giving point APV (( -30)
The distance (HP)) caloulated as:
Distence squazed: (17222 (-96) "2 14400V)
Distance: (14900 120V) meters
Thus, the greatest length of rope that can be saved is (280 120 160N1 meters.
The rinal anawez is ((boxed (1600 -<llend_ff_eentenee
(2830 tokens)
39

---

Published as a conference paper at ICLR 2026
Case 3: Mix-PPG with Increased UTD Ratio
## Stage Quetion Rehearsal
sirst, identify initial lengtha ropes BC and HD. are given 150 130 m respectivelyy totaling 280 m.
Next, analyzo the rope EP Point lies somewhere the line segmant Since and D are recetlly south and north of o respectivelyy cb 1a a vertical line. The ooon positioned directly height of the balloon.
horisencal distance
## Stage Spatial Configuration
Using the Pythagorean theorem for triengle HIP, where HC is 150 and CP is the variable distance from Sqrt (140# h+1 150 m.
up the equation:
Solving for h, the height of the balloon, I Find h Bert (150 110?) Sqrt (22100) 148.68 m.
Then, for triangie HOP, with DP CD CP 140 X, set up the equation: Sqrt (1302 h+) (1304 22100).
Calculating this 195.41 m.
length of as sqet
2210C) eqrt (36200)
adding both lengthe gives appreximately 390.88 88 compared to the original 280 The difference is approxinately 110.86 m, which is the maximum length of rope that can be soved.
</thinks
## Stage 3- Step-by-Step Computation
determine the maximum langth rope that can saved by replacing ropea HC with single rope HP, analyze the situation step by atep.
avinttial Setupiva
**Ropes 90 and HDEAY Given lengths are 150 m and 130 m, respect ively. ..Total Initial Lengthe 150%, (texttm) 130%, \text 280%, (text(m) V.
**After seplacement
*illew Rope IPiv Point A AI lies segment / CD Mr replacing \f IC M and A HD W.
**Coordinates and Jistances:**
Assuma
the origin, A(B
vertically above V
h) V
\ C.C.C V) and M V are vertically agned with A below and M D M above C W. triangles \ ICC
V); 1. For M 90 M \agrt("4002 2 + h"21 150 A >> V (150"2 140"2] T-- 148.66%,
Using Fythagozas
stext [m) V.
2. For M DP M:: (2g00(130'2 h \sgrt(130'2 148.6612) -- 195.44%, (mext(m) M.
AkCalemlation SavingstAv
.oTota New Length:we VI 150%, Atext 195.44%, im) --" 44%, \text W. Savinggs: 280%, \text(m)
\text(m) 68.56%, \text (m)
However, precise calculstion shows maximum saving occurs when \ P V is imtimally placed, yielding
AwED+ggrt (10) 1"=" 187.08 total, saving "-" 92.92 m) But standerd approach gives approximate *+110.89 m saving.
## Stage 4 4 Final Answer
s+tinal AnSMEF:** \(
\boxed(110.8g 881
\]<lend sentencee
(908 tnkens)
40


Reviews:
[{"review_id": "jTX2B78AMG", "reviewer": "Reviewer_cESh", "summary": "This paper proposes ReMix, a method for efficient off-policy Reinforcement Finetuning (RFT) of Large Language Models. ReMix integrates three key components: a Mix-Policy Proximal Policy Gradient to utilize historical data with an increased Update-to-Data ratio, a KL-Convex policy constraint that dynamically blends penalties from the base model and the previous policy, and a Policy Reincarnation mechanism that switches the base model and reverts to on-policy training mid-way. The authors demonstrate strong empirical results on mathematical reasoning benchmarks, showing that ReMix-based models can achieve state-of-the-art or competitive performance with a dramatic reduction (30x to 450x) in the amount of rollout data required compared to several strong baselines.", "questions": "- The KL-Convex constraint uses a dynamically decaying coefficient $\\lambda(t)$ to balance between the base and previous policy. The design of this decay schedule $\\lambda(t) = \\max\\left(1 − 0.1 \\cdot \\lceil \\max(t − 50, 0) / 10 \\rceil, 0.5\\right)$ appears to be heuristic. Could you provide an analysis that justifies this specific form?\n- The Policy Reincarnation mechanism is a critical component, yet its triggering is based on a predetermined step $T$. How was this specific hyperparameter chosen, and what is the sensitivity of the final performance to this choice?\n- The analysis reveals a strong \"preference for shorter responses.\" Do you think this bias could be harmful in tasks where comprehensive reasoning is crucial, and if so, how could ReMix be adapted to control for this effect?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Well-Designed and Synergistic Method: The three-component architecture is intuitive and addresses key challenges in off-policy RFT. The components work synergistically, as shown by the ablations, with Policy Reincarnation being a particularly clever and novel idea for bridging early-stage efficiency and late-stage stability.\n\n- Rigorous Ablation Studies: The ablation study in Table 3 is comprehensive and effectively validates the contribution of each proposed component, clearly showing that the full combination is necessary for optimal performance.", "weaknesses": "- Overly Optimistic and Potentially Misleading Framing: The paper's central claim of \"30x to 450x\" reduction in rollout data is framed in a way that overstates the advantage. This is achieved by comparing ReMix's early-stage performance against baselines' final performance (e.g., ReMix-PPO at 100 steps vs. PPO at 900 steps). A comparison of final performances shows a much more modest efficiency gain, undermining the dramatic narrative.\n- Limited Scope of Evaluation: The empirical validation is constrained in two key dimensions: model scale (≤7B parameters) and task domain (mathematical reasoning only). This leaves the generality of ReMix an open question. Its effectiveness on larger models (e.g, 70B+) or on fundamentally different tasks (e.g., code generation) is not established. Furthermore, the absence of ReMix-GRPO results for the 7B model is a notable omission, as it prevents a consistent assessment of the method's performance across different underlying RL algorithms at a more capable scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BytFFAEqk", "reviewer": "Reviewer_pHkv", "summary": "This paper presents **ReMix** (*Reincarnating Mix-policy Proximal Policy Optimization*), a framework designed to make reinforcement fine-tuning (RFT) of large language models more sample-efficient by effectively leveraging off-policy data.  \n\nReMix extends standard on-policy approaches such as **PPO** and **GRPO** through three main components:  \n\n- **Mix-policy proximal gradient:** Combines on- and off-policy trajectories to achieve higher update-to-data (UTD) ratios and more efficient data reuse.  \n- **KL-convex constraint:** Interpolates between the base and current policies to ensure updates remain both stable and adaptive.  \n- **Policy reincarnation:** Periodically resets the base model to the current policy, enabling a smooth transition from rapid early learning to stable convergence.  \n\nAcross five math-reasoning benchmarks—**AIME24**, **AMC23**, **Minerva**, **OlympiadBench**, and **MATH500**—ReMix achieves state-of-the-art Pass@1 accuracy while using **30×–450× fewer rollouts** than leading baselines such as **DeepScaleR** and **AceReason**.  \nThe authors also analyze the behavioral effects of off-policy learning, observing patterns like shorter responses and reduced self-reflection when off-policyness becomes excessive.", "questions": "1. In Eq. (3), why do the authors use \\(\\frac{\\pi_K}{\\pi_{K-i}}\\) to control the trust region? What is the motivation or benefit compared to using a fixed constant region? It would be helpful to see empirical evidence or ablation results demonstrating that this adaptive trust-region design improves performance.  \n2. In Fig. 3, ReMix-PPO appears to achieve roughly \\(6\\times\\) faster improvement than vanilla PPO in the early training stage. Does this imply that early trajectories are reused multiple times for updates? If so, could this repeated use of the same trajectories lead to training instability or bias accumulation?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear Motivation & Contribution** – Addresses the well-known issue of *sample inefficiency* in RFT with a principled and well-structured solution.  \n**Novel Integration of Off-Policy Concepts** – The combination of mix-policy updates, convex KL constraint, and policy reincarnation is original and intuitively justified.  \n**Strong Empirical Results** – Achieves SOTA reasoning accuracy while reducing training cost by 30×–450× across multiple benchmarks.  \n**Comprehensive Experiments** – Includes ablation studies, efficiency–accuracy trade-off curves, and behavioral analyses (e.g., response length, self-reflection rate).  \n**Readable and Well-Presented** – The paper is clearly written and easy to follow.", "weaknesses": "Overall, there are no major weaknesses in this paper. A minor concern is that the authors did not compare **ReMix** with other *off-policy* algorithms, relying only on *on-policy* baselines. Including comparisons with off-policy methods in future work would provide a more comprehensive evaluation and strengthen the empirical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROSru51Gkq", "reviewer": "Reviewer_h1SL", "summary": "This paper proposes a new RL workflow for reasoning models, composed of three key designs: 1) mixture of online and offline data 2) mixture of current policy and base policy in KL regularization 3) mixture of training algorithms. The experiments are conducted on 5 different benchmarks, showing advantages over vanilla PPO and GRPO, in performance and training cost.", "questions": "- why does ReMix-GRPO make the model even worse then base model for AIME in table 1?\n- what is the exact formulation of distribution $i\\sim \\nu$ used in practice, and is there any theoretical explanation?\n- is pass@1 enough? as [2] indicates that, the performance on pass@k might not be improved. \n- vanilla GRPO has a length normalization term, while some variants like Dr. GRPO [3] removed it. Did you apply length normalization in your experiments?\n\n[2] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? https://arxiv.org/abs/2504.13837\n\n[3] Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective. https://arxiv.org/abs/2503.20783", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- clear figures\n- extensive experiments on various benchmarks\n- prominent cost reduction compared with vanilla PPO and GRPO\n- the performance is generally comparable with online versions, even better sometimes", "weaknesses": "- only trained on Qwen model, which is known to be unreliable for reasoning training [1].\n- no theoretical explanation for the design of KLC loss and ReMix algorithm\n- no explanation for the choice of $\\lambda$ and $T$\n- figure 4 is too noisy for solid analysis\n- proof for eq.6 is missed\n\n[1] Shao et al. Spurious Rewards: Rethinking Training Signals in RLVR. https://arxiv.org/abs/2506.10947", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uOVNtHtKVu", "reviewer": "Reviewer_1DV6", "summary": "This paper introduces a novel method designed to improve the sample efficiency of Reinforcement Finetuning for LLMs. The core problem addressed is the inefficiency of on-policy RL algorithms like PPO, which are standard in RFT but discard historical data after each update. ReMix combines three main components to enable the use of off-policy data including mix-policy proximal policy gradient objective, KL-Convex policy constraint and policy reincarnation step. This paper demonstrates that ReMix achieves better performances on several math reasoning benchmarks than baseline approaches and models with a great reduction in the required volume of rollout data.", "questions": "How did you determine the optimal values for policy reincarnation step T in {50, 100}? Is there a risk that a poorly chosen T could erase the early-stage efficiency gains or lead to premature convergence before the model has fully benefited from off-policy exploration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a critical and expensive problem in the development of advanced LLMs: the prohibitive computational cost of reinforcement finetuning. The promise of achieving comparable or better performance with orders of magnitude less data is a highly significant contribution that could democratize the development of powerful reasoning models and enable further scaling.\n\n2. The empirical evaluation is extensive. The method is tested on two different model scales across several math reasoning benchmarks in comparison with a lot of recent models. Moreover, this paper includes a lot of abalation study such as UTD ratios, KL-convex and policy reincarnation, and experiments in different scenarios such as training with constrained max response length and with differnet prompts.", "weaknesses": "1. Insufficient Comparison with SOTA Off-Policy Methods: The paper's primary weakness is its failure to adequately benchmark against other contemporary off-policy RFT methods. As a lot of papers discussed, GRPO is a biased estimation of KL penalty.[1, 2] Therefore, a lot of recent baselines should be included, such as DAPO [3]. Moreover, in Appendix F.3, the paper discusses the reason why it does not include other baseline algorithms including RePO, SRPO, SPO, AGRO and AsymRE with excuses like the lack of public checkpoints, different scales, or restrictive generation lengths.. However, such reasons are not sufficient to support the reason why the paper chooses only 2 baselines methods to re-implement. In addition, this paper chooses several baseline models which are trained with different datasets and settings. Therefore, it is unclear whether the quality of datasets and training hyper-parameters contributes to the improvement of the proposed method over these models.\n\n2. Limited Scope of Evaluation: All experiments are conducted exclusively on math reasoning tasks. While this is a challenging and important domain, the paper makes general claims about improving RFT for LLMs. It is unclear whether the method's effectiveness, particularly its implicit bias toward shorter responses, would be validated in other domains such as code generation. The findings may be domain-specific, a limitation that is not acknowledged.\n\nReferences: \n\n[1] Liu, Z., Chen, C., Li, W., Qi, P., Pang, T., Du, C., ... & Lin, M. (2025). Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.\n\n[2] Zhang, Y., Liu, Y., Yuan, H., Yuan, Y., Gu, Q., & Yao, A. C. C. (2025). On the design of kl-regularized policy gradient algorithms for llm reasoning. arXiv preprint arXiv:2505.17508.\n\n[3] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., ... & Wang, M. (2025). Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)