On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Authors: Wenhao Zhang
Yuexiang Xie
Yuchang Sun
Yanxi Chen
Guoyin Wang
Yaliang Li
Bolin Ding
Jingren Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Post-Training
Reinforcement Learning
Supervise Fine-Tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, we present a novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens. We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting, which reframes SFT not as a separate stage but as a dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off-policy expert data's influence at both holistic and granular levels, we incorporate a dual-control mechanism in CHORD. Specifically, the framework first employs a global coefficient to holistically guide the transition from off-policy imitation to on-policy exploration, and then applies a token-wise weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that CHORD achieves a stable and efficient learning process. By effectively harmonizing off-policy expert data with on-policy exploration, CHORD demonstrates significant improvements over baselines. We will release the source code to inspire further research.

Published as a conference paper at ICLR 2026
ON-POLICY RL MEETS OFF-POLICY EXPERTS: HAR- MONIZING SUPERVISED FINE-TUNING AND REIN- FORCEMENT LEARNING VIA DYNAMIC WEIGHTING
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Lif Bolin Ding, Jingren Zhou
Alibaba Group
ABSTRACT
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two promi- nent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs). Existing approaches that integrate SFT and RI often face the risk of disrupting established response patterns and inducing overfitting to expert data. To address this, present novel investigation into the unified view of SFT and RL through an off-policy versus on-policy lens, We propose CHORD, framework for Controllable Harmonization of On- and Off- Policy Reinforcement Learning via Dynamic Weighting, which reframes SFT not as separate stage but dynamically weighted auxiliary objective within the on-policy RL process. Based on an analysis of off- aff-pooicy expert data's 's influence at both holistic and granular levels, we incorporate a dual-control mechanism CHORD. Specifically, the framework first employs global coefficient to holisti- cally guide the transition from off- off-policy imitation to on-policy exploration, and then applies token-wisg weighting function that enables granular learning from the expert, which promotes on-policy exploration and mitigates disruption from off- off-policy data. We conduct extensive experiments across various practical tasks, providing empirical evidence that CHORD achieves a stable and efficient learning process. By effectively harmonizing off off-policy expert data with on-policy explo- ration, CHORD demonstrates significant improvements over baselines. We release the implemennaaiion to inspire further research.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in wide array of applications Yang et al.) 2024b| Zhang et al.] 2025a]] Mialon et 22.33 Gao et al.) (024). Such significant progress can be largely attributed to critical post-tuming paradigms that enhance the performance of LLMSs in real-world scenarios, i.e., Supervised Fine- TuneTTnnng (SFT) (Taori et al. 2023
Zhou et al.) 2023) and Reinforcement Learning (RL) (Ouyang al. 2022 Shao al. (0244).
These two paradigms present their pros and cons. SFT relies on high- expert trajectories to effectively mimic response patterns, which can be sensitive the quality and quantity of expert data (Ye et al.) 2025 Guha et al.] (025).. Recent studies also point out that SFT may struggle to generalize beyond mere memorization (Chu al. 2025) and is vulnerable exposure bias (Zhang
et al. 2019). In contrast, RL encourages LLMs to actively explore, which enables better generalization
through learning from direct feedback on their on-policy generations (Chu et al. 2025 Chen et al..
2025b). However, such explorations can sometimes be inefficient, leading to policy degradation
caused by entropy collapse (Yu al. 2025) or over-exploitation of suboptimal strategies.
A prevalent and straightforward approach for integrating the strengths of SFT and RL while mitigating their weaknesses the sequential SFT-then-RL. paradigm (Liu et al. 2025b; Lambert et al., 2024). Intuitively, the expert's 's reasoning patterns learned in SFT guide the RL exploration beyond local optima, and then the on-policy learning in RL mitigates exposure bias inherent in SFT and prevents
"Corresponding author. Email (zwh434786, yaliang.li) @alihaha-inc.com

---

Published as a conference paper at ICLR 2026
40
Pure RL
90
Policy Shift
Readaptation
Ovarfitingg Phase
SFT with 40k samples + RL
Phase
Accuracy
30
SFT with 10k samples + RL
Accuraoy(%)
80
20
70
10
60
60
80
100
120
20
60
100
Step
Step
Figure 1: We train Qwen2.5-1.5B 1.5B-Instruct on the Open-R1 dataset and evaluate the performance on a held-out validation set. These results show that the SFT-then-RI. training paradigm can yield suboptimal performance compared to pure RL.
Figure 2: We perform SFT on Qwen2.5- 7B-Instruct using expert data generated by Deepseek-RI. The observed learning curve (measured by accuracy on MATH-500) demon- strates "sift--radapt-oveerfit"" progression.
overfitting to a limited set of static examples. However, empirical observations show that the SFT. then-RL. paradigm does not consistently outperform the pure RL approach, as illustrated in Figure] which is also noted in recent studies (Zhang et al. 2025a Chen et al. 2025b).
In this study, we make a further investigation and demonstrate that such suboptimal performance may arise from training on expert data that significantly diverges from the model's established patterns. As progression consisting of
illustrated Figure the learning curve reveals a "shift-readapt-overiii three distinct phases. Firstly, there is initial disruption in capability due to the sudden policy shift, which is followed by a readaptation phase during which the model adapts to the expert's 's patterns and recovers performance. Finally, observe that the model eventually overfits the expert data. These observations highlight that while expert data can bring new capabilities, it may also disrupt established patterns and induce overfitting during the training process.
Drawing upon these insights, we unify SFT and RL through the lens of off-policy versus on on-policy learning. The SFT process is reframed auxiliary objective within the on-policy RL process. We further design CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting. CHORD features a global coefficient for controlling the overall influence of expert data throughout the training process, and a fine-grad -enigrri weighting function Φ(-) that helps maintain stability via down- down-weighting highly divergent tokens from off off-policy data that could disrupt on-policy training.
as separate tuning stage, but dynamically weighted
Our contributions can be summarized as follows:
We provide a systematic and in-depth analysis of the training dynamics when employing a separate SFT process to integrate off-policy expert knowledge into models with established policies. We identify the sehiftrradapptooverftt progression, revealing how off-policy data can disrupt the established response patterns of LLMs.
We propose CHORD, novel framework that unifies SFT and RL via a dynamically weighted auxiliary loss, which consists of global coefficient f and token-wise weighting function ((.). CHORD provides fine-grained and flexible control of the influence of off aff-policy expert data while ensuring training stability, promoting a harmonious integration of learning from both off-policy expert demonstrations and the model's on-policy exploration.
Extensive experiments demonstrate that CHORD outperforms the SFT-then-RI. paradigm and existing approaches. We provide both quantitative and qualitative analyses to show that CHORD strategically navigates training dynamics to selectively absorb expert knowledge without stifling the model's reasoning capabilities, highlighting its superiority and effectiveness.
2 PRELIMINARIES
The post-tuning of Large Language Models (LLMs) involves optimizing their policy, denoted by me and parameterized by 0, generate desirable responses. This typically follows two paradigms: Super- vised Fine-Tume Tuning (SFT), an off-policy paradigm driven by a static dataset of expert demonstrations; and Reinforcement Learning (RI.,, on-policy paradigm guided by dynamic feedback.
2

---

Published as a conference paper at ICLR 2026
Specifically, SFT adjusts the policy Ty mimic high-quality, static dataset of N expert demonstra- tions, DSFT = {z y, | Here, 2; is a prompt and yr = (V,,...... V. is the corresponding expert response with ly!" tokens. The SFT objective is to minimize the negative log-likelihood of expert responses, typically optimized with an empirical estimate from a mini-batch of size B:
CSFT
log To (VT, (.
21=1
In contrast, RL optimizes policy my by maximizing expected reward R(7) from a generated trajectory
T = =xx (r,y"). y²). Group Relative Policy Optimization (GRPO) (Shao ct al. 2024) suggests sampling K responses (T...... TK) from policy Tsample when given a prompt z. Each response T& evaluated with the reward function R(Ta), and To is updated to maximize a PPO-style clipped surrogate objective. Consistent with recent studies (Hu et al. 2025a] Yu et al. [025] Chen al.) (025a). our formulation does not include the KL divergence term to avoid restricting performance of LLMs. The objective function can be formulated as:
A ma
(f) Ein E
min (O)A,,
Ti,k
where B is the number of prompts in the mini-batch and 6 is the clipping hyper-parameter. The advantage Ag for each response is computed by A₁ [((r))HH where μR and oR are the mean and < standard deviation rewards {R(Tx)) within the group, and 6, a small constant for stability. [Ti, 4 ,...) c..c) denotes the token-wise Importance Sampling (IS) ratio, which re- re-weights the probability of actions sampled under nsample to simulate on-policy sampled distribution. For "strict on-policy setup" (Liu et al., (2255b) that Tssaplle To. this ratio should always be 1, and the gradient of (0) should be equivalent to V log Th To (T,,,, (Te T(.).)))
Here (0) To(TT.k.ele 2...
3 CHORD: HARMONIZING OFF-POLICY AND ON-POLICY LEARNING
3.1 THE SALIT-READAPT-OEEE PROGRESSION WHEN UTILIZING OFF-POLICY DATA
Before introducing CHORD, we first take close look the training dynamics of the SFT process, revealing how training on off-policy expert data can disrupt the established response patterns of LLMs. Such disruption ultimately leads to the failure of the SFT-then-RL paradigm (Zhang et al. Chen et al. 2025b), as evidenced by the results in Figure
2025a
We train Qwen2.5-77B 7B--ntruut (Yang
2024a) expert data generated by Deepseek-RI (Guo et al. 2025) and monitor the changes in test accuracy on the MATH-500 dataset. From the experi- mental results shown in Figure 2 we observe that model performance declines during the first few epochs, followed by continuous increase to level higher than that before training, and then a slight subsequent decrease. The performance curve reveals a "shift-readaptoovvrrt"t progression:
Policy Shift: The performance initially declines since the model is forced to follow off- policy expert demonstrations whose response patterns are significantly different, disrupting its established response patterns and causing a significant performance drop. Such degradation is further exacerbated by exposure bias (Zhang et al. 2019 Schmidt] 2019). as the model, trained exclusively on ground-truth expert data, struggles navigate the self self-generated contexts it encounters during inference.
Readapt: As SFT continues, the model policy Ta begins to integrate the expert's response patterns and generates responses similar those of the expert. The exposure bias can be mitigated by reducing the reliance on the model's response patterns, thereby allowing its performance to rise steadily as it adapts to the expert's response patterns.
Overfit: Extended training on the limited expert data ultimately leads to overfitting, resulting in a decline in generalization and a significant loss of output diversity. Such overfitting can also restrict the exploratory capacity that is crucial for the following RL optimization.
3

---

Published as a conference paper at ICLR 2026
Rallouts
Rewards
On-palicy Rallouts
Tu
ciaatt
calcolate-
Advartages
SUM Advanttvs LogProbs)
The arswer 1bored
LogProbs
Task Sample
The orswer \eooedfoz)
0.1
control through
Loss (1-yt) GRPO_loss SFT__loss
loss
Pulicy LLM
Prompt Response
Taken-wis Probs
Tokera:
Probs:
<theas
high
0.95
[hay
0.0
low
0.4 medium
|venty]
wiee through (0))
SUM( LogProbs)
Weight
Figure 3: An overview of the proposed CHORD framework that unifies SFT and RL, featuring a global coefficient μ and token-wise weighting function (6-).
The observed progression makes it challenging to control the influence of off-policy expert data. The SFT-then- -L paradigm demands careful timing for the SFT-to-RL transition, and even then, such a two-stage paradigm may still yield suboptimal solutions due to the inherent separation of the training phases. This highlights the limitations and fragility of the SFT-then-RL. paradigm, especially when expert data's ' response patterns significantly diverge from the model's established response patterns.
Drawing upon the above insights, we propose CHORD, a novel framework that effectively unifies SFT and RL. The proposed framework consists of dual-control mechanism. We first introduce dynamic loss coefficient to balance learning from on- and off- off-policy data (refer to Section 3.2), then further design a token- wiiee weighting function that provides fine- ane-grained stability control (refer to The overall architecture of CHORD is shown in Figure 3
Section 3.3
3.2 CONTROLLING THE INFLUENCE OF OFF-POLICY EXPERT DATA VIA μ
Firstly, in order to control the influence of off-policy expert data, we propose to reframe SFT as a dynamically weighted auxiliary objective within the on-policy RL process. rather than a separate tuning stage as the SFT-then-RL. paradigm. Specifically, we design a combined loss function that minimizes weighted sum of the RL and SFT losses:
Chyoria ( = (1 ()rro(( + (O)
where LGRPO (0) is the empirical GRPO loss defined in equation 2 (SF is the SFT loss defined in equation and μ [0, 1] is hyperparameter that governs the trade-off between SFT and RL.
using a fixed value μ. the influence of the off- off-policy expert data remains unchanged throughout the entire post-tuning process. An advanced strategy, however, is to change μ for achieving dynamic balance between off-policy and on-policy learning. For example, the SFT-then-RL. pipeline can be regarded as special case with binary schedule (initially setting f and then transitioning
0). Moreover, previous studies (Ma et al. 20255) Gao et al. 2025) that utilize interleaved SFT and RL can be interpreted as employing periodic μ schedule.
Moving a step forward, applying decay schedule of μ provides a more graceful and flexible transition from off-policy imitation to on-policy optimization compared the rigid and binary switch. As shown in Figure the training begins with a large μ value, encouraging the model to learn more from off off-policy expert data. As training progresses, f gradually decays smaller value, shifting the training focus towards on-policy exploration and annealing the influence of the off-policy expert data before overfitting on them. Such decay schedule has also proven successful in mitigating exposure bias (Zhang et al., 2019). Inspired by scheduled sampling (Bengio et al. 2015). we generalize the principle of mixing expert and self-generated data from the token level to the loss formulation, effectively bridging the distributional gap between training off- ool-policy samples and performing on-policy rollouts.
Beyond the Loss Coefficient μ Empirical comparisons (refer to Section 4 for more details) demon- strate that applying a decay schedule to i yields notable performance gains over the SFT-then-B paradigm. At the same time, two key observations motivate extend beyond p.
4

---

Published as a conference paper at ICLR 2026
1.0
Off-policy
On-policy
1.0
0.8
1.25
W/
0.8
1.00
With IS
Reward
0.6
0.6
Value
Entropy
0.75
Reward
0.4
0.50
Value
0.2
0.25
0.0
100 200 300 400 500 600 700 800
Step
Step
Figure 4: Decaying the value of μ enables a smooth transition from off-policy imitation to on-policy optimization.
Figure 5: Comparisons of entropy loss be- tween pure RL and mixed RL that integrates expert data (with or without the IS strategy).
Firstly, as shown in Figure 4 the learning curve still reveals a "shift-readapt" progress, where the reward initially declines before subsequently increasing. These observations indicate that, despite improvements in performance, learning from off-policy expert data might still disrupt established patterns and stifle the model's capacity for genuine exploration during on-policy training.
Secondly, the response patterns of the model trained with CHORD-ji (as shown in Appendix B appear converge those of the expert model. Case studies reveal that CHORD-p compels the model to adopt the expert's s verbose response pattern wholesale, hence overwriting its own inherent conciseness. This indicates that while μ controls the overall influence of expert data, lacks fine-grained precision. As result, it forces the model to indiscriminately adopt expert patterns, which can create conflicts with its own established style.
Towards the goal of utilizing off-policy data as an incentive and guidance for the model to explore novel and effective reasoning paths, rather than merely as a target to imitate, we further integrate CHORD with token-wise, fine-grained weighting function o(-). forming dual-control mechanism together with the global coefficient i for controlling the influence of the off-policy expert data.
3.3 ENHANCING THE STABILITY OF OFF-POLICY LEARNING VIA 66-)
feasible solution for controlling the influence of off- off-policy expert data from fine-grained perspec- tive is to differentiate the tokens based on their generation probabilities ( [x, y2.). For example, Importance Sampling (IS) (Schulman et al. 2017) has been widely used for stably integrating off- policy data in RL, which suggests re-weighting the objective by the probability ratio between the target policy Te and the behavior policy Taample that generated the expert data. Formally, the objective function can be given as:
Iw'l sg
(v y22) Tsample (y/ (xr, v222)
CSFT-IS (0) (e Darr
log Te ( w (x, v2e)
where sg(-) denotes the -oo-gradient operator. Note that the probabilities Tsample y. for the expert data DSFT are often unknown. Following the common practice Yan et al.) 2025 Wu al. 2025), we assume that the denominator is 1, treating the expert data as the ground broundd-ruite truth distribution.
From a token-wise perspective, IS enhances training stability by down-weighting low-probability tokens that could disrupt the established policy. As empirical observations shown in Figure mixing off-policy data without IS leads sharp rise in entropy, which implies that the model's established patterns are quickly disrupted by the unweighted off-policy data. However, we notice that IS can lead to a sharp collapse in policy entropy compared to pure RI,, which implies that can limit the exploration essential for the RL phase and trap the model in stable but suboptimal solution. The underlying reason is that IS prevents disruptive shifts the policy distribution by nown-weighting low-probability tokens, but it also aggressively reinforces existing gigobprobabylityy tokens while ignoring novel but low- boobbbibayyyity ones, thus causing the policy to become overconfident.
Stabilize Off-policy Data Training with d(-) To tackle this, propose fine- -raiggr por-token weighting function o(yf: To) that weights the learning signal for tokens at both ends of the probability spectrum, í.e., weig-weggh those tokens that are highly probable (to prevent
5

---

Published as a conference paper at ICLR 2026
entropy collapse) extremely improbable (to avoid disruption). More specifically, the weight for a given expert token is defined based on the policy's probability ps = To x. 32,), (22)), as follows:
<lyi: MT) - Pe - -P:),
(5)
which naturally forms parabolic curve that peaks at pt - 0.5 and decays to zero as pt. approaches 0 or 1. The SFT objective function can be updated as:
Iv'l
(0) = - (z.gg)) )x.y"")>Dar DaDr
o16i m)) log Te Te(vilas lz, yee)
(6)
where =T)) modulates the gradient contribution of each token in the expert trajectory.
From an information information-theoretia perspective, the term (r - pc) can be viewed as a measure of the policy's 's uncertainty (Wang al. 2025) for the binary event of generating token yf. Therefore, this approach biases learning towards tokens where the policy is most uncertain, and creates a "learning sweet spot" that focuses the off-policy learning tokens that novel enough to be informative but not so divergent as to disrupt the established policy.
By replacing the static LSFT in the proposed hybrid loss function (defined in equation 3 with LSFT- we obtain the final objective function of CHORD, which applies a global coefficient f for adjusting the overall influence of expert data and fine-grained weighting function o(-) that helps enhance the stability when learning from off- off-policy data.
4 EXPERIMENTS
4.1 SETUP
Datasets, Models, and Evaluations We conduct experiments mathematical reasoning problems and practical tool-use tasks. (i) For mathematical reasoning problems, we utilize the OpenR1- Math-220k dataset (Hugging Face (025), from which sample 5k instances for SFT and 20k for RL, ensuring no overlap. Our policy model is O0we2..5-7BInnstruce whose response patterns differ significantly from the expert (Deepseek-RI). We evaluate in-domain generalization performance on the AIME24. AIME25, and AMC benchmarks (Li et al.
2024), and use MMLU-Pro (Wang et al. (ii For tool-use tasks, we conduct experiments on the single-turn instances of the ToolAce (Liu et al. (024) dataset. We sample 5k instances for RL and 500 for SFT, for which the expert trajectories are generated by querying the Deepseek-R with the same system prompt. We use L.aMA3.2--33 --3.A-33---nstruuu (Grattafiori et al., 2024) as our policy model, which also differs in response patterns from the expert (Deepseek-RI) R1). We evaluate the model performance on BFCL (Patil et al. 2024).
2024] to monitor the changes in general reasoning.
Baselines We compare the proposed CHIORD with a comprehensive set of baselines, including: (i) Original Model: The original Qwen2.5-78 -eeen2.5-B--Intruueee Instruct/LLaMA3.2 L.M.3.2--BBBB In-Itruccct model. (ii) SFT-only: The model fine-tuned on the SFT dataset. We focus on two specific configurations: SFT-lighh, trained for single epoch, and SFT-best, the peak-performing checkpoint on the test set found by searching over different learning rates and training epochs. (iii) RL-only: The model fine-tuned directly on the RI. dataset using the GRPO algorithm. (iv) SFT+RL. The sequential SFT-then-RI. paradigm. (v) [UFFY Yan et al. 2025): A method that integrates expert demonstrations within GRPO rollout groups and reshapes the importance sampling ratio. (vi) SASR (Chen al., 2025c): A method that probabilistically interleaves SFT and RL steps. It prioritizes SFT when the model's outputs are dissimilar to expert demonstrations, adapting the training focus dynamically.
For more details of the experimental setups, please refer to Appendix A
4.2 COMPARISONS
The proposed approaches implemented based on CHORD include (i) CHORD-JI: We employ decay schedule for the loss coefficient y to gradually transition from off-policy to on-policy learning, as
For math reasoning problems, we utilize 20k samples for training. whereas the original paper utilizes 45k samples and achieves scores of 50.9 on AMC, 17.7 AIME24, and 14.8 on AIME25. For tool-use tasks, LUFFY utilizes 5k SFT samples instead of 500.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparisons on reasoning problems and tool-use tasks.
Math & General Reasoning Problems
Tool-use Tasks
MMLU
BFCL Live
BFCL Non-live
BFCL
AMC
AIME24
AIME25
-Pro
Overall
Original Model
43.8
11.7
6,66
24.7
50.9
39.9
46.2
SFT-light
42.5
8.54
7.80
28.0
30.8
38.4
34.0
SFT-best
55.9
15.8
15.2
38.4
59.2
84.2
69.8
SFT-light RL
52.5
11.9
11.6
44.6
68.2
89.4
77.2
SFT-best RL
58.4
17.1
16.3
51.3
67.4
87.9
76.1
SASR
54.0
12.7
11.1
45.1
66.0
86.5
74.7
CHORD-JI
18.1
43.3
69.4
88.6
77.6
GRPO (Pure RL)
52.1
13.2
8.54
45.8
68.5
88.8
77.1
LUFFY
52.8
16.6
14.3
44.0
67.2
88.0
76.1
CHORD-O
62.5
18.2
17.2
56.2
69.9
90.2
78.5
detailed Section 3.2 and (ii) CHORD-O: We fix the value μ and further integrate the token-witg
weighting function ((.) to achieve a dual-control mechanism the influence of off off-policy expert data, as introduced in Section 3.3
Model Performance Overall, the comparisons summarized in Table demonstrate the effectiveness and superiority of CHORD on both reasoning problems and tool-use tasks.
Specifically, the experimental results reveal challenge within the SFT-then-RL. paradigm. We notice that minimal tuning on off-policy data (SFT-light) degrades performance, and more thorough SFT phase (SFT-best) achieves better results. However, the optimal timing for transitioning from SFT to RL can vary across different scenarios. For example, initiating RL from SFT-best yields superior performance on math reasoning problems. while SFT-light+RL performs better on tool-use tasks. This divergence confirms that the SFT-RL balance is highly task-dependent and needs extensive efforts for careful adjustment.
These SFT-then- -LL approaches are surpassed by CHORD-H,, which enables smooth transition from off-policy on on-policy learning rather than a rigid switch. Specifically, CHORD-p outperforms the strong SFT-best+RL. baseline across all math reasoning benchmarks, achieving improvements of +2.4 on AMC, +1 1.0 on AIME24, and +1.6 on AIME25, respectively. Besides, CHORD-j also achieves better overall results compared to these SFT-then-RL. baselines tool-use tasks. These results demonstrate the superiority of its unified learning design.
Further, CHORD-O achieves consistent outperformance over the baselines. These results demonstrate the effectiveness our dual-control mechanism in flexibly controlling the influence of off-policy expert data. CHORD-o selectively applies the SFT loss to non-disruptive tokens, integrating expert knowledge without compromising foundational abilities. This enables robust learning from both off-policy expert data and on-policy exploration, leading the best performance both reasoning problems and tool-use tasks.
Response Patterns We further compare the influence of expert data (generated by DeepSeek- RI) on response patterns across different approaches. As shown in Table expert responses are substantially longer than the original model's ''s both math 16, vs. 659 tokens) and tool-use tasks (315 vs. 147 tokens), SFT models (SFT-light and SFT-best) initially mimic this verbosity. However, subsequent RL can help mitigate the issues of overly lengthy responses by training the models to conduct on-policy exploration. The response length produced by SFT-light+RL. is much shorter than that SFT-best+RI. (1,322/1 vs. 4,830/489 tokens), as fewer epochs of SFT allow the model to retain its original response patterns. Besides, from Figure 6 similar trend, where the average response length initially increases to align with expert patterns and then gradually converges
observe that CHORD-ji exhibits
lower length on-policy training progresses.
On the other hand, Pure RL on instruct-tuned models lengthens math responses (from 659 to 1,423 tokens) while shortening them for tool-use (from 147 to 118 tokens). This suggests that the response pattern changes can be task- deseedeenndnnt: math problems benefit from detailed step-by-step reasoning, whereas tool-use tasks favor shorter, concise action sequences. Such task-dependent property also

---

Published as a conference paper at ICLR 2026
Table 2: Average response length on math problems and tool-use tasks.
Response Langth
100000
CHORD
CHORD
6000
Average Length
2000
Math
Tool-use
520
Stap
Expert Data
6,132
315
Original Model
659
147
Responee Langth
350
CHORD-u cHORD
SFT-light
9,966
259
SFT-hest
8,442
527
SFT-light RL
1.322
119
SFT-best RL
4,830
489
CHORD-jI
6,081
197
Stap
Pure RL
1.423
118
Figure 6: Comparisons of average response length on math problems (top) and tool-use tasks (bottom).
CHORD-O
2,444
120
affects the SFT/RI, synergy dynamics, as MATH tasks benefit from imitating an expert's long and verbose reasoning via SFT, while such patterns can be detrimental to tool-use performance, distinction further detailed in Appendix D.2 The result shows that the proposed CHORD-O strikes more nuanced balance: while it also learns to produce more comprehensive mathematical reasoning (2,444 tokens), it generates concise and efficient responses for tool-use tasks (120 tokens). This suggests that the tosen-wise weighting in CHORD-Ó enables the model to selectively integrate patterns from those of expert data in lusk-specifes manner. Qualitative analysis shown in Appendix ( also confirms the effectiveness of such a flexible design, suggesting that the proposed CHORD-O can go beyond simply mimicking the expert, and learn to selectively absorb reasoning patterns from the expert, while exploring its own response strategies.
4.3 ANALYSIS ON THE EFFECTS OF 14 AND o6-)
We provide analysis the effects of the coefficient μ and the token-wisg weighting function d(-).
Dynamic j Versus Fixed μ Figure 7 we compare the model performance when applying a dynamic schedule for μ (decreasing from 0.9 to 0.05 over the first 200 training steps and keeping unchanged in the following steps) against several fixed schedules in CHORD. We observe that applying fixed μ consistently results in poorer performance compared to dynamic μ.. This indicates that naively incorporating off-policy SFT data with a static weight does not effectively serve as a solution for simultaneously learning from off-polic data and on-policy exploration. In fact, it might fail to match Pure RL, which directly encourages an instruction model to follow its own reasoning patterns, highlighting the importance and necessity of controlling the influence of off-policy data.
Besides, while using a smaller value of μ (e.g., 0.02) can mitigate the performance degradation compared to larger values (e.g.. 0.1 and 0.5), it does not provide significant improvement over pure on-policy RL. With a fixed μ4, the model is consistently required to accommodate two potentially divergent reasoning patterns, which might pull in different directions and prevent it from converging to stable and high-performance state. The decay schedule for effectively resolves this conflict by creating smooth transition from off off-policy supervision to on-policy exploration.
We also explore a natural extension to adaptively adjust the loss weight μ based on on-policy rewards. Specifically, the weight is computed as f4 max (0, r reward_mee seward_mean). which phases out the SFT loss on the expert data as performance improves. The experiment results in Appendix B. iimply that an automated, reward-aware aware schedule for μ can work effectively but requires heavy tuning, which further motivates the need for fine- grained control.
Training Curve of CHORD-O In Figures and 9 we compare the entropy loss and rewards of Pure RL with those of CHORD-Ō (with fixed μ = 0.1), to illustrate their training dynamics.
From the changes in entropy loss, we can observe that by applying Φ(-), the model maintains great balance between exploration and exploitation while performing off- off-policy and on-policy learning simultaneously. On one hand, CHORD-0 prevents the entropy from collapsing prematurely, which may occur when the SFT loss forces the model become over-confident on high-probability tokens
8

---

Published as a conference paper at ICLR 2026
0.8
Pure RL after 300 5 Stcc
0.800
0.8
0.6
Entony Loea
0.400
CHORD.6
0.6
0.5
0,200
Fixe p=d.1 wi $1
Reward
3.44
0.3
Fladd 5-0.5
CHORD o
0,100
0.4
CHORD-$
0.2
Flxed p=0:1
Pure RL
0.1
Fined p.002
0.025
0.2
Fixed y= at
500 800
100
200
400
300 400 500
600
Step
Step
Step
Figure 7: Reward versus train- ing step for CHORD-jU and vari- ous fixed-j strategies.
Figure 8: Entropy loss versus training step for CHORD-O and baseline methods.
Figure 9: Reward versus train- ing step for CHORD-o and base- line methods.
from the expert data. On the other hand, it avoids large entropy spikes and training instability that may occur if the off-policy expert data drastically conflict with the current policy's predictions, as the performance curve remains stable throughout the training process. The rewards curve indicates that CHORD-O achieves stable and continuous increase in rewards, resulting in significantly better performance than Pure RI.. These results demonstrate that the proposed token-wise weighting function is crucial for effectively unifying the SFT and RI. phases.
Tuning j When Applying CHORD-O Empirical observations show that, when d(-) is used for fine-grained control over the influence of expert data, complex and decaying schedule for is no longer essential. CHORD-o is effective to work with a fixed value for ju (e.g.. 0.1 this study) since it inherently prevents both token-level overfitting and the disruption of established response patterns. The design of 6(.) simplifies the practical usage of CHORD by making robust to the specific choice of p.. In Appendix B.7 provide experiments tuning the schedule of μ in conjunction with o(-).
Principle for Instantiating (b-) It is worth noting that the proposed weight o(.) = Pt (1 Pt) serves conerete and interpretable instantiation following general principle: stabilizing ooflpy integration requires down-weij woightiggt the learning signal for tokens at both ends of the probability spectrum. This instantiation is computationally efficient, requiring only element-wise multiplication of existing forward-pass probabilities. As grounded in our empirical observations, by assigning negligible weight to tokens that the policy is already certain about (where pa close to 0 or 1), the proposed method prevents off-policy data from disrupting the model's established reasoning patterns and focuses updates on tokens where the model is still uncertain. Beyond the specific formulation of Φ(),, this general principle that enables stable and selective learning from off-policy data can potentially inspire more advanced weighting schemes that are suitable for different scenarios.
We also experiment with several variants of the token loken-weighting function (such entropy- based variants, clipping variants, and focal loss), with detailed experiment results and discussions presented in Appendix B.2 We acknowledge that this <i-) boken-wise weighting design primarily illustrates a generalizable principle. While it is unlikely that a single universally optimal design exists across all tasks, datasets, and models, our empirical results confirm that our design nonetheless serves as highly effective and robust instantiation.
4.4 FURTHER ANALYSIS
Varying Expert Data Source We investigate the effect of using two different expert sources: the powerful DeepSeek-Re R1, and the weaker Qwen2.5-72B- -nntttree whose response pattern is closer to the LLaMA3.2-30 -n--nrtt policy model. Experimental results demonstrate that our proposed methods, CHORD-ji and CHORD-O) outperform Pure RI. and SFT+RL baselines regardless of the expert. We also observe that methods which rely more heavily on expert imitation (e.g., SFT+RL and CHORD-()) can yield greater gains when the expert is stylistically similar to the policy model. This aligns with our insight: the effectiveness of unifying SFT and RL depends not only on expert data quality but also on the degree of pattern shift it introduces. For detailed results and discussion, please refer to Appendix B.3
Extending to Non-verifiable Domains To test the generalizability of CHORD beyond verifiable tasks, we conduct experiments on Medicine, a medical puustion-answering dataset that lacks de- terministic verification. The results show that both CHORD-jI and CHORD-O significantly outperform

---

Published as a conference paper at ICLR 2026
pure RL. CHORD-O achieves faster convergence and higher final rewards, while CHORD- exhibits a similar "shift-readapt" pattern as observed in the main experiments (Figure IO Appendix [B (B.4). These findings validate that our approach successfully generalizes to more diverse, non-verifialle domains. Refer to Appendix B.4 for more details.
Training Weaker Policy Models While the effectiveness of on-policy exploration (RL) is often limited for weaker models, our experiments reveal that their capability to absorb knowledge from expert data simultaneously degrades. Our experiment on Qwen2.5-3B-instruct shows that due to this dual limitation, a weaker model tends more to suffer from a performance collapse when training on the same off poolcy expert data. In such a setting, naive imitation fails and a simple SFT+RL combination proves unstable, whereas our CHORD-O provides a robust objective to navigate this trade-off and maintain learning stability. We acknowledge that with larger quantities of higher-quality expert data, weaker model might overcome this imitation bottleneck, potentially making SFT-leaning methods more favorable. Conversely, this also suggests that our method becomes particularly advantageous when dealing with weaker models under limited expert data, highlighting its strong capability to flexibly balance this trade-off and create a robust synergy between SFT and RL. We defer detailed experiment results and analysis to Appendix B.5
RELATED WORKS
Recent advancements in RL show significant success in complex reasoning tasks (Guo al. 2025
Shao et al. 2024 Lambert et al., 2024). However, RL-based exploration is often constrained by the model's initial knowledge, making it difficult for the model to discover superior reasoning path- ways (Yue et al. 2025). Incorporating off-policy expert data into the on-policy RI. loop is a promising strategy address such exploration challenge. Some studies directly mix expert data with self-rollout generations, either through simple dataset mixing (Li & Khashabi (025). or mixing expert trajectories into on-policy rollout groups (Yan et al.) 2025 [uu et al. (2025) while others use expert data to guide
generation (Liu et al. 2025a Zhang et al.) 2025b Huang et al. 2025). A third category interleaves
RL updates with SFT steps on expert data, either on predefined or adaptive schedule (Chen et al.,
2025c), or for challenging examples (Ma et al. 2025). More recently, SRFT (Fu et al. (025) proposed a unified framework that combines data mixing with sample-level SFT loss. In this study, we focus on tuning instruct model that already establishes its own response pattern, which can be more challenging yet practical scenario compared to existing works that finetune a base model (Yan et al. Fu et al. 2025). For more comprehensive literature review, please refer to Appendix
2025
6 CONCLUSIONS AND FUTURE DIRECTIONS
In this study, we identify that the sequential SFT-then-RL paradigm often yields suboptimal perfor- mance by disrupting established patterns with off-policy expert data. To address this, we unify SFT and RL through the lens of on-policy versus off- policy learning and propose CHORD. By analyzing the influence of expert data at both the holistic and granular levels, CHORD first integrates a global coefficient/ y manage off-policy expert influence, enabling smoother transition from imitation to exploration. CHORD then introduces token-wise function (-)) to selectively absorb expert knowl- edge by down-weighting tokens that are either already highly probable or extremely improbable. Extensive experiments demonstrate that this unified approach achieves notable improvements over existing sequential pipelines and various baselines.
Looking forward, our work highlights several avenues for future research. First, given the inherent high variance and parameter sensitivity typical of RL processes, exploring further algorithmic advancements to fully stabilize the training dynamics represents promising direction. Specifically within our framework, since configurations for fu and (b-) can vary across setups, discovering more effective adaptive tuning functions reduce hyperparameter dependence remains a practical objective. Furthermore, while our analysis of pattern shifts is primarily empirical, a deeper understanding of how diverse CoT patterns influence learning would provide valuable insights for tracking training dynamics. Finally, we envision extending this unified paradigm to incorporate heterogeneous mixtures of expert data sources, enabling comprehensive study of how different model-generated reasoning patterns emerge and evolve throughout the training process. We hope our work inspires the community to continue refining unified post-training paradigms across broader spectrum of applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We would like to thank Qi Liu, Zexi Li, and Ao Li for their suggestions. We also thank the anonymous reviewers and Area Chairs for their constructive feedback during the review process.
REPRODUCIBILITY STATEMENT
We release our implementation to inspire further research.
REFERENCES
Charles Arnal, GaÄTtan Narozniak, Vivien Cabannes, Yunhao Tang, Julia Kempe, and Remi Munos. Asymmetric reinforce for off- off-policy reinforcement learning: Balancing positive and negative rewards. arXiv preprint arXiv:2506. 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022.
PhilipJ J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. International Conference on Machine Learning, pp. 1577-1594. PMLR, 2023.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506. 13585, 2025a.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? early investigation into training ri-like reasoning large vision-language models. arXiv preprint ar arXiv:2504.1 1468, 2025b.
Jack Chen, Fazhong Liu, Naruto Liu, Yuhan Luo, Erqu Qin, Harry Zheng, Tian Dong, Haojin Zhu, Yan Meng, and Xiao Wang. Step-wise adaptive integration of supervised fine-tuning and reinforcement learning for speciffi llms. arXiv preprint arXiv:2505. 13026, 2025c.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang. Wanlong Liu, Rongsheng Wang, and Benyou Wang. Towards medical complex reasoning with LLMs through medical verifiable problems. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACI. 2025, Vienna, Austria, 2025d. Association for Computational Linguistics.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shenghang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning, 2025.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, et al. Rl-plus: Countering capability boundary collapse of 1lms in reinforcement learning with hybrid-policy optimization. arXiv preprint arXiv:2508. 200.:22500.0022,, 2025.
Yugian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506. 19767, 2025.
Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei. Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al. Agentscope: A flexible yet robust multi-agent platform. arXiv preprint 2402 12.402..40.0 /4034, 2024.
11

---

Published as a conference paper at ICLR 2026
Dechen Gao, Hang Wang, Hanchu Zhou, Nejib Ammar, Shatadal Mishra, Ahmadreza Moradipari, Iman Soltani, and Junshan Zhang. In-ril: Interleaved reinforcement and imitation learning for policy fine-tuning, arXiv preprint arXiv:2505.10442, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymiss, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprini arXiv:2506.04178 2025.
Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. arXiv preprint 1.Xiv:2507.17746, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: incentivizes reasoning in lims through reinforcement learning. nature, 645:633-638, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24220
2025a.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024.
Xiao Hu, Xingyu Lu, Liyunn Mao, YiFan Zhang, Tianke Zhang. Bin Wen, Fan Yang, Tingting Gao, and Guorui Zhou, Why distillation can outperform zero-rl: The role of flexible reasoning, arXiv preprint arXiv:2505.21067 2025b.
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01677 2025.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URI https //github.co com www.o///ggggggaa////ogooooo rl
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1224 2013.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant pwwprsattionn-dmmcrattiiii large language model alignment. Advances in Neural Information Processing Systems, 36:47669-4766 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training arXiv preprint arXh::241..15124 2024.
Jack Lanchantin, Angelica Chen, Janice Lan, Xian Li, Swarnadeep Saha, Tianlu Wang. Jing Xu, Ping Yu, Weizhe Yuan, Jason E Weston, et al. Bridging offline and online reinforcement learning for 1lms. arXiv preprint arXiv:2506.21495, 2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 13:9, 2024.
Tianjian Li and Daniel Khashabi. Simplemix: Frustratingly simple mixing of off-and on on-policy data in language model preference learning. In Forty-second International Conference on Machine Learning, 2025.
12

---

Published as a conference paper at ICLR 2026
Zichong Li, Chen Liang, Zixuan Zhang, llgee Hong, Young Jin Kim, Weizhu Chen, and Tuo Zhao. Slimmoe: Structured compression of large moe models via expert slimming and distillation. arXiv preprint arXiv:2506.188449 2025.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning, arXiv preprint arXiv:2505.16984 2025a.
Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong WANG, et al. Toolace: Winning the points of I1m function calling. In The Thirteenth International Conference on Learning Representations, 2024.
Zihan Liu, Zhuolin Yang. Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotroo 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506. 3284, 2025b.
Lu Ma, Hao Liang, Meiyi Qiang. Lexiang Tang, Xiaochen Ma. Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527. 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
Ofrr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Xuchen Pan, Yanxi Chen, Yushuo Chen, Yuchang Sun, Daoyuan Chen, Wenhao Zhang, Yuexiang Xie, Yilun Huang, Yilci Zhang, Dawei Gao, et al. Trinity-rft: A general- purpose and unified framework for reinforcement fine-tuning of large language models. arXiv preprint 3XXx:2505.17826 2025.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544-126565, 2024.
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning curated data is reinforcement learning (and can be improved), arXiv preprint #rXiv:2507.12856, 2025.
Nicolas Le Roux, Mare G Bellemare, Jonathan Lebensold, Amaud Bergeron, Joshua Greaves, Alex Fréchette, Carolyne Pelletier, Eric Thibodeau-Laufer, Sándor Toth, and Sam Work. Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for 1lms. arXiv preprint arXiv:2503. 14286, 2025.
Florian Schmidt. Generalization generation: A closer look at exposure bias. Proceedings of the 3rd Workshop on Neural Generation and Translation, pp- 157-167, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Yunhao Tang, Taco Cohen, David W Zhang, Michal Valko, and Rémi Munos. Rl-finetuning Ilms from on-and off- off-poliiy data with single algorithm. arXiv preprint arXiv:2503.19112 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research Foundation Models. https://erfm. stanford. cdd/20223/03/11/aappaac html, 3(6)7, 2023.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang, Zhenru Zhang, et Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprini arXiv:2506.0199 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng. Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: reinforcement learning perspective with reward rectification. arXiv preprint vrXiv:2508.0562 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under pof-py guidance. arXiv preprint arXiv:2504. 14945, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxin Yang. Jingren Zhou, Junyang Lin, et al. Qwen2.5 technical report. arXiv preprint 2xXv:2412.151155, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self- infrrrveoeeett.... arXiv preprint rrXiv:2409.12122 2024b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.0338% 2025.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, al. Yi: Open foundation models by 01. ai. arXiv preprint 2031v22403.04652 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in ilms beyond the base model? arXiv preprint ar01v:2504. 13837, 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforeement learning for open base models in the wild. arXiv preprint ar arvv 18892, 2025.
Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. cemootronrrssaurc-ttooo with reinforced reasoning. arXiv preprint
Exploring tool-using language models 2025a.
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 4334 4343, 2019.
14

---

Published as a conference paper at ICLR 2026
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning. arXiv preprint orXiv:2506.17211, 2025b.
Yaowei Zheng, Richong Zhang, Junhao Zhang, YaYanhan Yanhaa, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400-410, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-5555 55021, 2023.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETUPS
A.1 HYPERPARAMETERS
Across all experiments, we adopt the Adam optimizer with 31 = 0.9, A2 0.999. The learning rate tuned within {1 10-6 5 x 10-5 1 10- and the temperature for both rollout and evaluation is 1.0. The max response length set to 16k tokens. For SFT, we train for maximum of epochs. For RL, we employ "strict on-policy training" similar to (Liu et al. 2025b), where we generate K rollouts per prompt before each policy update.
For mathematical reasoning problems, the batch size for SFR/RL is 64/32, and the maximum number of RL steps is 1,500. For tool-use tasks, the batch size is 96 for both RL and SFT, and the maximum number of RL steps 100. The y decay schedule is to decrease from 0.9 to 0.05 over the first 30 training steps.
A.2 IMPLEMENTATION DETAILS
In our experiments, the reward function is tailored to the task-specific requirements. For mathematical reasoning problems, we use hierarchical reward scheme encourage both correctness and format adherence. To guarantee the precision of our correctness evaluation, we exclusively sample problems that have integer answers when preparing our dataset. A response receives reward of +1.0 for correct final answer. If the format correct (e.g., step-by-step reasoning ending with a boxed answer) but the answer is wrong, it receives a neutral reward of 0.0. A small penalty of -0.1 is applied for responses that are both factually incorrect and improperly formatted. Finally, we penalize overly long and inconclusive responses (Yu al., 2025), and apply strong penalty of 1.0 for exceeding the predefined token limit without final answer. For tool-use tasks, we employ a simpler binary reward. response given a reward of + 1.0 if completely correct, and 0.0 otherwise.
We implement SFT algorithms based on LLaMA- Factory (Zheng et al. 2024), and implement RL algorithms based on Trinity-RFl -FT (Pan al. 2025). Experiments are conducted on NVIDIA A100 GPUs and 8 NVIDIA H20 GPUs.
For evaluation, we adopt accuracy as the metric. To avoid high variance in results and ensure fair comparisons, we report avg@32 32 AIME24 and AIME 25, and @vg@8 on AMC, respectively. Reported results are on the best checkpoint determined by the validation set.
A.3 PROMPTS
Prompt for Math Problems
The adopted prompt for math problems shown below.
Example: Prompt for Math Problems
<I iin_sssattt/// >ssstem
You are a helpful assistant that solves MATH problems. You should first think about the reasoning process in mind and then provide the user with the answer. You should present your reasoning process using the format: <think> your reasoning process here... </think> \n first. You should always include your final answer \boxed() } as closed-form results.<| im_end|>
<I iiimimstttt ir__strrr___ >user
1. A bus leaves the station exactly 7:43 a.m. and arrives its destination exactly 8:22 a.m. on the same day. How long, in minutes, was the bus trip?< imm_mddd
im__stare rt >assistant
For the performance of the base model, we report the higher score achieved using either the above prompts for math problems or the default prompt provided by Qwen (Yang et al. 2024b): "Please reason step by step, and put your final answer within \boxed
Prompt for the MMLU-Pro Dataset The adopted prompt for the MMLU-Pro dataset is shown below. We use the same system prompt as for the math problems, except that for multiple-choice
questions, we modify the answer format to require the corresponding integer as the response.
16

---

Published as a conference paper at ICLR 2026
Example: Prompt for MMLU-Pro Question
< im_start >system
You are a helpful assistant solves MATH problems. You should first think about the reasoning process in mind and then provide the user with the answer. You should present your reasoning process using the format: <think>\n... your reasoning process here... </think>> An first. You should always include your final answer in \boxed ) closed-form results.<| im__ndd
1im imm_ttart >user
Let V be the set of all real polynomials p(z). Let transformations T. S be defined on V by T p(z) rp(æ) and S p(z)- ('(x) = d/ dzp(z), and interpret (pT)))())) as ([[(p))))) Which of the following is true? Below are multiple choice options. You should answer your choice by selecting the index of the option number:
0. ST + TS is the identity map of V onto itself.
1. T.TS-
2. ST
3. ST TS
4. ST = T
5. ST =0
6. ST =SS
7. ST - TS is the identity map of V onto itself.
8. TS T
9. ST = S <I Lnn_edd
im start Passistant
Prompt for the Tool-use Tasks For the tool-use tasks, we follow (Zhang et al. 2025a) to adopt their experimental setup and use the prompt provided in their Figure 8. This prompt consistently applied to train the LLaMA3.2-3B-Ime
policy model and to generate SFT data with the DeepSeek-RI
expert model.
B EXPERIMENTAL RESULTS AND ANALYSIS
B. ADAPTIVE TUNING μ
In addition to the fixed decay schedule for we explored an adaptive strategy to dynamically adjust the SFT loss weight based the model's ongoing performance, as measured by the average reward. We conducted experiments to validate this idea.
On the Tool-use task, we implemented strategy where μ adjusted based on the mean reward of the rollouts. Specifically, for a given reward threshold the new μ is calculated as μ max(0, reward mean). This mechanism ensures that as the model's average reward surpasses the threshold, the SFT component is gradually phased out (μ' - 0), allowing the training to focus purely on RL. We tested this with thresholds 0.5 and r 0.7.
Table 3: Performance comparison of Adaptive μ strategies on ToolACE.
Method
Live
Non-live
Overall
Original Model
50,9
39,9
46.2
SFT-best
59.2
84.2
69.8
SFT-best RL
67.6
87.9
76.1
GRPO (Pure RL)
68.5
88.8
77.1
CHORD-o (Ours)
69.9
90.2
78.5
CHIORD- (Fixed Schedule)
69.4
88.6
77.6
Adaptive (7 0.5)
69.7
89.4
78.1
Adaptive a (T 0.7)
65.9
88.6
75.6
17

---

Published as a conference paper at ICLR 2026
The results, presented in T show that setting the reward threshold 0.5 yields strong overall score of 78.1, which highly competitive with our main approach using a fixed decay schedule. This indicates that dynamically reducing the SFT contribution as the model improves is viable and effective strategy.
However, also found that this configuration is still dependent on task-specific hyperparameter tuning. When we set a higher threshold of 0.7, performance degraded significantly. This is likely because the policy was subjected excessive SFT even when achieving moderately high rewards, disrupting the optimization process.
These experiments serve proof of concept, demonstrating that automated, reward-aware schedule for can work effectively and represents a logical extension of our core ideas. However, since this method still requires tuning another hyperparameter(the reward threshold), its practical implementation is not necessarily simpler to tune. Therefore, we present this as preliminary exploration into adaptive mixing coefficients, leaving more thorough investigation robust and generalizable adaptive schemes as promising direction for future work.
B.2 VARYING THE & FUNCTION
To validate the robustness of our approach and explore alternative weighting strategies, we conduct ablation studies comparing different variants of the token-wiin wise weighting function Φ(-) against proposed method. We evaluate these variants on both the tool-use and mathematical reasoning tasks.
Evaluated e Variants. We compare the following token-wise weighting strategies:
CHORD-O (Ours): Our proposed method, with <(p) p (1 11-98 p).
Entropy Top:
Only trains on the top 5% tokens with the highest entropy, setting d(.) = 1 for these tokens and <(-) = 0 for others.
Entropy Norm: Normalizes the SFT loss weights based entropy magnitude, with o(p:) x (t).
IS Clip: Applies importance sampling correction but clips tokens with pe 0.4.
Focal Loss: Adapts focal loss (Lin al. 2017) to the SFT context, giving higher weight to tokens with lower probability: <(p)
Experimental Setup. For the mathematical reasoning experiments here, we relax the strict on- policy training protocol: we synchronize the policy model every 2 training steps (instead of after each update) and increase the number of rollouts per prompt to 16. Training is conducted for 400 steps. For tool-use tasks, we maintain the same setup as described in Appendix A
Results and Analysis. Table 4 presents the results the ToolACE benchmark, while Table shows the performance on mathematical reasoning tasks.
Table Performance comparison of different & function variants on ToolACE (BFCL benchmark).
Method
Live
Non-live
Overall
GRPO (Pure RL)
68.5
88.8
77.1
CHORD-O (Ours)
69.9
90.2
78.5
Entropy Top
69.1
89.4
77.8
Entropy Norm
69.6
89.4
78.0
Clip
66.2
89.1
75.9
Focal Loss
65.6
84.0
73.4
The experimental results reveal several interesting patterns across the different weighting strategies:
CHORD-o (Ours): Our proposed method achieves consistently strong performance across both tool-use and mathematical reasoning benchmarks, demonstrating its effectiveness and robustness.
18

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison of different ó function variants on mathematical reasoning tasks.
Method
AMC23
AIME2024
AIME2025
GRPO (Pure RL)
55.0
12.6
7.3
CHORD-O (Ours)
59.7
14.0
14.2
Entropy Top
58.8
17.2
13.8
Entropy Norm
52.5
15.0
9.2
IS Clip
55.0
13.9
12.0
Focal Loss
34.4
3.8
4.2
Entropy-based Variants (Entropy Top & Norm): These methods validate the intuition of focusing uncertain tokens. Entropy Top shows particularly strong performance AIME2024 (17.2), proving that selectively emphasizing high-entropy tokens can effectively integrate expert knowledge. However, their performance gains not consistent method across all benchmarks.
Clip: This variant, which clips high-probabillty tokens, shows limited effectiveness and even underperforms the pure RL baseline on the tool-use task. This suggests that simply clipping tokens not a sufficiently nuanced strategy.
This strategy. which aggressively up up-weights low low-probability (high-surprise) tokens, leads to severe training instability and significant performance collapse on both task types. This confirms our hypothesis that giving excessive weight to tokens the model deems unlikely can disrupt its learned reasoning abilities and lead to overfitting on expert patterns.
Focal Loss:
These results highlight the importance of fine-grained control in token-wise weighting. While various strategies can provide improvements over pure RL in specific scenarios, the choice of weighting function significantly impacts both training stability and final performance across different task domains. We note that our proposed s(.) instantiation represents one effective realization of the general principle of down-weighting tokens at both probability extremes. The varied performance of different variants suggests that there remains room for exploring alternative weighting schemes that may be better suited to specific task characteristics training scenarios, and hope these empirical observations can inspire future research in this direction.
B.3 VARYING EXPERT DATA SOURCE
To further validate the robustness and generalizability of our approach, conduct additional experiments using expert demonstrations generated by RI. This setup is particularly interesting because Swen2.5-72B-Instmuu model compared to DeepSeek-R1, produces responses with reasoning patterns that are more aligned with the base LLaMA3.2- 1.--A....BBBBB- 3---truutttt model. This allows to investigate how the choice of expert data source and specifically, the degree of pattern shift introduced affects the effectiveness of different training methods.
instead of DeepSeek- while being weaker expert
Table 6 presents the performance comparison on the BFCL benchmark using expert data from both DeepSeek-RI and Qwenn..5-7BBIInstnuue The results lead to several key insights.
When using expert data from Wiven2.5-72B-Instteee which exhibits reasoning patterns closer to those of L.L.MA32-333 -...MMA3..--B--nstmme Instrruct,, CHORD-y4 achieves improved performance (78.1 vs. 77.6) compared to using DeepSeck-RR data. This validates hypothesis that the distributional shift introduced by expert data is critical factor. When the expert's reasoning pattern is more compatible with the base model's existing policy, the progressive integration strategy of CHORD-ju more effectively leverage this alignment, leading to better final performance.
Despite the weaker quality of Qwenn.5-72B-BB Instruct compared DeepSeck-RI, CHORD-ó maintains strong and consistent performance (78.3 vs. 78.5) across both expert data sources. This demonstrates the robustness of the token-wise weighting mechanism, which allows the model to selectively absorb
19

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison on BFCL benchmark using different expert data sources.
Methed
Live
Non-live
Overall
GRPO (Pure RL)
68.5
88.8
77.1
Original Model
50.9
39.9
46.2
CHORD-j (DeepSeek-RI)
69.4
88.6
77.6
CHORD-j (Qwen2.5-72B)
68.8
90.6
78.1
CHORD-o (DeepSeek-R1)
69.9
90.2
78.5
CHORD-O (Qwen2.5-72B)
68.9
90.9
78.3
SFTThest + RL (DeepSeek-R1)
67.4
87.9
76.1
SFT-best + RL (Qwen2.5-72B)
68.1
90.6
77.7
usefuul patterns while mitigating the negative effects of distributional mismatch, regardless of the expert's absolute strength or stylistic differences.
Interestingly, the baseline SFT-best RL method also shows notable improvement when using Swen2.5-72B-Instruu data (77.7 vs. 76.1). This further corroborates our core motivation: the effectiveness of SFT is not solely determined by the quality of expert demonstrations in isolation, but also heavily influenced by the degree of pattern shift they introduce relative to the base model. A smaller pattern shift, even from weaker expert, be more beneficial than larger shift from stronger but stylistically divergent expert.
Pattern Examples To provide qualitative insight into these pattern differences, we present example responses from DeepSeek-R1, Qwen2.5-72B-Instreee and the LLaMA3.2-3B-Instme model after pure RL training below.
Example: Response from DeepSeek-RI (Stronger Expert, Larger Pattern Shift)
<think>
Okay, let me figure out how to respond to the user. The user mentioned they have important fact about the Mars climate that they want to store. The assistant asked for the specific fact, and the user provided it: "Mars has the largest dust storms in the solar system which can last for months and can cover the entire planet."
Now, the assistant's 's role is to help store this information. Looking at the available functions, there's storeMemory function that takes "knowledge" parameter. The user's fact the knowledge to be stored. So, the correct action is to call storeMemory with the provided fact as the argument. There's no other function needed here since user just wants to store the information. I need to make sure the parameters are correctly formatted as JSON object with the "knowledge" key. No other parameters are missing here. So, the tool call should be straightforward.
</think>
<tool Looo__cal>>_> [("name": "storeMemory". "arguments": ("knowledge": "Mars has the largest dust storms in the solar system which can last for months and can cover the entire planet." ]]
c/tool_calls iim_end
Example: Response from Siven2.5-72B-Instte (Weaker Expert, Smaller Pattern Shift)
<think>
The inquirer wants know the recent disclosed securities report for Apple Inc. with the company code AAPL'. I can use the 'getDisclosedToday" function to fetch this information. However, since the inquirer did not specify date range, will use the current date as both the start and end strings.
</think>
<tooL_call> [[ [["name": "geDDisclosedToday" "arguments": ("company_com code"; "AAPL", "start_string" "2023-04-01". "end_string": "2023-04-30") </tool_call> < <lim_ndd>>
20

---

Published as a conference paper at ICLR 2026
Example: Response from LLaMAA32-3B-Insite (Pure RL trained)
<think>
To answer the question, I need use method retrieve list of available phone brands from the available database.
</think>
<tool _all_ [("name":
"Get All Phone
Brands",
"arguments":
{) </tool_call
< <lim_end|>
As illustrated in these examples, DeepSeek-R1 produces much more verbose and elaborate reasoning with extensive meta-commentary, while Bwen2.5-72B-Instrrue adopts more concise style that is closer to the direct, structured approach learned by LLaMA3,2-3B-Imgg This qualitative analysis confirms that Qwen2.5-72B Instruct introduces smaller pattern shift, which aligns with the quantitative improvements observed in Table 6
through pure RL training.
B.4 NON- VERIFIABLE TASKS
To further test the generalizability of CHORD beyond verifiable tasks, conduct additional exper- iments on the RaR-Medicine dataset (Gunjal et al. (025). a medical question-answerrine task that requires reasoning and explanation without deterministic verification.
We perform training on the Qwen2.5-775 7-1-I-trrBn model for 200 steps, with both SFT and RL batch sizes of 96, rollouts per prompt, learning rate of x 10-6, and the μ decay step to 50. We use Qwen3-30B-344 -nn-tr-I as the judge model. The expert demonstrations are sourced from the English subset of the medical-ol -reasoonigg dataset (Chen al. 2025d), which contains high-quality reasoning traces for medical questions.
Table 7 and Figure 10 present the experimental results. Both CHORD-H and CHORD-O significantly outperform pure RL, achieving testset scores of 80.6 and 81.3 compared to 76.8 for pure RL. Figure further show that CHORD-O achieves faster convergence and higher final rewards compared to pure RL, indicating more efficient exploration guided by expert demonstrations, where CHORD-- possesses a similar readdttreddatt pattern similar to the main experiment. These results validate that our approach can further generalize to more diverse post-training domains.
0.85
0.80
Table 7: Comparison of test score and response
0.75
length on RaR-Medicnn task.
0.70
0.65
Score
Length
0.60
Expert Data
550
0.55
CHORD-H
0.50
CHORD-6
Original Model
67.5
402
0.45
Pure RL
76.8
685
20
40
60
80
10012014016018000
CHORD-j4
80.6
1395
Step
CHORD-O
81.3
1128
Figure 10: Reward curves for training on the dataset.
B.5 SFT/RL SYNERGY FOR WEAKER POLICY MODELS
important consideration is how the SFT/RL. synergy works when the initial policy model less capable. Intuitively, one might assume that for weaker model, supervised fine-tuning (SFT) on expert data would become more critical, as the model's own on-policy exploration likely to be less effective.
However, our experiments reveal a more nuanced reality: a weaker model can also struggle to effectively absorb knowledge from expert data. As shown in Table 8 naively fine-tuning the weaker Qwen2.5-3BB 3-nn2.5-3B-Intttiti 3--Instruct model with SFT leads to a performance collapse(RL settings are similar to Appendix B.2). This is in stark contrast to the result observed when training the Qwwen.557BB-nnstme Instruct
21

---

Published as a conference paper at ICLR 2026
model, whose performance significantly improves with the same 5k SFT samples (e.g., AIME2024 accuracy rising from 11.7% to 15.8%). For the weaker model, while Pure RL still provides a consistent performance lift, naive SFT+RL combination proves unstable. This instability highlights a dual limitation: a weaker model is constrained not only in its on-policy exploration but also in its capacity absorb off-policy expert data, making the trade-off between exploration and imitation challenging to navigate. In such a setting, the CHORD-O method successfully maintains learning stability by providing a robust objective balance this trade-off. We acknowledge that with larger quantities of higher-quality expert data, weaker model might overcome this imitation bottleneck, potentially making SFT-leaning methods more favorable. Conversely, this also suggests that our method becomes particularly advantageous when dealing with weaker models under limited expert data, demonstrating its strong capability flexibly adapt and create a robust synergy between SFT and RI even when naive imitation fails.
Table 8: Performance on MATH tasks with a weaker policy model (Qwen2.5-3B- Instrruuc)....
Method
AMC23
AIME2024
AIME2025
Original Model (3B)
33.1
4.8
1.6
SFT
22.5
1.9
1.0
Pure RL
39.1
7.0
2.4
SFT+RL
36.2
6.2
3.7
CHORD-O (Ours)
41.9
7.9
4.0
B.6 DIVERSE MODEL ARCHITECTURES
To assess the broader applicability of our method, we extended our evaluation to model with distinct architecture and origin: the Phi-mini-MoE-instruuc model (Li et al. (025) (a light-weight Mixture of Experts (MoE) model with 3.8B total, 1.IB active params). This experiment also tests our method's effectiveness beyond the dense Qwen and LLaMA models.
As shown in Table the MoE model exhibits a similar vulnerability to naive SFT in tool-use tasks, with performance collapsing significantly. In stark contrast, our CHORD-ó effectively achieves the highest performance and boosts the overall accuracy from 49.5 to 61.6.
These results show that method is architecture-agnostic and further demonstrates its effectiveness across diverse model families and architectures.
Table 9; Performance the Tool-Use task with
model on tool-use tasks.
Method
Live
Non-live
Overall
Original Model
42.3
59.2
49.5
SFT
18.2
45.0
29.6
Pure RL
51.2
69.3
58.9
SFT+RL
44.1
63.0
52.1
CHORD-o (Ours)
52.1
74.5
61.6
B.7 TUNING y IN CONJUNCTION WITH
The proposed CHORD employs dual-control mechanism: global coefficient and token-wise weighting function o(-). While this raises the question their joint scheduling, we find that the fine-grat fine-grained control from o(-) makes the framework more robust to the specific schedule of μ. This innovation alleviates the need for meticulous tuning of the global coefficient, simplifying the practical application of CHORD.
The aggressive decay schedule for μ (starting from a high value) was designed to manage the "shift- readapt" progression. However, since the weight function o(-) also aims to stabilize learning and prevent pattern disruption, such an aggressive start may be unnecessary. A more theoretically aligned approach would be gently introduce the expert data via warmup-then-decay (Hu et al. 2024)
22

---

Published as a conference paper at ICLR 2026
schedule for (e.g., warming up from 0 to 0.3 before decaying). This would align with the stabilizing nature of o(-).
0.8
0.9
Pure RL after 1O0 Steps
0.7
0.8
Mw
Reward
0.6
0.5
0.4
CHORD-tune-bae CHORD-$
CHORD-y CHORD-6
0.3
0.4
0.2
100
200
300
400 500 Step
600
700
800
10
20
30
40
50
60
70
80
90 100
Step
Figure 11: Reward curve comparison for Figure 12: Reward curves for training on the CHORR variants.
ToolAce dataset.
We compare these two schedules in Figure 11 Although CHORD-tune-ban buth- that leverages a more refined warmup-then-decat μ schedule yields slightly better reward progression during training, the final performance gap between the two approaches is not that significant.
This observation is consistent with our insight: the primary purpose of introducing o(.) ) is to enable expert data to continuously and stably guide exploration. By inherently preventing both the disruption of existing patterns and overfitting at a token level, 6(.) makes the aggressive expert-first approach (a large initial p)) less critical. The token-wise control provides stability, making the overall system less sensitive to the global trade-off hyperparameter. We argue that adopting Φ(-) not only improves stability but also simplifies the practical application of our framework by making it robust to the specific choice of the schedule.
B.8 EXPERIMENTAL RESULTS ON TOOL-USE TRAINING
We provide the training curves on tool-use tasks Figure 12 and more detailed experimental result on the BFCL benchmark in Table I0 The average performance reported in the BFCL benchmark is averaged by instance, meaning that categories with more instances have greater contribution the final average score. All methods are evaluated using the same system prompt format.
Table 10: Detailed performance comparisons on BFCL bench.
Non-Jive
Ovenall
Simple
Multiple
Parallel
Parallel Multiple
Simple
Multiple
Paralldd
Parallel Multiple
Live Avg
Non-live Avg
Overall
LLaMA3.2-1B 2-22--BBaaa Inssiccttt
52.3
51.8
25.0
12.5
38.5
45.5
22.5
22.5
50.9
39.9
46.2
SPT-light
8.3
50.5
46.0
37.5
77.0
77.0
SITT-light
22.9
67.5
08.5
50.0
90.3
95.5
86.0
85.0
68.2
89.4
77.2
SPT-best RI.
66.1
75.0
58.3
91.5
91.5
84.5
79.0
67.4
K7.9
76.
SASR
69.4
65.3
62.5
58.3
92.0
92.0
74,0
82.5
66.0
86.5
74.7
CHORDay
50.0
83.0
CHORN-9
85.0
94.5
69.9
B.9 EXPERIMENTAL RESULTS THE MMLU-PRO DATASET
We provide more detailed experimental result on the MMLU-me pro dataset in Table prompts for generating these results can be found in Appendix A.3
II The adopted
C DETAILED DISCUSSIONS OF RELATED WORKS
C.I FINETUNING FOR LLMS
SFT for LLMs. SFT has established itself as a cornerstone for aligning LLMs, primarily due to its conceptual simplicity and cost-effectivenness, making it favored approach within the open-source
23

---

Published as a conference paper at ICLR 2026
Table 11: Detailed performance comparisons on the MMLU-Pr dataset.
ZAG (O) calogery)
Astrage
Llastry
Phisopply
Quest 5-7 15-mnicctt
11:18
13:72
23:11
26.22
26:15
21.73
12:40
22.16
25.15
25.75
25.45
25.12
21:34
21012
24.71
$4.90
13501
31:10
-4.12
21.18
10.54
22.51
+8.76
59:39
-7.96
42.93
12.35
18.74
19.03
Mir Rt.
08:00
03:42
51:00
61:00
03.41
41.21
43722
66.32
13773
09:39
12:19
18.10
11.03
13:03
(4A)
12.57
44.73
42.56
55.0
52.31
51.49
46.10
36.40
30.99
45.00
CHoRDy
15384
18.71
31.55
43.36
36.16
30.71
34.20
54.00
45.14
66.05
5481
41.80
28.65
35.81
43.28
LIIEY Nan
N235
13.33
14.25
65:11
51.91
41.13
00
49.30
51.77
03:00
17.05
4190
CHOWBON
06.75
30.88
08 87
40.10
05955
11.19
5511
66
55
68.07
61 66
13:41
$5.09
13.84
$6.33
community for creating capable instruction-followiin models (Taori et al. 2023 Köpí ct al. 2023)
Early work emphasized the power of high-quality datasets (Zhou et al. 2023 Young al. 2024).
while the required expert curation is labor-intensive and costly. Moreover, to cover the diverse use cases of modern LLMs, the paradigm has shifted towards massive- scale SFT (Grattafioni et al. 2024
Lambert et al.] 2024). This trend makes it computationally prohibitive for many to fine-tune from a base model, promoting continued tuning on pre-aligned instruction models instead. Furthermore, the interplay between SFT and RI. has grown more complex, from recent methods like DFT (Wu et al. 2025) or iw-SFT (Qin & Springenberg 2025) that incorporate RL-inspired importance sampling into SFT, to reasoning models like DeepSeek-RI (Guo et al. (025) that strategically integrate both paradigms, highlighting that the optimal, principled integration of these methods remains a critical and open area of research.
RL for LLMs. Recent applications of Reinforcement Learning (RI.) for Large Language Models (LLMs) have expanded beyond traditional human preference alignment (Bai et al., [2022] Ouyang ct al. 2022). demonstrating significant progress complex reasoning domains such as mathematics and code generation (Shao et al. 2024 Yang et al. 2024bb Guo et al. (025). In particular, surge of recent work has focused on Reinforcement Learning from Verifiable Rewards (RLVR) (Lambert et al.) 2024; Guo et al.] (025). where rewards are derived from definitive outcomes like correct answers or passing unit tests. This paradigm has achieved remarkable results on various benchmarks. However, fundamental challenge persists in how RL can facilitate effective exploration to surpass the inherent capabilities of its base model (Yue et al. 2025). The search for novel solutions is often constrained by the model's 's pre-existing knowledge, limiting its discovery of superior reasoning pathways. To address this, introducing external expert data- either for distillation (Hu et af.) 2025b Liu et al. 2025b
Guha et al.] 2222), cold start (Guo et al. 2025). , or guide exploration towards diverse, high-quality
patterns (Yan et al.) (0255 Ma et al.) 2025) emerges as a promising approach to transcend these
limitations and unlock new mooblem-solving frontiers.
C.2 ON- AND OFF-POLICY REINFORCEMENT LEARNING
Combining On-policy and Off-policy Data in Traditional RL In traditional RL domains like robotics (Kober et al. (013)) or games (Mnih et al. 2015), combining on-policy and off-policy data a potent strategy. Methods ranging from alternating training phases (Gao et al. (025). to mixing data from separate buffers (Ball et al. 2023), or directly augmenting on-policy replay buffers with expert trajectories (Nachum et 20.77 have been proven useful. While such methods yield good results in the traditional RL fields, the discrepancy arises from two fundamental distinctions of LLMs: their strong initial priors, where aggressive off- off-policy updates risk disrupting established reasoning patterns, and their vast, autoregressive action space that radically increases the off- off-policy degree of expert data, especially for long reasoning chains, and invalidates the assumptions underpinning conventional off-policy algorithms.
Combining On-policy and Off-policy Data in RL for LLM Leveraging off-policy data to improve the sample efficiency is a well-established strategy in RL. Several studies have focused on leveraging stale, self- generated data by employing techniques such as refining importance sampling corrections (Tang al. 2025), mixing the optimization loss objective (Roux et al. |025]|Amal AAmal et al.) 2025). or adjusting the synchronization frequency between online and target policies (Lanchantin al. 2025)
and off pooicy gradients (Li & Khashabi] (025). modifying
24

---

Published as a conference paper at ICLR 2026
More closely related to work methods that leverage external expert data to guide the reinforee- ment learning process for LLMs. These methods can be broadly categorized. One strategy is direct
data mixing Yan et al. 2025 Dong et al. 2025; Li & Khashabi 2025). For example, SimpleMix (Li) & Khashabi 2025), operates within a DPO framework and combines off-policy and on-policy data via simple dataset-level sampling. LUFFY (Yan et al. 2025) the other hand, incorporates off- expert trajectories directly into the on-policy rollout groups within a GRPO framework. While such approaches expose the model to expert data, they also introduce significant constraints: they usually require strict prompt alignment between datasets or lack the dynamic, token-level weighting needed to manage severe distribution shifts. Another strategy involves using expert data as guidance for generation. For instance, UFT (Liu et al., 2025a) and BREAD (Zhang et al. 2025b) utilize supervised fine-tuning (SFT) trajectories as prefixes for on-policy rollouts; UFT progressively masks the suffix of the expert demonstration, while BREAD initiates new rollouts by branching from intermediate steps. A third category interleaves RL updates with SFT steps on expert data, either selectively for challenging examples (Ma et al. 22255 or based on probabilistic schedule (Chen et al. 2025c). Most recently, SRFT (Fu et al. (025) unifies these approaches into a single-stage framework by not only mixing SFT samples into the on-policy rollout groups but also applying a dedicated SFT loss whose influence is adjusted at the sample level.
Our work diverges from these methods in crucial aspect. The aforementioned approaches, including state-of-the-art methods like SRFT (Fu et al.) 2025). LUFFY (Yan et al.] (025), and Reift (Ma et al. 2025). primarily operate under a "zero-RL" paradigm, initiating training from base model with a nascent policy. In stark contrast, our work addresses the challenge of fine-tuning model that already possesses well-developed,
eestruction-oolloiing policy. This advanced starting point inherently creates a more significant distributional shift between the model's existing policy and the external expert data, thereby exacerbating the off-policy correction problem that our method aims to solve. For further empirical analysis and results, please refer to Appendix D.I
D FURTHER DISCUSSIONS
D.1 THE INFLUENCE OF OFF-POLICY DATA ON BASE VS. INSTRUCTION MODELS
The challenges of controlling the influence of off-policy data and maintaining training stability are significantly amplified when fine-tuning instruction models. This is mainly due to the established policy inherent in these instruction models.
Starting from Base Model vs. Instruct Model A base model. having been pre- mee-triied solely with a language modeling objective, lacks a coherent, task-specific policy for instruction following. It often has not yet converged data, the training process is akin to initial policy formation. The model learns a new skill without the risk of confficting with an existing pattern, thus avoiding significant instability during training.
particular response pattern. When learning from off-policy expert
In contrast, an instruction model has already developed a sharply-peaked policy. Training these models on off-policy expert data that may reflect different reasoning patterns introduces substantial distributioonal mismatch. The RL algorithm's 's efforts to reconcile this mismatch can result in large, disruptive policy updates, destabilizing the established policy and potentially leading to a collapse in performance.
Figure 13 provides empirical observation to support the above discussions. When learning from a mixture of on-policy and off-policy data, the reward of base model improves monotonically, displaying none of the instability issues that can affect instruction models under similar conditions.
Different from most existing studies (Zeng et al. 2025 Yan et al. 2025 Fu et al. 2025), which focus on the "Zero-RI"" setting that trains from a base model, this paper addresses a more challenging yet practical problem: how effectively integrate knowledge from off- oif-policy experts into model that already possesses an established policy. Training from a base model is not always feasible in practical applications. For instance, such methods are ineffective for tool-use tasks, the base model typically lacks the basic capability to follow the necessary instructions.
Applying "Zero-RL" Methods to Our Setting To demonstrate the unique advantages of CHORD for aligning already instruction-tuned models, we conduct additional experiments comparing our proposed CHORD with LUFFY (Yan et al. 2025) and SRFT (Fu et al. 2025) on the tool-use task.
25

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison with other "Zero-RL" methods on BFCL benchmark. CHORD significantly outperforms "Zero-RL" methods.
0.7
0.6
Reward
0.5
0.4
Method
Live
Non-live
Overall
0.3
GRPO (Pure RL)
68.5
88.8
77.1
0.2
From Base Model, μ=0.1
CHORD-p
69.4
88.6
77.6
0.1 wV
From Instruct Model, fa=0.1
CHORD-6
69.9
90.2
78.5
50
100
150
200
250
300 350
400
SFT-best
59.2
84.2
69.8
Step
SFT-best RL
67.4
87.9
76.1
LUFFY
67.2
88.0
76.1
Figure 13: Reward curves for training the base or instruct model with fixed μ 0.1.
SRFT
64.6
85.8
73.6
Note that both LUFFY and SRFT require strict alignment between expert demonstrations and RL prompts, as they directly mix expert trajectories into on-policy rollouts. Hence, we generate expert demonstrations for all 5,000 training prompts using DeepSeek-R. In contrast, CHORD only uses 500 expert demonstrations without requiring prompt-level alignment.
The results in Table 12 show that CHORD significantly outperforms both methods. As discussed in Appendix
C when applied to instruction-tun -eurnd models with established policies, directly mixing expert trajectories causes significant distributional mismatch, leading to training instability. Specifi- cally, LUFFY's upweighting of low-probability tokens on top of importance sampling can still cause policy shifts when the distribution gap between expert and policy gap is large. SRFT's uniform sample-level weighting cannot distinguish valuable tokens from irrelevant ones within a trajectory, leading inefficient and misguided updates. In contrast, adaptive weighting, enabling selective absorption of expert patterns while maintaining policy stability. These results validate that method achieves superior performance with better expert data efficiency and maintains training stability on instruction-tuned models, making it more practical for many more real-world applications.
<(-) function provides token-wise
D.2 ON DIFFERENT RELATED PERFORMANCE
The differing performance gains on the MATH and tool-use tasks stem from the fundamental distinctions between these two domains. We deliberately chose these tasks to represent two distinct paradigms, thereby demonstrating the robustness and flexibility of our proposed method.
The math domain benefits from complex, structured, and long-form reasoning. For such tasks, acquiring the necessary problem-solving patterns through pure on-policy exploration (i.c., Pure RL) can be inefficient in comparison. Supervised Fine- Tine-Tuning (SFT) on expert data is highly beneficial in this context, it directly exposes the model well-structured, step-by-step reasoning chains. This allows the model to efficiently learn complex reasoning frameworks that are difficult to discover from scratch. As discussed in Section 4.2 the model's performance math problems often correlates with its ability to produce more comprehensive and detailed reasoning steps, a pattern effectively taught by expert data. Therefore, method that can successfully integrate these expert reasoning patterns, like ours, is expected to yield substantial improvements.
The tool-use domain, in contrast, relies more on the exact tool call result rather than the reasoning process. In this setting, naive imitation of expert trajectories through SFT can even be detrimental, as an expert's solution may contain stylistic artifacts (e.g., verbosity) that are not conducive to performance. As shown in Table 2 and discussed in Section 4.2 tool-use tasks favor concise and efficient responses, pattern that Pure RL naturally learns by shortening response lengths. The primary challenge here is not just to imitate the expert, but leverage expert guidance accelerate exploration without being overly constrained or picking up suboptimal habits. The consistent performance gain of our method over the strong Pure RL baseline demonstrates its ability to achieve this delicate balance: successfully extracting useful signals from expert data while avoiding the pitfalls of naive imitation.
26

---

Published as a conference paper at ICLR 2026
These two domains present different challenges for unifying offline SFT and online RL, and our proposed method proves its effectiveness by excelling in both scenarios. learns to produce comprchensive reasoning for MATH while generating concise, efficient tool calls for tool-use tasks, demonstrating its capability to selectively absorb expert knowledge in a task-specific manner. This validates approach robust and versatile framework for diverse applications.
D.3 SCALING SFT IS NOT ENOUGII: THE NECESSITY OF ON-POLICY LEARNING
A crucial question is whether extensive SFT on high-quality expert data could eliminate the need for combining SFT and RL. Indeed, as the quantity and diversity of data increase, the problem of exposure bias (Zhang et al., 2019) can be alleviated, leading better generalization. And for knowledge-intensive tasks like MATH, model performance can be highly correlated with the volume and quality of SFT data. To investigate this, expanded the MATH SFT dataset from 5k to 20k examples, which substantially boosted the pure SFT model's AIME accuracy from 15% to approximately 24%.
However, even with larger volumes of SFT data, a principled transition to on on-policy learning remains critical for reaching the performance frontier. Recent literature (Liu al. 2025b) also shows that extensive SFT followed by RL fine-tuning is an effective strategy for maximizing model capabilities. By applying our SFT/RL combined approach, This demonstrates that RL is not redundant but complementary, enabling the model to refine its policy beyond the static distribution of expert data.
further elevate the accuracy from 24% 33%.
E
CASE STUDIES
For better understanding, we compare the generation patterns of RI-only (i.e., pure RL), SFT-only, and the proposed CHORD.
RL-only: The model trained solely with RL exhibits a concise and structured, yet ultimately rigid, reasoning pattern. It tends to follow a fixed template, such as beginning with "To..." and using connectors like "First" and "Next", and proceeds linearly without engaging in self-correction exploring alternative solution paths. While this approach leads to efficient responses, it may result in less robust solutions.
SFT-only and CHORD-JC: In contrast, the model trained solely expert demonstrations
verbose and exploratory. exhibits a "think-aloud style with frequent meta-commentary (e.g., "Let think..."), backtracking, and verification of intermediate steps. The generated responses are comprehensive, but often lack a concise structure. Note that the proposed CHORD-p has similar reasoning pattern SFT-only.
CHORD-0: exhibits a hybrid reasoning style that retains a clean and logical structure while selectively incorporating the expert's sophisticated verification strategies. As shown in the example below, it develops patterns such as "Alternatively.." and "Both methods confirm..." produce responses that are both well-structured and robust. We select the checkpoints at 800 steps as the models to generate the examples here.
Example: Response generated by Qwen2.5-7BBInstrrus
<think>
To calculate the duration of the bus trip, we need to subtract the departure time from the arrival time.
The departure time 7:43 a.m.
The arrival time is 8:22 a.m.
Subtracting the two times: hour
From the hours: 8
From the minutes: 22 43 = -21 minutes
27

---

Published as a conference paper at ICLR 2026
Since we can't have negative minutes directly, we borrow hour (60 minutes) from the hour part, making it hours and adding 60 minutes to the 22 minutes. Then, 60 + 82 minutes. So, subtract 21 minutes from 82 minutes: 82 = 61 minutes
Therefore, the bus trip duration is hour and 61 minutes, which simply expressed as 61 minutes.
</think>
\boxed (61) < <lim_endi>
Example: Response generated by the model trained solely with RI.
<think>
To determine the duration of the bus trip, we need calculate the difference in time between the departure time and the arrival time. The departure time is 7:43 a.m. and the arrival time is 8:22 a.m. on the same day.
First, let's identify the departure and arrival times:
Departure time: 7:43 a.m.
Arrival time: 8:22 a.m.
Next, we need to find the difference between these two times. We can do this by subtracting the departure time from the arrival time.
From 7:43 a.m. 8:00 a.m. the time from 7:43 8:00. This is 17 minutes because 8:00 7:43 = 17 minutes (since 8:00 is 17 minutes after 7:43).
Now we need to add the time from 8:00 a.m. to 8:22 a.m., which is 22 minutes.
So, the total trip time is 17 minutes (from 7:43 to 8:00) 22 minutes (from 8:00 to 8:22) = 39 minutes.
Therefore, the duration of the bus trip Aboxed ( 39) minutes. </think> The duration of the bus trip is Aboxed (3) minutes. end_
Example: Response generated by the model trained solely with SFT
<think>
Okay, let me try to solve this problem step by step. Let me think aloud so can explain my reasoning.
So, the problem is: A bus leaves the station at exactly 7:43 a.m. and arrives at its destination exactly 8:22 a.m. on the same day. How long, in minutes, was the bus trip?
Let me visualize this. Let me think about the time elapsed from 7:43 a.m. to 8:22 a.m.
Let me break this down. Let me think of it as two parts: from 7:43 a.m. 8:00 a.m a.m., and then from 8:00 a.m. to 8:22 a.m.
Let me check that. Let me see if that makes sense.
So, from 7:43 a.m. 8:00 a.m., how many minutes is that?
Let calculate that. Let me think, from 7:43 8:00 is 17 minutes. Let think, let count:
28

---

Published as a conference paper at ICLR 2026
7:43 to 8:00 is 17 minutes. Because 8:00 minus 7:43 is 17 minutes.
Let confirm that. Let me think. 7:43 to 7:50 is 7 minutes, and then 7:50 to 8:00 is 10 minutes. So, 7 10 = 17 minutes. Okay, that seems right.
So, that's the first part: 17 minutes from 7:43 a.m. to 8:00 a.m.
Then, the second part is from 8:00 a.m. to 8:22 a.m., which is 22 minutes.
So, adding those two parts together: 17 minutes + 22 minutes = 39 minutes.
Let me check that. Let me think, 17 + 22. Let me do the addition:
+ 12
So, that seems correct.
Let me think if there is another way to calculate this,
Let think, maybe using different approach.
Let me think, the total time from 7:43 a.m. 8:22 a.m.
Let me think, let me calculate the difference between 8:22 and 7:43.
Let me break it down hour by hour.
From 7:43 a.m. to 8:00 a.m. 17 minutes, as we already knew.
Then, from 8:00 a.m. to 8:22 a.m. is 22 minutes.
So, adding those together gives 17 + 22 = 39 minutes.
Let me think there a formula or a quicker way.
Let me think, maybe using the formula:
Total time = (End time hour Start time hour) 60 (End time minute- Start time minute)
Let me try that.
So, let me denote the start time 7:43 a.m.
End time as 8:22
So, End time hour
Start time hour
End time minute 22
Start time minute
So, Total time = (8 7) 60 + (22 43)
Let me calculate that.
29

---

Published as a conference paper at ICLR 2026
First, (8- 60 = 60 = 60 minutes
Then, (22 43) -21 minutes
So, Total time 60 (-21)= 39 minutes
Oh, that's interesting! So, using this formula, get 39 minutes well.
Let think, let check if this makes sense.
So, when we subtract the start time from the end time, we get:
8:22 7:43
Let me think of this as hours and 22 minutes minus 7 hours and 43 minutes.
But since we are subtracting times, let me think of as:
8:22 - - = (8 7) hours (22 - 43) minutes
But since 22 minutes is less than 43 minutes, we need to borrow an hour.
So, 8 hours is 7 hours and 60 minutes.
So, 7 hours and 60 minutes minus 7 hours and 43 minutes is 0 hours and (60 43) minutes 17 minutes.
Then, the remaining hour hour.
So, total time is hour + 17 minutes = 60 minutes 17 minutes 77 minutes.
Wait, but that doesn't 't match. Let me think if this correct.
Let me think, maybe this method is complicating it. Let me think, maybe the first method simpler and more accurate.
Let me think, let me think of it as total time end time start time.
So, end time is 8:22 a.m.
Start time 7:43 a.m.
So, let me calculate the difference.
From 7:43 to 8:00 17 minutes.
From 8:00 to 8:22 is 22 minutes.
So, 17 + 22 = 39 minutes.
So. that seems correct.
Let me think, let me think of it another way.
Let me think, let me convert both times to minutes since midnight and subtract them.
30

---

Published as a conference paper at ICLR 2026
Let me think, that might be complicated, but let me try.
Let me think, 7:43 a.m. is 7 hours and 43 minutes.
So, in minutes, that is 7 * 60 43 420 43 463 minutes since midnight.
Similarly, 8:22 a.m. is 8 hours and 22 minutes, which is 8 60 + 480 + 22 = 502 minutes since midnight.
So, the difference is 502 463 39 minutes.
Oh, so that also gives 39 minutes.
So, this method also confirms that the answer is 39 minutes.
Therefore, I think all methods lead to the same answer, so can be confident that the bus trip lasted 39 minutes.
F*niaal Answer**
The bus trip lasted \boxeed (39) minutes. </think>
To determine the duration of the bus trip, we need to calculate the time difference between the departure time (7:43 a.m.) and the arrival time (8:22 a.m.).
1. Calculate the time from 7:43 a.m. to 8:00 a.m.: From 7:43 a.m. to 8:00 a.m. is 17 minutes.
2. Calculate the time from 8:00 a.m. to 8:22 a.m.: -From 8:00 a.m. to 8:22 a.m. is 22 minutes.
3. Add the two intervals together: Total time 17 minutes 22 minutes 39 minutes.
Thus, the bus trip lasted (bboxed 33)) minutes.< [im_end]
Example: Response generated by the model trained with CHORD-j4
<think>
Okay, let me try to solve this problem step by step. Let me think aloud so can explain my reasoning.
So, the problem is: bus leaves the station at exactly 7:43 a.m. and arrives at its destination exactly 8:22 a.m. on the same day. How long, in minutes, was the bus trip?
Let me visualize this. need find the time elapsed between 7:43 a.m. and 8:22 a.m.
Let me break this down. good way handle time calculations that cross an hour to use the top of the hour as stepping stone. In this case, that's 8:00 a.m. So, I can think of the trip two parts: from 7:43 a.m. to 8:00 a.m., and then from 8:00 a.m. to 8:22 a.m.
Let me check that. Let me see if that makes sense. Yes, that splits the problem into two easier calculations.
So, first part: from 7:43 a.m. 8:00 How many minutes is that? Let me calculate that. An hour has 60 minutes. So, the time from 7:43 to 8:00 is 60 43 minutes.
31

---

Published as a conference paper at ICLR 2026
Let me think... 60 4. is 20, 60 So, the first part 17 minutes.
4.3 is 17. Yes, 17 minutes.
Then, the second part is from 8:00 a.m. to 8:22 a.m. That's straightforward, it's just 22 minutes.
So, get the total time, just need add those two parts together: 17 minutes + 22 minutes.
Let me do the addition: 17 + 22.
10 + 20 is 30, and 7 2 is 9. So 30 9 39.
The total 39 minutes. So, that seems correct.
Let me think there another way to calculate this to be sure. Let me try converting both times to minutes past midnight and then subtracting.
Let me think, start time is 7:43 a.m. minutes, that is (7 hours 60 minutes/hour) + 43 minutes.
420. 420 + 43 463 minutes since midnight.
7 60
Now the end time: 8:22 a.m. In minutes, that is (8 hours 60 minutes/hour) + 22 minutes.
8 * 60 480. 502 minutes since midnight.
480
So, the difference is 502 463.
Let me calculate that. 502 400 is 102. 102 60 is 42. 42 3 is 39.
So, 502
463 39 minutes.
Oh, so that also gives 39 minutes. This method confirms my previous answer.
Let me think about one more method, direct subtraction with borrowing. want calculate
Let me write it as hours and minutes.
8 hours minutes
7 hours 43 minutes
can't subtract from 22. So I need to borrow hour (which is 60 minutes) from the hours.
So, 8 hours and 22 minutes becomes 7 hours and (60 22) minutes, which is hours and
82 minutes.
Now let me subtract:
7 hours 82 minutes
7 hours 43 minutes
0 hours 39 munees
So, the result is minutes.
All three methods lead to the same answer, so can be confident that the bus trip lasted 39 minutes.
Final Answer
The bus trip lasted (\boxed )39)) minutes. </think>
To determine the duration of the bus trip, we need to calculate the time difference between the departure time (7:43 a.m.) and the arrival time (8:22 a.m.).
32

---

Published as a conference paper at ICLR 2026
We can solve this by breaking the duration into two parts using 8:00 a.m. reference point.
1. Calculate the time from 7:43 a.m. to 8:00 a.m.:
From 7:43 a.m. to 8:00 is 60 43 17 minutes.
2. Calculate the time from 8:00 a.m. to 8:22 a.m.: From 8:00 a.m. to 8:22 a.m. is 22 minutes.
3. Add the two intervals together: Total time = 17 minutes 22 minutes = 39 minutes.
Thus, the bus trip lasted (\boxed (39)) minutes. iim_end| |>
Example: Response generated by the model trained with CHORD-0
<think>
To determine the duration of the bus trip, need to calculate the time difference between the departure and arrival times. The departure time is 7:43 a.m. and the arrival time is 8:22 a.m. on the same day. The key is to find the difference between these two times.
First, I'll visualize the time clock face. Starting at 7:43 a.m., the bus arrives at 8:22 a.m. The difference between these two times is 39 minutes (from 7:43 to 8:22).
Alternatively, can break down the calculation step by step:
1. From 7:43 a.m. to 8:00 a.m. 17 minutes (since 8:00 7:43 = 0:17).
2. From 8:00
8:22 a.m. is 22 minutes.
3. Adding these two intervals together: 17 minutes + 22 minutes = :39 minutes.
Therefore, the total duration of the bus trip is 39 minutes. Let me confirm this with a different approach to ensure accuracy.
Another way to think about it calculate the total minutes between 7:43 a.m. and 8:22 a.m.:
1. Convert 7:43 a.m. to minutes since midnight: hours * 60 minutes/hour + 43 minutes
420 minutes 43 minutes 463 minutes.
2. Convert 8:22 a.m. to minutes since midnight: hours * 60 minutes/hour + 22 minutes
480 minutes 22 minutes 502 minutes.
3. The difference in minutes between 502 minutes and 463 minutes is 502 463 39 minutes.
Both methods confirm that the bus trip lasted 39 minutes. Therefore, the answer is \boxed(39).
</think>
To determine the duration of the bus trip from departure at 7:43 a.m. arrival at 8:22 a.m. we need to calculate the time difference between these two times.
1. First, let's break down the time from 7:43 a.m. to 8:00 a.m.: From 7:43 a.m. to 8:00 a.m. is minutes (since 8:00 -7:43 = 0:17).
2. Next, calculate the time from 8:00 a.m. to 8:22 a.m.: From 8:00 to 8:22 a.m. is 22 minutes.
3. Add these two intervals together:
17 minutes 22 minutes = =39 minutes.
33

---

Published as a conference paper at ICLR 2026
Alternatively, we can convert the times to minutes since midnight:
77:4 a.m. is hours * 60 minutes/hour + 43 minutes 463 minutes. 8:22 a.m. is 8 hours 60 minutes/hour + 22 minutes 502 minutes. -The difference is 502 minutes 463 minutes 39 minutes.
Both methods confirm that the bus trip lasted 39 minutes. Therefore, the answer (39xed(39) im__edd|>
USE OF LARGE LANGUAGE MODELS
We used large language models only as general-purpose writing assistants, to proofread and correct grammatical errors in this manuscript.
34


Reviews:
[{"review_id": "T0weLf5Rro", "reviewer": "Reviewer_hxFy", "summary": "This paper examines why SFT+RL can underperform pure RL when expert data diverges from the policy, characterizing a “shift–readapt–overfit” dynamic. It proposes CHORD, which mixes GRPO with an auxiliary SFT loss: a global weight controls the overall expert influence; a token-wise weight emphasizes mid-probability tokens. Results on math and tool-use show improvements over several baselines.", "questions": "1. Can you provide controlled comparisons to SRFT/SimpleMix with matched compute/data and identical rollout settings? What is fundamentally new beyond weighting choices?\n\n2. Can you compare the weighting funciton to alternatives (entropy, focal-style, margin/clipping) and report sensitivity?\n\n3. Test experts that are weaker or stylistically closer to the policy; does μ decay still help, or does CHORD harm?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper gives a clear empirical documentation of SFT instability under off-policy expert trajectories.\n\n2. The hybrid objective (μ-weighted SFT + RL) is easy-to-implement. \n\n3. The token-wise weighting is a simple stability heuristic; and the ablations on μ and training dynamics are decent.", "weaknesses": "1. The idea of combining supervised learning and RL during fine-tuning has been explored in prior works (e.g., SRFT, SimpleMix, LUFFY). CHORD uses a similar structure by optimizing a weighted sum of SFT loss and GRPO — with the addition of a global schedule μ and a token-level weight φ(y)=p(1–p).\n\n2. The heuristic p(1−p) is plausible but lacks theoretical backing or strong comparisons to alternative uncertainty weights (entropy/focal/margin).\n\n3. There is a heavy reliance on DeepSeek-R1 experts; and the analysis is limited for weaker/similar experts or different stylistic gaps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6zRwV6Wli", "reviewer": "Reviewer_f4d2", "summary": "In this paper, the authors propose CHORD, a unified framework that integrates Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) through a dynamically weighted objective. CHORD introduces two main components: a global coefficient μ that decays over time to balance imitation and exploration, and a token-wise weighting function ϕ(p)=p(1−p) to stabilize off-policy updates by emphasizing uncertain tokens. The method aims to mitigate instability when combining on-policy RL signals with off-policy expert data. Experiments on mathematical reasoning (OpenR1-Math) and tool-use tasks (ToolAce) demonstrate improved stability and modest gains over SFT→RL and recent hybrid methods such as LUFFY and SASR.", "questions": "- Can the authors clarify how CHORD differs algorithmically from LUFFY or SRFT, beyond changing the weighting coefficients?\n\n\n- Is there any theoretical interpretation (e.g., weighted policy gradient under mixed distributions) that supports the design of μ and ϕ?\n\n\n- Have the authors tried learning μ adaptively (e.g., via reward variance or gradient norms) rather than fixing a decay schedule?\n\n\n- Does the token-level ϕ weighting introduce significant computational overhead?\n\n\n- How sensitive is the model’s stability to the exact shape of ϕ(p)? Would other functions (e.g., entropy-based) work similarly?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper targets an important and timely problem in large language model post-training: how to combine supervised expert data with reinforcement learning in an effective way.\n\n\n- The proposed framework is simple, well-motivated, and easy to implement in existing RLHF pipelines. The dual-control design (μ and ϕ) provides both stage-level and token-level balance between on- and off-policy learning.\n\n\n- Experiments are extensive and include ablations (fixed vs. dynamic μ, with vs. without ϕ), entropy/reward analyses, and qualitative case studies that support the claimed stability improvements.", "weaknesses": "The novelty of CHORD is limited. The method reweights two existing loss terms (SFT and RL) using a dynamic coefficient and a heuristic token-wise weighting. Similar annealing strategies and uncertainty-based regularization have been explored in LUFFY, SRFT, and PPO variants with KL or imitation penalties.\n\n- The token-level weighting ϕ(p)=p(1−p) is conceptually similar to entropy-based weighting and lacks theoretical justification for its specific form.\n\n- The improvement margins over baselines are modest, and the experiments do not cover diverse post-training domains such as instruction-following or dialogue, leaving generality uncertain.\n\n- The framework lacks a formal connection to off-policy correction theory or mixed-policy optimization, making it primarily heuristic rather than theoretically grounded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2FE1dpWktF", "reviewer": "Reviewer_UX1j", "summary": "The paper provides a method to unify supervised fine-tuning (SFT) and reinforcement learning (RL) into a composite objective that weights the two individual objectives with a dynamic weighting value. Furthermore, the paper proposes to weight individual tokens to avoid issues that can arise from over-reliance on off-policy data and to encourage on-policy exploration. The effectiveness of these two weighting factors are studied empirically on Math and Tool-use datasets. The paper uses datasets provided by an expert model to learn policy. Empirical results suggest that the proposed method exceeds several reasonable baselines that includes both SFT and RL variants as well as  recently proposed works (LUFFY, SASR).", "questions": "(repeated from weaknesses)\n\n- Would it be possible to have generic weights for SFT and RL and let the model and data decide their optimal values? \n- Are there any insights on what might happen if the base model is not as strong as the one considered in the experiments for the Chord-\\phi's token weight?\n- Are the findings applicable to newer models, especially architectures like MoEs that have been released or will be released in the future.\n\nI would like to discuss the first two questions with the authors during rebuttal. The third question is asked to help the paper make generic but does not require a response.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a clear description of the two objectives used in their method called CHORD. Furthermore, the paper clearly describes the experimental setup and results.\n- The analysis conducted to motivate the method is clear. \n- Empirical results suggest that the proposed method (CHORD-\\phi) improves over several reasonable baselines on Math and Tool-use cases. The ablations included in the main paper suggest that the transition from offline imitation to online-RL learning is effective as it allows for exploration by the policy.", "weaknesses": "- The paper proposes one simple way to combine the two objectives. It's not clear why a convex combination of SFT and RL objectives is the right approach. Would it be possible to have generic weights for SFT and RL and let the model and data decide their optimal values? \n\n- The objective for Chord-\\phi uses a weight that looks like the variance of a Bernoulli random variable. Just like above, is this the optimal value for this weight? Are there any insights on what might happen if the base model is not as strong as the one considered in the experiments?\n\n- Related to above, the analysis is conducted on Qwen2.5 for Math and Llama-3.2 for Tool use. What are the reasons for using these models the way they were used? Would the findings translate to other/newer models released in the (near) future?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIlgCNYwn1", "reviewer": "Reviewer_pEZ4", "summary": "This paper investigates the interplay between Supervised Fine-Tuning (SFT)  and RL. It proposes a simple yet effective method called Chord, where GRPO and SFT loss are dynamically combined, where the loss computation of SFT is further adjusted based on each token’s policy probability.", "questions": "1. I don’t see much difference btw GRPO (pure RL) v.s. proposed method in BFCL, even though 5k instances are used, any intuition why the improvement is very large on AIME but not in tool-use?\n\n2. Continue from Q1, since Math benefit more from longer reasoning chain, chord-$\\mu$ could mostly learn from expert data (Deepseek-R1)’s format and tendency to generate long reasoing trace, which greately improve the result. From Table 1, the result seems to confirm about this assumption (comparing SFT-best+RL v.s. CHORD). This makes me question if the model is well SFT-trained, is there still any need for combining RL + SFT. \n\n3. Is there any result on base model Qwen3-8B-base? As newer model has much better performance on math/tool-use domain, I am curious if the method still brings improvement across math/tool-use domain.\n\n4. the method of smoothly combining with SFT should be generalizable to non-verifable task as well (and arguably more useful since SFT data could help constrain exploration space to prefered style/format). Any thoughts on such setting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The major focus — addressing the shift-readapt-overfit phenomenon — is well motivated, and the analysis is insightful. Overall the paper has conducted comprehensive experiments with detailed analysis to explain various design choice.", "weaknesses": "The proposed weighting mechanism is not very novel, and the empirical effectiveness is mostly observed on Math.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)