Trust-Region Adaptive Policy Optimization
Authors: Mingyu Su
Jian Guan
Yuxian Gu
Minlie Huang
Hongning Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reasoning Model
Reinforcement Learning
Trust Region
Knowledge Distillation
Primary area: reinforcement learning

Abstract: Post-training methods, especially Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), play an important role in improving large language models' (LLMs) complex reasoning abilities. However, the dominant two-stage pipeline (SFT then RL) suffers from a key inconsistency: SFT enforces rigid imitation that suppresses exploration and induces forgetting, limiting RL's potential for improvements. 
We address this inefficiency with TRAPO  (**T**rust-**R**egion **A**daptive **P**olicy **O**ptimization), a hybrid framework that interleaves SFT and RL within each training instance by optimizing SFT loss on expert prefixes and RL loss on the model's own completions, unifying external supervision and self-exploration. To stabilize training, we introduce Trust-Region SFT (TrSFT), which minimizes forward KL divergence inside a trust region but attenuates optimization outside, effectively shifting toward reverse KL and yielding stable, mode-seeking updates favorable for RL. An adaptive prefix-selection mechanism further allocates expert guidance based on measured utility. Experiments on five mathematical reasoning benchmarks show that TRAPO consistently surpasses standard SFT, RL, and SFT-then-RL pipelines, as well as recent state-of-the-art approaches, establishing a strong new paradigm for reasoning-enhanced LLMs.

Published as a conference paper at ICLR 2026
TRUST-REGION ADAPTIVE POLICY OPTIMIZATION
Mingyu Su', Jian Guan', Yuxian Gu', Minlie Huang', Hongning Wang "Conversational AI (CoAT) Group, Tsinghua University Ant Group sumy250ma 1 t tsinghua edu.cr, cr, aw-ai@tsinghuul edu. cn
ABSTRACT
Post-training methods, especially Supervised Fine-Tuning (SFT) and Reinforce- ment Learning (RL). play important role in improving large language models' (LLMs) complex reasoning abilities. However, the dominant two-stage pipeline (SFT then RL) suffers from a key inconsistency: SFT enforces rigid imitation that suppresses exploration and induces forgetting, limiting RL's potential for im- provements. We address this inefficiency with TRAPO (Trust-Region Adaptive Policy Optimization), hybrid framework that interleaves SFT and RI. within each training instance by optimizing SFT loss on expert prefixes and RL loss on the model's own completions, unifying external supervision and self-exploration. To stabilize training, we introduce Trust-Region SFT (TrSFT), which minimizes forward KL divergence inside a trust region but attenuates optimization outside, effectively shifting toward reverse KL and yielding stable, mode-seeking updates favorable for RL. An adaptive prefix- seefix-eelection mechanism further allocates expert guidance based on measured utility. Experiments on five mathematical reason- ing benchmarks show that TRAPO consistently surpasses standard SFT, RI., and SFT-then-RL pipelines, as well as recent state-of-the-ue approaches, establishing strong new paradigm for reasoning-cnhanee LLMs. Our code and data are publicly available at htthttps:///// /ttps:///ittuur com/ Su my/ TRRAPO
INTRODUCTION
Driven largely by post-training, especially Supervised Fine-Tuning (SFT) and Reinforcement Learn- ing (RL) techniques, large language models (LLMs) have demonstrated substantial advances in complex reasoning tasks, such as mathematical deduction (Lghhmman et al. (023), program synthe- sis (Chen et al., 2021), and multi-step decision-making (Shridhar al. 2021). shifting them from surface-level bots toward deep reasoners (OpenAl 2025) et al. 2025). Particularly, current dominant LLM post-training pipeline integrates SFT and RL in a two-stage fashion, where SFT pre- cedes RL. This design counts on the SFT stage to teach an LLM imitating carefully curated expert demonstrations, and then sharpens the model's ' reasoning skills via anl-ann-error in the RL stage.
However, fundamental barrier exists that hinders the synergy between SFT and RL. On the one hand, SFT tends to lock the trained models into imitative and rigid behavior modes (Chu al.) 2025] Chen et al. 2025), which hinders effective exploration crucially needed in the RL stage. On the other hand, SFT also prone to cause catastrophic forgetting in the trained models, impeding the RL. stage from exploiting the pretraining knowledge for improvements. These inconsistencies raise an important challenge: how can we effectively incorporate the knowledge-distiiee into RL training without undermining the model's 's exploratory capacity and pretrained knowledge?
benefits of SFT
In this work, we pinpoint the root cause of the aforementioned barrier to be the disjoint two-stage SFT and RL training, and propose hybrid post-training framework as our solution. Specifically, standard SFT training requires "sufficient" reduction of its training loss, but this is not sufficiently an indicative metric for later RL training- In other words, lower SFT loss does not suggest better starting point for RL training. Worse still, over-training in SET might push the model out of the region suited for RL, while there no immediate signal measuring that. Our idea then to interleave SFT with RL in each training instance: only perform SFT on the prefix of the given expert
"Corresponding Author

---

Published as a conference paper at ICLR 2026
Profix
25% Prefik
50% Proffx
100% Prefix
Cumulative Expert Return
SFT
Yall Tap "ock
Ratio
TrSFT
Fae Are Youd
RL loss the self generated rollouts
25%
maxty' On (o
Grodlen Weggte
Prompt
RL loss en the target model's continuation
solume of space abtained byy
Trust Region
rotuting the region bounded y= lnx, y= [0.e*)
TYSFT loss on 50% expert prefts
III loss on the target model's bossinwanton
0.29
100%
Expert Model
0
expert
Target Model
Tokin Probabiity
Figure 1: An overview of our TRAPO framework, which synergistically combines two key com- ponents: Trust-Region SFT (TrSFT) and Adaptive Expert Guidance. Left: By clipping the gradient weight with trust-region parameter a, TiSFT prevents exploding gradients on low-probability kens, ensuring stable learning signal when combining with RL. Right: The adaptive guidance mechanism implements a "learn-while-pres siaile-prattciigg" loop. When the target model fails rollout, its cumulative return dynamically dictates the length of an expert prefix provided for guidance. The model then continues generation and the full trajectory is optimized using both the TiSFT loss on expert prefixes and standard RI objective on the model rollout.
trajectory, and continue RL training by completing the trajectory thereafter. In this way, can best integrate the utility of expert demonstrations and self-exploration at the finest grain.
Though intuitive, several technical challenges prevent straightforward combining scheme. First, using the language of RL., the negative log-likelihood (NLL) objective in SFT aims to minimize the forward Beilbackk-eibble (KL) divergence between the target and expert policies (i.e., behavior cloning (Torabi et al. 2018)). which exhibits a strong mode-covering property by assigning relatively high probabilities even regions where the expert policy has no support (Minka et al. 22055 Malinin & Gales] 2019). This is especially toxic when one interleaves SFT with RL in a per-instance basis, as those inlatedd modes immediately cause degenerations in the target policy (e.g.. repetition or erroneous decoding) and keeps RI. away from effective exploration. In this work, to mitigate the negative impact of SFT on RI., we propose a new form of SFT, named Trust-Region SFT (TrSFT), which minimizes forward KI. divergence inside trust region, while outside the region it down- weights the optimization strength and effectively shifts the objective toward reverse KL. As reverse KL minimization is characterized by mode-seeking behaviors (Gu et al. 2023). TrSFT avoids blindly balancing the modes in the entire space, and only seeks the most prominent mode of the expert policy when the expert policy is clearly distinct from the current target policy. This provides stable and robust policy for the following RL training.
Second, determining the optimal amount of expert guidance for each training instance is nontrivial. cone-size-fits-all prefix length is inherently inefficient: it stifles valuable exploration on problems the model could solve independently, while offering insufficient support on challenging problems where more guidance is crucial. To address this, we introduce an adaptive prefix-selection mecha- nism that dynamically allocates expert guidance based on its utility. The core idea scaffolding approach: the model first attempts a problem without any guidance. If it struggles, indicated by a low return, we incrementally provide longer expert prefixes in subsequent rollouts for that same problem. This strategy ensures that expert guidance is used judiciously only as much as needed, thereby creating a dynamic balance between self-exploration and expert guidance.
Building on these designs, we propose TRAPO (Trust-Region Adaptive Policy Optimization), which unifies SFT and RL in a principled way. Extensive experiments on five mathematical rea- soning benchmarks demonstrate the substantial benefits of our approach. TRAPO delivers signifi- cant gains over conventional post-tran training algorithms. Specifically, it outperforms standalone SFT and pure RL by +6.3 and +6.2 points, respectively. More critically, TRAPO surpasses the highly competitive SFT-then-B baseline by +2.3 points, proving the superiority of our integrated frame- work over the disjoint two-stage approach. Crucially, we show that a naive, direct combination of
2

---

Published as a conference paper at ICLR 2026
SFT and RL objectives is counterproductive, causing catastrophic performance collapse of over 18 points compared to the pure RL baseline. This result validates our initial hypothesis and highlights the necessity of our TrSFT objective for achieving stable and synergistic integration. Furthermore, our pass@k k analysis reveals that TRAPO, unlike pure RI methods, effectively expands the model's underlying solution space, leading stronger test-time scaling properties. These compelling results establish TRAPO as a new, effective paradigm for developing -oooinggennhaeed LLMs.
METHODOLOGY
Our proposed framework, TRAPO, guides RL by leveraging prefixes from offline expert trajectories (Figure 2) These prefixes serve dual purpose: they act in-context demonstrations to guide ex- ploration and as direct supervision signals for skill internalization. The successful implementation of resolving two key challenges: (1) Guidance internalization: how to effectively learn from the prefixes, and (2) Guidance selection: how to choose the optimal prefix length for each prompt. We will now provide overview of the framework before detailing each component.
TRAPO hinges
2.1 OVERVIEW OF TRAPO
0.355
3350
The design of TRAPO is to realize a synergistic "learn- while- practicing" paradigm for LLM post-training, which couples learning from offline expert trajectories with on- line RL. updates. The core workflow unfolds follows: for each prompt, (1) a selected prefix from an offline ex- pert trajectory is provided get policy rolls out from there to complete the reasoning; and (3) dual update then performed, where the gen- erated completion is used for standard RL updates, while the expert prefix is used for direct policy optimization to internalize the expert's ss reasoning skills. TRAPO is ex- pected to guide the target policy to find high return com- pletions via the selected expert prefixes (Liu et al. 2025a Huang et al., 2025) and pruning unproductive solution space via skill internalization from the expert. As shown in Figure 2 we empirically validate the benefit of introducing prefixes from expert trajec- tories in target policy rollouts. On the MATH-500 benchmark, we provided Qwen2.5-3BB 5-3-.I----tttt with prefixes from DeepSeek-R1, and then calculated the response accuracy and counted the fre- quency of two key reasoning behaviors, i.e., backtracking and backward chaining (Gandhi et al., 2025), in the completed suffixes. Detailed experimental procedures are provided in Appendix C. It is clearly observed that longer expert prefixes steadily improve accuracy and stimulate the emer- gence of advanced reasoning behaviors.
0.73
Accuracy
0.102
1090
159
starting context; (2) the tar-
0425
259
0.0
0.2
0.4
0.6
Prefix Ratio
Figure 2: Accuracy and characteristic of Awen2.5-3B-Inntreee reasoning MATH-500 with different amount of to- kens from DeepSeek-RI as prefxxe.
However, integrating RL with expert demonstration presents two significant challenges: (1) Effec- tive learning objective for guidance internalization. The aim of incorporating expert prefixes is to not only immediately guide the target policy to find high return trajectories, but also enable the target policy to distill the expert's 's core problem-solving skills, highlighting the need for an effec- tive joint learning algorithm. (2) Efficient and adaptive guidance selection. When using expert prefixes to guide exploration, a "one-size-fits-ae
strategy, which applies a uniform prefix selection and fixed number of rollouts to all prompts (Liu et al. [025a Huang et al. 22255), can hardly be optimal. This simple method would over-guide in easy prompts, thus stifling valuable exploration, while under-guiding in difficult ones, leading to failed rollouts. It remains challenging to design an adaptive, prompt specific mechanism that allocates only the minimal guidance necessary to ensure the target policy can effectively discover the correct solution paths.
2.2 TRUST-REGION SFT
Regarding the first challenge of internalizing an expert's 's reasoning skills, a straightforward approach is combine the standard SFT loss expert prefixes with RL objective model generated roll- outs. However, this naive combination leads to severe performance degradation in our experiments To uncover the root cause, we first conduct pilot study of SFT's training dynamics.
model-gemerated
(see 3.3
3

---

Published as a conference paper at ICLR 2026
SPT T1STT
SFT
PSFT
0.35
SFT
Tagion
Brbobiity in Vove Reglon
0.30
0.25
0.20
0.10
12.6
250
Phase
Phose 2
0.00
Phesl]
Prusee
Phase]
Phesel
Phase
Phase 3
40-
SFT
T-SFT
Expert
Expert Fallcy
<L Divergence
Voe Nogan
hoyaan
20-
20-
0
-20 050
LE5
15.0
200
(a) Training dynamics
(b) Training results
Figure 3: An illustrative experiment showing the training dynamics during SFT. Panel (a) shows snapshots of learnt target policy at four consecutive training phases, corresponding to training steps of 0, 50, 100, and 1000, respectively. Panel (b) presents the KL divergence curve along with the change in cumulative probability of the target policy within the void regions.
The findings motivate our proposed Trust-Region SFT (TrSFT), an objective designed to resolve this instability, whose theoretical properties we then analyze.
Training Dynamics of SFT We begin by examining the standard SFT training objective. Given an expert policy pe and target policy pl parameterized by 0, SFT forces pt mimic PE by minimizing the NLL in every trajectory = (11,... .y.) from the expert policy in given prompt set X,
LSFT (0) = Em-x [T [ log of
The SFT objective is equivalent to minimizing the cumulative token-level forward KL divergence (see Appendix A. for derivation). To understand its training dynamics, we conduct an illustrative experiment traming two-mode Gaussian Mixture Model (GMM) mimic three-mode expert GMM. As shown in Figure [3] this process reveals the distribution-blendin phenomenon (Minka) al. 2005 Malinin & Gales 2019): the target policy assigns probability to void regions unsup- ported by either policy (e.g.. shaded regions in Figure 3 (a)). This effect is detrimental to RL, as these regions lead degenerated outputs (c.g.. repetitions) that hinder effective exploration.
We point out that the underlying cause of this phenomenon lies in the gradient of standard SFT loss:
In'l
VeLSET
(Uii vn Vepi vk
The term Pf [x' [=' v22)
Eq (2) can be considered the weight of training token vi, given context
(x*, y2k). When y4, is drawn from a mode expert policy which is far away from the current pol- icy's mode, e.g., the rightmost mode of expert policy in Figure 3 this weight term will be extremely large. Consequently, the resulting V LSFT is more detrimental than helpful for of update, though points to the correct the direction of policy improvement: the inflated gradient tends to first push pf into the voided regions during the online gradient update of SFT, as shown in Figure
While sufficient training can eventually correct for petributtion-lenddinn in standard SET (e.g., phase 4 in Figure B this issue is critical in TRAPO. Here, the interleaving of SFT and RL means that any probability mass allocated to void regions will immediately cause degenerated rollouts. This can significantly hinder model leamnnn, making modifications to the standard SFT objective essential.
4

---

Published as a conference paper at ICLR 2026
Clipping the gradient of standard SFT loss To mitigate the negative impact of distribution- blending during online model update, we propose crust-Region SFT (TrSFT), mechanism de- signed to adaptively leverage the SFT updates. The core idea is to establish region where the gradient from standard SFT loss can be trusted, while intervention is needed outside this region to prevent model collapsing into undesirable parts of the solution space:
1
Ve
Veri ykn)
max (p9 vkn))
where a E [0, 1] is a hyperparameter defining the boundary of trust region. The optimization objec- tive exhibits several desirable properties:
(1) Safe knowledge instillation within the trust region. Standard SFT risks distribution blend- ing by applying large updates towards "distant" modes of the expert policy. TrSFT mitigates this by defining a dynamic trust region based on the target policy's own beliefs (i.e., whether (f., x= v 2 a). Within this region, it employs the standard SFT objective to aggressively mimic expert policy's 's behavior. Outside the region, the constant weight significantly dampens the gradients, thereby reducing the disruptive impact of large gradient updates on the target policy's immediate behaviors.
The benefit of TrSFT can be easily understood by our illustrative experiment in Figure As illus- trated in Phases and 3, TrSFT prioritizes learning from high-overlapping regions-monn modes jointly favored by PE and pt. This helps the target policy first instills expert's wisdom that is "close" and readily tangible, while preserving its existing strengths. This conservative, trust-region-basee approach is safer and more stable than aggressive, dust-iisttinttion matching, especially when the target policy's 's capacity is limited.
(2) Leading reasonable optimization endpoint. We present the optimization problem defined by the gradient defined in Eq. (3). and theoretically derive the solution.
Proposition 1. Let S()) = c pE(c) > ad, e E C} and C is the vocabulary. There exists a unique E (0, 1) such that X = ECES(A) PE(C). And for this X, the optimal solution of the optimization problem defined by the gradient defined in Eq. (3) is given by
PE(C)
PE > ad.
p+(c)
X
0,
otherwise
The optimal solution for TrSFT counters distribution-blending by pruning low-probability regions in expert policy (c) = 0) and rescaling primary modes (pt(c) = PE(c)/A). This dual action effectively transforms the objective from mode-covering of Forward KL mode-seeking akin to reverse KL, forcing the policy to focus on the expert's 's core skills and thereby facilitating high-return
rollouts for RL. Appendix A.. shows more details.
2.3 MICRO-GROUP SAMPLING
To elegantly address the second challenge of guidance selection, we propose micro-group sampling, which adaptively allocates guidance from expert prefixes based on the observed returns from the current policy rollouts, thereby both minimizing unnecessary reliance on expert prefixes and accom- modating the heterogeneity of prompt difficulty within each training batch.
As illustrated in Figure T in each training prompt, TRAPO creates N micro-groups in order, where each micro-group 9: (for = 1, ...., N) is specified by three key apper-parameters: the prefix length ratio L,, the return threshold 6, and the sampling budget n,, For micro-group g.. TRAPO first computes the average return from all samples generated in the preceding micro-groups, If the average return is smaller than the threshold t. TRAPO provides the current target policy with a prefix whose length is set by the ratio L, of the complete expert trajectory, and then samples n, completions from the target policy. Otherwise, no expert prefix is provided, and n, policy rollouts obtained directly from the target policy.
5

---

Published as a conference paper at ICLR 2026
We set 0 always starts from guidance-free self-exploration RL, and LN = allows the target policy, when necessary, to access the complete reasoning path from the expert. As result, increasing level of I, ensures richer guidance provided only when shorter prefixes prove insufficient. For clarity, Appendix (opendix|Rderaani details the complete training procedure specified by TRAPO in Algorithm
L2 < LN 1. 0 is to ensure that in each training prompt, TRAPO
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Training Details Our primary training dataset OpenRI MppnRI-Math-460-8811 Yan et al., 2025), )
which consists of large collection of verified reasoning trajectories generated by DeepSeek-R1 for complex mathematical problems. To enhance the diversity of guidance, we additionally pair each problem with another trajectory sampled from OpenR 1-Math-200000 (Facel 2025). Following recent work Yan et al. 2025 Huang et al. 2025 Fu et al. 2025), we use 5wath..--Mat--7B (Yang et al. 2024b) the base model. To further validate the generality of our approach, additionally evaluate it on Qwen2.5-7B -B--truutttt (Yang et al. 2024a). a -cnerra--prppoee model.
Implementation Details We adopt the Group Relative Policy Optimization (GRPO) (Shao al.
2024 Liu et al. (025c) algorithm without the KL penalty (Hu et al. (025) for RL. The training configured with batch size of 128 and constant learning rate of 5 x 10-6 Our adaptive guidance mechanism operates on a total group size following prior work, which is partition into four micro-groups of sizes {4, 1, 1}. These micro-groups correspond to relative expert prefix- length proportions of (L1, La) = 0.2 0.5, 1.0) and are activated by reward thresholds of (ti....,tt) t4) 0.5, 0.7, 0.9), respectively. The threshold t -1 ensures the first micro micuo--rouu is always without guidance. For our TrSFT objective, the trust-region parameter in Eq. (3) is set 0.1. More details can be found in Appendix C.2
Evaluation benchmark and metries We focus on mathematical reasoning tasks, while also eval- uating various methods on both mathematical and ceneral-domate benchmarks. Specifically, the mathematical benchmarks include AIME2024 (Li et al., 2024), AMC (He et al.) (024). Min- erva Tewkowyez et al. 2022), OlympiadBench (He al. 2024), and MATH-500 (Hendrycks) et al. 2021). Given the relatively small number of test samples in AIME2024 and AMC, we re- port @2 on these benchmarks, while employing pass@ for the remaining three. For general- domain reasoning benchmarks, we report pass@ ARC-c (Clark et al.) 2018) and MMLU- Pro (Wang et al., 2024) to examine whether the improvements in reasoning ability generalize to other reasoning tasks.
Baselines
We consider two categories of baseline methods:
Pure RL without External Expert Guidance. This category includes GRPO (Shao et al. 2024),
PRIME Zero (Cui et al.) 2025). SimpleRL-Zero (Zeng et al. 2025). OpenReasoner- Zero (Hu
et al. 2025), and Oat-Zero (Liu et al. 2025c).
RL Incorporating External Expert Guidance. This category includes (1) SFT: directly train- ing the model to imitate expert trajectories, (2) SFT-then-RL. following the standard two-stage pipeline where SFT precedes RL, (3) LUFFY (Yan et al. 2025): augmenting each group of eight rollouts with one expert trajectory perform offline RL, and (4) ReLIFT (Ma et al. (025): alter- nating between SFT and RI. training across different batches, thereby dynamically switching the optimization objective.
3.2 MAIN RESULTS
Mathematical reasoning performance As shown in Table TRAPO achieves an average score of 56.6 across five mathematical reasoning benchmarks, outperforming all baselines. In particular, TRAPO yields improvements of +6.3 and +6.2 over SFT and GRPO, respectively, and +2.3 gain over the SFT-then-RL RL baseline. These results validate core hypothesis: TRAPO effectively enables the model to both internalize expert skills and leverage guidance for superior exploration, leading to more robust acquisition of reasoning abilities.

---

Published as a conference paper at ICLR 2026
Table 1: Main experiment results mathematical and general reasoning benchmarks based Qwen2.5-Math-7B. Bold and underline indicate the best and second-best results, respectively. means the results are taken from the corresponding paper.
Mathematical Reasoning
General Domain Reasoning
Model
AIME2024
AMC MATH-500 Minerva Olympiad Avg.
ARC-e MMLU-Pro
Avg.
11.9
33.7
47.0
12.5
21.9
25.4
34.5
23.3
28.9
Qwen2.5-Math- -nns-------sttu
11.3
48.2
82.6
39.7
43.7
72.4
55.4
Pure RI without External Expert Guidance
GRPO
24.0
59.0
84.0
39.3
45.8
50.4
80.5
47.2
63.9
PRIME-Zero'
81.4
39.0
40.3
46.3
53.0
SimpleRL-Zero"
27.0
54.9
76.0
25.0
34.7
43.5
30.2
34.5
32.4
OpenRensoner-Zeroo
16.5
52.1
82.4
33.1
47.1
46.2
66.2
58.7
62.5
Oat-Zero"
33.4
61.2
78.0
34.6
43.4
50.1
70.1
41.7
55.9
RI. Incorporating External Expert Guidance
SFT
27.7
56.0
84.8
38.2
44.7
50.3
51.5
33.1
42.3
SFT-then-RL
33,5
62.3
86.6
41.2
47.6
54.3
52.9
36.1
44.5
ReLIFT
87.4
33.8
52.5
64.4
LUFFY
29.4
65.5
88.4
38.2
56.0
55.5
80.5
53.0
66.7
Our Method
TRAPO
28.3
66.2
89.2
41.5
57.6
56.6
83.7
$2.8
68.3
2220
6300
5.5
Resporse Length
0.6
Estcopy
2000
Figure 4: Training dynamics of TRAPO compared with GRPO. From left to right: average reward, generation length, and output entropy during training. For fair comparison, both reward and gener- ation length are computed by excluding trajectories guided by expert prefixes.
Gemeral-domain reasoning performance On the general reasoning benchmarks, TRAPO attains an average score of 68.3, surpassing all baselines. By contrast, SFT and SFT-then-RL ex- hibit notably lower scores these benchmarks, indicating that TRAPO, while leveraging external guidance, does not confine the model to rigid reasoning patterns; instead, it yields stronger general- ization.
Training dynamics Figure 4 presents a comparative analysis of the training dynamics between TRAPO and GRPO, revealing three key advantages of our approach: (1) TRAPO consistently achieves higher rewards throughout the training process and finally converges to significantly higher final reward level. (2) The generation length curve highlights that TRAPO rapidly increases its output length in the early stages, indicating swift internalization the expert's extended rea- soning patterns. In contrast, GRPO struggles to produce longer solutions, consistently maintaining

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study TRAPO components.
Model
AIME2024
AMC
MATH-500
Minerva
Olympiad
Avg.
GRPO
24.0
59.0
84.0
39.3
45.8
50.4
Micro-group sampling
26.0
63.7
84.2
39.3
50.1
52.7
Micro-grav grooo sampling SFT Loss
14.6
32.8
60.2
25.4
28.5
32.3
Micro-grove group sampling LUFFY Loss
26.2
65.1
87.2
36.8
52.5
53.6
Micro-group sampling TiSFT Loss
28.3
66.2
89.2
41.5
57.6
56.6
a short output length. (3) While both methods show an initial drop in policy entropy, their long- term behavior differs. TRAPO stabilizes at relatively higher entropy level. We attribute this to its ability to maintain dynamic balance: it simultaneously refines its own high-probability reasoning paths while remaining open to learning from externaly provided, potentially cowprobbbiility expert guidance.
Extending to general-purpose LLMs. To further validate the generality TRAPO, we conduct additional experiments Owen2.5-7B-Instueee
Qwen2.5-78-Instruee
47.5
general-purpose instruction-tune tunee model. As shown in Figure 5, TRAPO still achieves higher average accuracy than all other baselines across five mathe- matical reasoning benchmarks.
45.0
40.60
40.0
3.3 ABLATION STUDY
35.0
32.
To assess the contributions of micro-group sampling and TrSFT, conduct ablation study on Qwen2.5-Math-7B (Table Results show that micro-group sampling alone, even without explicit prefix learning, surpasses GRPO by adaptively determining prefix length to enhance reasoning and reward density. Adding TrSFT further improves performance: unlike standard SFT loss (which degrades) or the offline RL loss from LUFFY (which brings limited gains), TISFT effectively internalizes expert prefixes. We also analyze the effect of the trust-region param- eter a in Appendix
32.5
Base
SFT
GAPO
TRAPO
Figure 5: Average accuracy five mathematical benchmarks.
3.4 TEST-TIME SCALING
0.7
We evaluate p the success rate over indepen- dent rollouts, better estimate the upper bound of model capability (Snell et al.] 2024). as recent studies show that multiple generation attempts reveal reasoning poten- tial more accurately than few rollouts (Yue et al. 2025a Wang et al. 2022).
GRPO
Figure illustrates the pass@k performance on the AIME2024 benchmark, from which derive key
0.2
insights: (1) We observe that the base model (Qwen2.5- Math-7B) surpasses the GRPO-trained model when eval- uated with a sufficiently large k. This aligns with prior findings Yuue al. 2025a) and suggests that standard RI primarily stimulates the model to select better solutions from its existing knowledge space, but does not funda- mentally expand that space with new problem-solving skills. (2) Both TRAPO and the SFT-based methods demonstrate strong performance scaling with larger k. indicating they possess a richer un- derlying solution space. The superior performance of TRAPO highlights its success in effectively internalizing the external knowledge from expert trajectories, thereby expanding the model's intrin- sic capabilities.
0.15
2
2'
k(Number Generations)
Figure 6: Comparison pass@k re- sults the AIME2024 dataset
8

---

Published as a conference paper at ICLR 2026
4 RELATED WORKS
RL for LLM reasoning RL has recently driven major advances in LLM reasoning, as demon- strated by milestone systems such as OpenAI-ol (Jaech et al.) 2024). DeepSeek-Ri (Guo et al.] 2025), and Kimi-1.5 (Team et al. 2025). Follow-up studies have examined RL from three com- plementary perspectives. (1) Empirical analyses investigate its effect on reasoning. e.g.. Yue et al. (202a)) shows that RL refines existing solution trajectories rather than expanding reasoning capabil- ities. (2) Data-centric approaches reshape training signals, such as R3 (Xi al. (024)) with reverse curriculum learning, and ADARFT (Shi et al. 2025 or Logic-RI (Xie et al. (025) with dynamic difficulty scheduling to improve sample efficiency. (3) Optimization methods reñne RL objectives, from PPO's clipped trust region and GRPO's ' group-based advantage estimation to recent variants like DAPO (Yu et al.) 2025), Dr.GRPO (Liu et al.) 2025c), and VAPO (Yue et al. 2025b). Distinct from these directions, TRAPO enhances RL efficiency by adaptively incorporating expert demon- strations, which both guide exploration and act as direct supervision, simultaneously improving reward density and reasoning capability.
Combining SFT with RL While the traditional training paradigm primarily followed serial approach first applying SFT and then RL, recent works have shifted focus towards concurrently performing both SFT and RL. A straightforward approach is to directly combine the SFT and RL losses, adjusting their respective weights. For instance, SRFT (Fu et al.. (2225) proposes a single- stage training method that dynamically adjusts the weights of SFT and RL losses based token entropy. AMFT (He et al. 2025) views the balance between SFT and RL as a learnable parameter and introduces meta-gradient adaptive weight controller. HPT (Lv al. 2025), on the other hand, dynamically decides whether to apply SFT or RI. based on the model's 's rollout performance, with binary weights of 0 or 1. Another approach involves interleaving SFT within the RL pipeline, as seen in ReLIFT (Ma et al. 2025), which collects poorly performing samples during RL training and stores them in butfer to later apply SFT. Additionally, LUFFY (aan al., 2023), inspired by offline RL, treats one expert trajectory as offline data and mixes it with the remaining seven online trajectories in a group. using an importance ratio calibrate the distribution shift, achieving state-of-th the performance. Similar to our work, Prefix-RFT (Huang et al., (0255) samples prefixes as guidance and uses entropy to select expert tokens for SFT. These works collectively emphasize the significant benefits of combining these two post-trainne post-training paradigms. We present the first study to theoretically investigate the challenge of combining SFT and RL objectives, and propose a simple but effective solution to resolve the distribution--lenddinn effect and foster synergy with the RL objective.
CONCLUSION
In this work, propose TRAPO, a one-stage training paradigm that unifies SFT and RL under pert prefix guidance. TRAPO enables both the internalization of guidance and its use in generating high-quality continuations. To stabilize training, introduce TISFT, which constrains policy up- dates within trust region, mitigating the disruptive effect of low-probability tokens and effectively shifting Forward KL's mode-covering behavior toward Reverse KL's mode-secking behavior. TiSFT integrates seamlessly into RI. to enhance guidance utilization. We further design micro-group sam- pling, which adaptively adjusts guidance length based on return improvements, balancing explo- ration with expert supervision. Experiments on mathematical reasoning tasks show that TRAPO significantly outperforms standalone SFT, RL, and the conventional SFT-then-BR pipeline.
6 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experimental results, we have open-sourced our training and evaluation code, including detailed training scripts and configurations, at: httpss: //qithub..
com/ Su-myuummnntt / The specific training and evaluation settings are described in $3.1 and Ap-
pendix C.2

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported by the National Science Foundation of China (under Key Project No. 92570203), Beijing Natural Science Foundation (under project No. Z250001) and the National Key Research and Development Program of China (under project No. 2024YFC3606800).
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The twelfth international conference on learning representations, 2024.
Hardy Chen, Haoqin Tu, Fali Wang. Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training ri-like reasoning large vision-language models. arXiv preprint ar arXiv:2504. 11468, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL httpt:ttts/aaaaaee
org/ abs 02111007773774
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le. Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1 hos/1803.05457. 2018. URL http:/t:://axx/iv..o http:/awwiv..ogg/booo abs 803 05457
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.014566 2025.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL wti.thub.comm//ggiiinn.....
https:
Yugian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506. 19767, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cogni- tive behaviors that enable self self-immun ioprovigo reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.0 01 3/707, 2025.
Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, and Mark Crowley. Kkt conditions, first-order and second-order optimization, and distributed optimization: tutorial and survey. arXiv preprint 1rX0v:2110.01858, 2021.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. arXiv preprint arXiv:2306.08543, 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 20..2.5511.99... 2025.
10

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Associa- fion for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11828-3850 Association for Computational Linguistics, 2024. URL https: /ww.//oii..o. www.oi..ogg....o 10. 1...0...8660 18653/11613vv 20t.//2222.....oo..... .211
Lixuan He, Jie Feng, and Yong Li. Amft: Aligning Ilm reasoners by meta-learning the optimal imitation-exploration balance. arXiv preprint arXiv:2508 0504:25508009944,, 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and
Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL
benchmarks ottpapoc..a...pppoeeeeee.eoieana
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405. 1143,, 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zerro An open source approach scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.242000 2025.
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16721 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 111-626,111 2023.
Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quan- titative reasoning problems with language models. In Advances in Neural Informa- fion Processing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL htttt:::/paaprs..pss..oo
html
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang, 7iju Shen, al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. https: //huggingface. ct/ datatatsssesss.// Numi Namah) 2024. Hugging Face repository, 13:9.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning, arXiv preprint arXiv:2505. 16984, 2025a.
Zichen Liu, Changyu Chen, Chao Du Wee Sun Lee, and Min Lin. Oat: A research- research-friendly frame- work for Il online alignment, 2025b.
11

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078: 2025c.
Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, et al. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509. 04419, 2025.
Lu Ma, Hao Liang, Meiyi Qiang. Lexiang Tang. Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, et al. Learning what reinforcement learning can't: In- terleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.0755 2025.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved un- certainty and adversarial robustness. Advances in neural information processing systems, 32, 2019.
Tom Minka al. Divergence measures and message passing. 2005.
OpenAL. OpenAl o3 and 04-mini system card. http::/wwpuuu..cmmmnmmcdddddddds 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning. arXiv preprint arXiv:2504.0m 055200, 2025.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL http::tt
Hausknecht. (ALFW forld: ford:
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 20Xi::2408.03314 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 4950-4957 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou coff-consitencc improves chain of thought reasoning in language models. 111221003.17,,,,1111111111112111 2022.
arXiv preprint
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more ro- bust and challenging multi-task language understanding benchmark. Neural Information Processing Systems 38: Annual Conference Neural Informa- tion Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: nips ctppppp files/paper/2n/ http://22ecccaeeeeeeeeeeeeeaaaaa
Advances in
Datasets and
Benchmarks. Track.html
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.05880 2024.
12

---

Published as a conference paper at ICLR 2026
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXtv:2502.14768. 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under oof-policy guidance. arXiv preprini arXiv:2504. 14945, 2025.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 251.::24/2.1511555 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via oof--improvement. arXiv preprint arXiv:2409. 220.::2299..1222,, 2024b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 14X1v 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025a.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.0511 18, 2025b.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv:2503.18880 2025.
13

---

Published as a conference paper at ICLR 2026
A DERIVATIONS AND PROOFS
A.1 DERIVATION OF THE EQUIVALENCE BETWEEN SET AND KL
Following Agarwal al. (2024), remove the length normalization factor and define the discrep- ancy between token-level distribution of PE and pr over given sequence y:
(p (y)z)
Σ D ((PE(-lVen I-M
IN æ)).
(4)
The problem then becomes proving the following equivalence:
min LSFT (0) min En-x (Eympeclae) [DRI. (pellp)) [x]
(5)
start by substituting Eq. (4 into the objective expression on the right-hand side of Eq. (5):
(y-rr(((a)) [DKL [KL (pe (pell)) E By-re(-1e))
DEL (pE(-l wym x)]]
l
PE(Um Men x))
E (x
PE PE(UnlVen x) log
(7)
(yr We x
== BVEV
We then remove the term that are independent of 0:
y
PE(Um Ben, x) log PE(Um Wenn Pe(Un|Nen,,)) x)
min Exnx
E
(1)))
pl( M
(8)
Mi, EV
+ min Exrx
E
PE log
(9)
0
Ev
y
< min Ex-x
18
pp.. log bog
(10)
M
< min Ex~x 0
E B-ppp(-aa))
- log pf (f (VnllVenn æ]]]
(11)
+ min E
E (y))
log (z) [
(12)
The last line follows from the equivalence between the logarithm of product and the of loga- rithms.
A.2 PROOF OF PROPOSITION
Proposition Let S()) = PE(c) > ax,cE and C the vocabulary. There exists a unique E (0, 1) such that X = Eees(N) PE(C) And for this X the optimal solution of the optimization
problem defined by the gradient defined Eq. ① given by
Pr(C)
Pf at.
(c) =
0,
otherwise
Proof. First, we present the equivalent form of this optimization problle,,ionrroblem,,, with the derivation similar that Eq. 5pp
min I (0) + min Ex-x (yy-e((la)) [iiSTT (pm.l114) P
(2)) [y)=
0
0
P +1 log at p 0,
where
Di = (pe(c) and (p) =
<EC
log P.
p 2 a,
14

---

Published as a conference paper at ICLR 2026
Then apply the KuckeTucckr (KKT) (Ghojogh al. 2021 conditions.
We introduce multiplier À E R for the equality constraint and multipliers {Mele E C} for nonnega- tivity constraints. The Lagrangian is
[(pr(e),.. .Pree)) A, Me Hex) Σ (p(())) Eepr(()) + +N( Epr(c) 1).
(13)
For each e, stationarity gives Pe(c) pr(o) ( (pr(c)) Me + 0, together with He 2 and Meff = 0. The derivative of ea is
0 a,
mr l" (pT)
Pr > a.
Case A (pr(c) > a). In this case, V (pr(c)) (/)r(c). The stationarity condition with pe 0 gives
PE(C)
PE(C) + - pr(c)
Pr(c)
Since we assumed pr(c) > this requires PE(C) > ad.
Case B (0 < pr(c) sa).
Here (pr((c)) = -1/a. With Ma 0, the condition becomes
Pe (c)
+ - 0
pF(c) = ad.
Thus interior solutions (0, af are only possible pF(c) ad.
Case C (pr(c) 0).
At the boundary pr(c) O, we use (0*(0+)= -1/a. Stationarity then gives
PE(() Mo
PE(C)
Me
a
For feasibility we require Pe 2 hence this case occurs only when pE(c) at.
Collecting the above conditions, the optimal solution must satisfy:
PE(c)
pe(c) ad,
(i(c)
0.
pE < ad,
any value in [0, if pe ax.
In realistic token distributions the set of tokens satisfying PE(c) aA has measure zero and can be ignored. Equivalently, one may simply enforce (T(c) for all such tokens. The optimality condi- tions derived above remain valid. Then from (T(((c = we can derive X = =cES(s) pr(c).
Observe that the normalization condition requires
(F(C) =1
Σ CES(A)
As À increases continuously from to 1, the left-hand side decreases monotonically from +oo to the sum of probablity less than 1, while the right-hand side remains constant at 1. Therefore, by the intermediate value theorem, such a value of must exist.
15

---

Published as a conference paper at ICLR 2026
Algorithm TRAPO Micro-group Sampling with Adaptive Guidance
Require: Dataset Dasta {a, y*)}; expert policy Psi target policy pf: trust-region parameter o;
number of micro-groups N; metadata {[ns, L,, k with 0 = Ln < La < .... Ln
Ensure: Updated student parameters
1: function PASSRATE(S)
2: return []](] L [Correct(y)) y*)]
D compare to ground truth y
3: end function
4: for each mini-batch B Daata do
5: for each prompt a E do
6: S 0
collected samples for this far serial micro-groups
7: for i = to N do
8:
pr; <- PASSRATE(S)
samples from previous 1, ..... micro-groups
9:
if pr; S t then
inject expert guidance
y PE(- x))
11:
y t first L yll tokens of
for k 1 to n, do
13:
ÿ (. n. y <مم
14:
S t SU (x, <he @ g)}
end for
16:
else
b no guidance
for k 1 to n, do
18:
i (. x)
S S ((x
20:
end for
21:
end
22:
end for
23:
end for
24:
Compute VoLisr on guided tokens only (those fed the target) via Eq. 3
25:
Compute V LGRPO on self-generated trajectories
26:
VaL VoLLIET VaLoRFO
27:
θ + OptimizerStep(O, VoL)
end for
B PSEUDOCODE FOR TRAPO
Algorithm T summarizes the complete training procedure of TRAPO.
C EXPERIMENTAL DATAILS
C.I PRELIMINARY EXPERIMENT
Data Collection We use DeepSeek-RI generate eight solutions for each problem in the MATH- 500 benchmark. For each solution, extract prefixes with varying token proportions guidance, which are then concatenated to the original prompt. At each token proportion level, we obtain 4,000 prompts, which are subsequently fed into Qwen2.5-33 --nttruuttttt
Instruct for continuation.
Evaluation
Accuracy is computed by comparing the generated answers with the ground truth using Math- Verify
We further input the generated continuations into GPT-4o-mint to count the occurrences of distinct reasoning behaviors. The specific prompt templates used are as follows:
16

---

Published as a conference paper at ICLR 2026
Backtracking
You given math reasoning problem and the latter half of reasoning trace generated by Qwen2.5-3B-Instruce
Problem: srroblem_tett)
Model output: {output.text]
Detect BACKTRACKING: moments where the model abandons line of attack and starts a new, distinct approach (not merely the next algebraic step of the same plan). Count an instance when the text signals restart/switch such as:
"That doesn't it work leads nowhere contradiction, so try..." "Instead, I'll..,, "An- other approach:", "Restart with...", "Consider a different method/case".
Dropping the current construction and trying fresh one (new substitution, inequal- ity, identity, factorization,
split that resets the plan).
Do NOT count routine sequential steps in one coherent plan backtracking, Return only the number between <count> and </ count>> none, return <count>0</count>
Backward Chaining
You are given math reasoning problem and the latter half of reasoning trace generated by --B-----ttttt
Problem: (problem_tex))
Model output: {output.text]
reasoning that starts from (or is framed by) the desired conclusion/target and derives necessary conditions that would make it true. Count an in- stance when the text says things like:
Detect WARD-WWAIAHNNNEE
"We want show P, suffices show Q", "Work backwards from the target...", "If the result were true, then must hold", "To get X, we need Y", rearranging "from the goal form to requirements*.
Proof-by-contraadicton setup qualifies if it explicitly frames the goal via assuming its negation to derive impossibility.
Do NOT count ordinary forward algebra unless is clearly framed as working from the goal backward. Return only the number between <count> and < </count>. If none, return <count>0</count>
C.2 MAIN EXPERIMENT
In this section, we provide training details not mentioned in the main text.
Base Model Our main experiments are conducted using Qwen2.5-Math-7B. model designed for mathematical reasoning tasks. Due to the limitation of 8k tokens in the DeepSeek-RI expert trajectories and the 4096 context length of Qwen2.5-Math increase the model's 's rope theta from 10000 to 40000.
we follow LUFFY's approach and
Dataset Our data source is OpenR 1-M--hh-461--8192 Yan et al. 2025 which filters out incorrect trajectories and those with token lengths greater than 8192 from OpenR 1-Math-2001, (Face) 2025). Since OpenRI-Math
contains at least two trajectories generated by Deepseck-RI for each problem, we also collect the corresponding second trajectory for each entry OpenR1-Math-46kk 8192, resulting in approximately 46k prompts and 92k trajectories. Furthermore, since we did not require the model to split the output into think and answer parts the system prompt, we retain only the tokens between the <think> and </think> tags from the original Deepseek-RI output as candidate trajectories. During micro-group sampling, we first calculate the corresponding token position based the desired ratio, then locate the first reasoning step delimiter (typically a double newline in the Deepseck-FF RI style) after this position to ensure the atomicity and completeness of the reasoning steps.
17

---

Published as a conference paper at ICLR 2026
SFT For all SFT methods, train the 92k <prompt, trajectory> pairs for 2 full epochs as described above. The hyperparameters are set as follows: sequence length of 16,384, learning rate of 5e-5 with warmup ratio of 0.1, and batch size of 32. The remaining experimental settings consistent with those used in the OpenRLHI (Hu et al. 2024) framework.
--eRl--7Qee (Face 2025). All SFT training is conducted using
RL All RL methods use a variant of GRPO, Dr.GRPO (Liu et al. 2025c). which removes length normalization and the standard deviation regularization in advantage calculation. The hyperparam- eters are as follows: batch size of 128, 8 rollouts per group. max prompt length of 1024. max response length of 8192, and a constant learning rate of 1e-6. We use vL..M (Kwon et al. (023) for rollouts during training with sampling temperature set to 1.0. RL training is conducted using the verl (Sheng et al. 2025 framework.
TRAPO
Since the dataset contains only two expert trajectories per prompt, and provide expert guidance for up to 4 out of rollouts in each group, we randomly select one expert trajectory as a potential guidance source for each micro-group
ensure diversity in the expert data. Except for setting the trust region parameter a to 0.1, all other training parameters are consistent with those used RL and are implemented within the verl framework.
Evaluation For evaluation, we use lower temperature of 0.6 and maximum generation length of 8, 192 tokens. We We employ Math-verify and OAT-Grader (Liu et al. 2025b) frameworks for evalu- ation.
Prompt Template Following SimpleRL (Zeng et al. 2025), we use the simplest prompt template across all methods and models to ensure that the improvement in reasoning capability comes solely from the training. The template is as follows:
Prompt Template
im start >system
You are helpful assistant. im_end
im start >user
{problem}
Please reason step by step, and put your final answer within \boxed
iim_end
D HYPERPARAMETER STUDY
D. ABLATION OF TRUST-REGION PARAMETER
We apply equations and to single token, which leads to the following form:
VoL
V Vori vCn).
(14)
(y /., Ven) or max (pf 2 ven),
The weight (the fractional factor) in front of the gradient term varies with ( lx, V<n) as shown in Figure aa). When a 0. TrSFT is equivalent SFT, meaning that all tokens lie within the trust region. As a increases, the gradient update weight of tokens outside the trust region is reduced to a fixed value of 1/a, significantly mitigating the disruptive impact of large gradient updates from these tokens on the model parameters. However, when becomes too large, all tokens are assigned the same gradient weight, completely losing the characteristic of SFT where tokens with lower probability require more learning. Therefore, choosing appropriate value of is crucial to balance the resistance against probbabbyyyyity tokens and the need to effectively learn from them. Figure 7) shows the training performance of TRAPO under different values of (0. 0.05, 0.1. 0.2, 0.5, 1). We observe that the best performance is achieved when a = 0.1, with both higher and lower values of failing reach the optimal performance.
18

---

Published as a conference paper at ICLR 2026
20.0
SFT TTSFT
17.5
Trust Region
55
15.0
Gradient Weight
50
12.5
Avg Acc
10.0
7.5
5.0
2.5
35
0.0
0.6
0.8
1.0
0
0.05
0.1
0.2
0.5
1.0
Token Probabliity
a
(a) Gradient weight of SFT and TYSFT
(b) Average accuracy with different
Figure 7: (a) The plot showing the change in gradient weight with respect token probability in SFT and TrSFT. (b) The average accuracy on five mathematical reasoning benchmarks for Qwen2.5-
Math-7B trained with TRAPO for different values of
D.2 ABLATION OF MICRO-GROUP HYPERPARAMETERS
In micro-group sampling, we adopt an adaptive strategy for determining expert prefix length. Unlike fixed-prefix approaches, this dynamic mechanism selects an appropriate amount of expert guidance prompt, providing stronger assistance when unguided rollouts perform poorly.
Each micro-group 9i contains several tunable hyperparameters: (1) the prefix-length proportion Li (2) the reward threshold f; and (3) the number and sizes of micro-groups All variants were trained on Qwen2.5-Math-77 for 100 steps, with average accuracy reported across six mathematical reasoning benchmarks.
1. Prefix-length proportion.
Table 3: Ablation of the prefix-length proportion.
Variant
L
L2
L3
La
Avg. Perf.
A
0.2
0.5
1.0
39.4
B
0.2
0.5
0.7
37.6
C
0.1
0.2
0.5
1.0
34.1
D
0.1
0.8
1.0
39.2
E
0
0.3
0.4
1.0
39.5
As shown in Table 3 compared with Variant A, Variants B and C confirm the importance of our two design principles: (I) the first micro-groot -roup must receive no expert prefix to preserve unguided explo- ration, and (2) the final micro-group should expose the full expert trajectory. When modifying only the intermediate prefix proportions (Variants and E), performance remains largely unchanged, suggesting that TRAPO is robust to these internal ratios.
2. Reward threshold.
We maintain a monotonically increasing threshold schedule to match the naturally increasing ex- pected reward of longer prefixes. As shown in Table constant thresholds (B) decreasing sched- ules (C) reduce performance. As long as thresholds increase in a reasonable range (D and E), their exact values have minimal effect.
19

---

Published as a conference paper at ICLR 2026
Table 4: Ablation of the reward threshold.
Variant
t2
tz
f
Avg. Perf.
A
-1
0.5
0.7
0.9
39.4
B
-1
0.5
0.5
0.5
38.1
C
-1
0.9
0.7
0.5
37.4
D
-1
0.55
0.75
0.95
38.9
E
-1
0.45
0.65
0.85
39.3
3. Number and sizes of micro-groups.
Table 5: Ablation of the number and sizes of micro-groups.
Variant
(71, 122...)
Avg. Perf.
(4, 2, 1, 1)
39.4
PB
4, 1, 1)
37.2
(4, 4)
38.0
(4, 3, 1)
39.1
For the organization micro-groups, two structural principles are crucial: (I) the first unguided micro-group must be sufficiently large to support exploration, and (2) at least intermediate group should provide partial guidance to bridge unguided and fully guided rollouts. As shown in Table shrinking the unguided group (B) removing intermediate groups (C) harms performance. Configurations satisfying both principles (A and D) achieve similar accuracy.
E ADDITIONAL COST AND EFFICIENCY ANALYSIS
This section provides further details on the computational efficiency of TRAPO, supplementing the discussion in the main paper. Due to the longer generation lengths and the sequential execution required by micro-group sampling, TRAPO incurs a higher per-step computational cost compared to standard GRPO. However, under matched wall-clock training-time budgets, TRAPO consistently achieves stronger reasoning performance, even though it executes slightly fewer training steps than the baselines. This suggests that TRAPO benefits from more effective training-time scaling.
Overall GPU-hour usage. Table reports the total GPU hours consumed by each method. For the SFT-then-RL pipeline, this measurement includes both the SFT stage and the subsequent RL stage. We also report the average number of expert- preffx tokens (Expert Tokens) and target-model- generated tokens (Target Tokens) per RL trajectory.
Training dynamics under matched wall-clock budgets. To assess training-time efficiency, we measure the number of steps completed by each method under equal GPU-hour budgets, together with the mean reward of the collected rollout batches. As reward is strongly correlated with down- stream reasoning performance, serves as a useful indicator learning progress. For SFT-then-RLE the GPU-hour measurement below begins at the start of the RL stage.
Summary. Across all matched training-time conditions, TRAPO consistently achieves higher re- ward, which is indicator of stronger reasoning ability than GRPO and LUFFY, even though performs fewer training steps. This highlights TRAPO's advantageous sample efficiency and the effectiveness of combining SET and RL through trust-region formulation. Moreover, compared to the SFT-then-RL. pipeline, TRAPO attains better performance while requiring less total GPU hours overall.
20

---

Published as a conference paper at ICLR 2026
Table 6: GPU hours and token statistics across training methods.
Model
GPU hours
Expert Tokens
Target Tokens
GRPO
68 8
0
1189
SFT-then-RL
(42 + 105) x 8
4760
LUFFY
81 8
519
2439
TRAPO
89 8
501
2545
Table 7: Reward progression under matched GPU-hour budgets.
GPU hours
GRPO
SFT-then-RL
LUFFY
TRAPO
10 8
0.521
0.650
0.472
0.601
20 8
0.579
0.712
0.594
0.684
30 8
0.626
0.726
0.665
0.744
40 8
0.621
0.710
0.697
0.722
50 8
0.652
0.732
0.675
0.768
F PREFIX USAGE DYNAMICS DURING TRAINING
To better understand how the adaptive prefix seleit-sllection mechanism behaves during training, we track the number of trajectories receiving expert prefix guidance each training step. The results are shown in Figure Two key phenomena emerge:
Unguided trajectories consistently dominate across the entire training process. This ensures that the target model maintains strong self-exploration capability and remains aligned with the evaluation setting, which uses expert prefixes at test time.
The number of expert-guided trajectories decreases steadily training progresses. Early in training, model frequently falls below the reward thresholds and therefore ben- efits from partial full expert-prefix guidance. As performance improves, fewer trajec- tories trigger guidance, reflecting that the model becomes increasingly capable of solving problems without external intervention.
This pattern confirms that TRAPO's adaptive mechanism realizes the intended behavior: it provides expert prefixes only when helpful, and its reliance on expert data naturally diminishes as the tar- get model improves. This demonstrates that TRAPO leverages smarter guidance rather than guidance, and avoids over-conditioning on expert trajectories.
G
CASE STUDY
To intuitively demonstrate the differences in style and problem-solving ability of models trained with different methods, select polynomial factorization problem from the MATH-500 bench- mark as a case study. As shown in Figure 9 the model trained with GRPO exhibits a consistent forward-reasonin style in problem solving, also retaining the base model's tendency employ code verification at the final step. However, due to limited self-reflection and self-correction abil- ities, prone to simple computational and logical mistakes. contrast, the model trained with SFT demonstrates rigid imitation of DeepSeek-R1. often engaging in excessive reasoning that ex- ceeds the maximum sequence length budget. The model trained with TRAPO not only incorporates the verification and reflective reasoning patterns observed in expert trajectories, but also preserves the base model's 's ability to provide concise and clear planning before each reasoning step (e.g., out- lining "Step 1 Step 2 Step 3 ...."! "). This demonstrates that TRAPO effectively balances the use of expert data with the exploration of its own solution space, combining the strengths of both to maximize training efficiency and reasoning performance.
21

---

Published as a conference paper at ICLR 2026
guidance guldance
800
Number of Trajectories
600
400
200
100
100
300
400
Step
Figure 8: Dynamics of prefix usage across training steps. We plot the number of trajectories that receive expert-prefix preffx guidance versus those that remain unguided.
H THE USE LARGE LANGUAGE MODELS (LLMs)
We ensure that ideas and scientific contributions originate solely from the human authors, with no involvement of LLMSs in research ideation. In addition, LLMSs were not used to retrieve discover related works. Their usage was strictly limited to language polishing and to the automated counting of reasoning behaviors reported in Appendix [. All generated content was carefully reviewed by the authors ensure faithfully reflects our original intent.
of LLMs in this work fully complies with the ICLR 2026 policy. All research
22

---

Published as a conference paper at ICLR 2026
Question
The polynomial x² - 3xz + 4x is a factor of x" pxe + qx² + Enter the ordered triple (p.4.r).
Answer: (6,31,-1)
GRPO
SFT
[Problem decomposition
[Recall knowledge]
problem, we need to determine coefficients p. and
Hmm, okay. Let me start by recalling that if polynomial ((A())) is factor of another polynomial (B(x))) then the roots (A(x))) are also roots of (B(x))....
that the polynomial x +
px* qx
divisible by xa 3x 4x This means that if f(x) x² 3x + 4x -1, then f(x)
divides x + px" + qx" + r. other words, the roots of (x) must also be roots of pz + qx
[Overthink]
But since Val), Mbly, ((c)) roots of (xx33 3x"2 + 4x
[Method choice/ plan]
1 = OV)..
Let's denote the roots x 3x
by
Alternatively, maybe we can use polynomial division... But polynomial division might be tedious here. Maybe there's a smarter way. Let me think...
Val), Vbl), and (ee)...
[Forward algebra (power reduction]]
We need to express ((^91), (b*9)), and (c*91) in
Wait, let's re-examine. Wait, the ((x66) terms come from:...
(x*71) term....
terms of ((*33)) (b*3)), and (c*31), and then terms
(at).. \(b)). and (cl).
But we don't know ((cl) yet. Let's continue...
[Constraint extraction
Wait, no. Wait, when multiplying (S(x)l) by 1(*3 3x*2 + 4x - 11), the constant term...
First. let's find ((a*6)): Now, let's find like terms, we get:..We can solve these equations step by step. First, solve the first two equations for (pl) and Val):
Wait, let me do each term.... Wait, perhaps
use equation equation 6 again? Wait, but already used them. Wait, perhaps we need another equation. Wait. but all equations have been used. Hmm. Wait, maybe we need to use the original polynomial equation...
[Computation error)
So the ordered triple ((P, q. rJ)) is
778
A99
19829 1418
[Verfiiatioon Let's verify this solution using Python ensure accuracy.
Wait, no. Wait, actually. \x*9 p x"6 q x +r =01), then substituting expression for (x*91) from above...
ppython...
TRAPO
[Clear Sufficient moderate contemplation
Sure, help you solve this problem. The goal to find the ordered triple V(p. rJ)) such that the polynomial
Vx*3 - 3x'2 4x 11) is a factor of ((x19 px*6 qx*3 ri).
#*# Step 1: Understand the Problem
Given that 1(x*3 3x*2 + 4x 11) is a factor of (xx19 + px16 qx"3 + rl). this implies that when we divide
### Step 2: Find Roots the Cubic Polynomial
First, let's the roots of the cubic polynomial (x*3 3x*2 4x O1). This might be bit tricky....
###l Step 3: Polynomial Division or Substitution
Since Vx*3 3x*2 4x 11) is factor of Vx*9 px*6 + qx*3 ri), we can polynomial division subatitute-
ion. However, polynomial division might be cumbersome for degree 9, Instead, let's use substitution...
##U Step 4: Substitute and Simplify
Let's compute Vy*3 py*2 qy rl) with Vy 3x*2 - 4x 11)...
### Step 5: Express Higher Powers in Terms of Lower Powers
We need to express (x*91) terms of ((x22)), Vxx)), and constants using the relation Vx*3 3x 11)..
*uu Step 6: Substilute into the Original Polynomial
Now, substitute these expressions into the original polynomial (xx99 + px16 + qx"3 ri):.. #** Step 7: Solve the System of Equations
First. let's solve the first two equations for (pt) and Val)... ### Step 8: Verify the Solution
solution correct, we can verify by substituting V(p 61), 311), Vfr = -1)) back into the original polynomial and checking 1xx3 3x"2 4x divides ####f Canclusion
ensure
The ordered triple V(p. q. r))) is 1(6, 31, -1)). Therefore, the answer is [boxed 31, ###ff Reflection
This problem required understanding polynomial division, substitution, and solving a system of equations.
The key was to express higher powers of ((x)) in terms lower powers using the given cubic
Figure 9: A case study the MATH-500 benchmark; polynomial factorization problem.
23


Reviews:
[{"review_id": "PHJ2GvmFi6", "reviewer": "Reviewer_KiUB", "summary": "The paper proposes TRAPO, a one-stage post-training framework that interleaves SFT and RL at the instance level. For each prompt, the method applies SFT on an expert prefix and RL on the model-generated suffix. The key ingredient is Trust-Region SFT (TrSFT), which replaces the standard per-token weight 1/p with 1/max(p, alpha) to prevent large updates on very low-probability tokens (reducing forward-KL “mode-covering” drift and behaving more mode-seeking outside the trust region). A second component, micro-group (adaptive prefix) sampling, increases the prefix length only when the current group’s pass rate is low, providing “just enough” guidance. On five math benchmarks with Qwen2.5-Math-7B, TRAPO reports higher average accuracy than SFT, GRPO, SFT-then-RL, LUFFY, and ReLIFT; it also shows gains on ARC-c and MMLU-Pro. Ablations indicate that (i) micro-group sampling helps even without explicit prefix learning, (ii) naively adding a standard SFT loss during RL can collapse performance, and (iii) TrSFT recovers and improves on top of micro-groups.\n\nDisclosure: I used assistive writing tools to draft this review; all evaluations and judgments are my own.", "questions": "Clarify alpha everywhere and correct any plots/tables if needed; report which alpha was used per experiment.\n\nCompute parity and statistics: publish a single evaluation script; report tokens/steps/rollouts/GPU-hours; add 5+ seeds and 95% CIs (paired bootstrap for Pass@k).\n\nMechanistic diagnostics: track per-token gradient weights 1/max(p, alpha) vs 1/p, token-level entropy, and expert-mode KL through training to demonstrate TrSFT’s intended effect in transformers.\n\nData and grader robustness: dedup train vs test; report a small human-audited error rate; test sensitivity to grader noise or an alternative grader.\n\nMicro-group sensitivity: ablate thresholds, ratios, and group sizes; show a histogram of prefix usage; include a fixed-prefix baseline to isolate the value of adaptivity.\n\nBroader baselines: add at least one of Prefix-RFT, AMFT, SRFT, HPT, or UFT under matched compute and training code.\n\nStronger generalization test: add Putnam-AXIOM (especially the functional variations) with matched compute, 95% CIs, and p-values. This suite is designed to probe true extrapolative generalization. If TRAPO shows robust gains there, it would materially strengthen the claim that the method expands the solution space rather than just re-ranking known patterns. I would raise my score if you include convincing Putnam-AXIOM results (see https://arxiv.org/abs/2508.08292).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Motivation is concrete. The paper clearly diagnoses why forward-KL SFT can hurt exploration when combined online with RL (distribution blending, degenerate rollouts) and proposes a minimal, targeted fix (clipping the per-token weight with alpha).\n\nSimple and practical. TrSFT is a one-line modification of the SFT loss; micro-group sampling is a lightweight, per-prompt procedure (ratios 0, 0.2, 0.5, 1.0; thresholds −1, 0.5, 0.7, 0.9; group sizes {4, 2, 1, 1}).\n\nConsistent improvements. On Qwen2.5-Math-7B, TRAPO’s average across AIME2024, AMC, MATH-500, Minerva, and Olympiad is 56.6 vs 55.5 (LUFFY), 54.3 (SFT-then-RL), and 50.4 (GRPO). It also leads on ARC-c/MMLU-Pro (avg 68.3).\n\nUseful ablations and training dynamics. The paper shows why naïvely summing SFT+RL fails, why adaptive prefixes help, and how TrSFT stabilizes learning. Reward/length/entropy curves and Pass@k scaling support the narrative that TRAPO preserves solution-space diversity and improves with test-time compute.", "weaknesses": "Alpha inconsistency; please clarify.\nThe main setup and alpha sweep indicate alpha = 0.1 works best. In Appendix C.2, however, TRAPO is described “except for setting the trust-region parameter alpha to 1,” which would nullify clipping and contradict the rest of the paper. Please reconcile and state the alpha used for every table/figure.\n\nTheory–practice gap and Proposition 1 details.\nThe toy GMM example and the TrSFT optimum (Proposition 1) are helpful intuitively, but the proof sketch appeals to behavior (e.g., step-function arguments) that could be made more rigorous. More importantly, there are no transformer-level diagnostics to show TrSFT’s intended effect in practice (e.g., distributions of per-token gradient weights, expert-mode coverage over time). Please add empirical diagnostics on real runs.\n\nCompute parity and disclosure.\nMicro-groups introduce guided rollouts; budgets may differ from baselines. Please disclose training tokens/steps, rollout counts, wall-clock/GPU hours, and decoding settings for every method, and either confirm compute parity or normalize the comparison.\n\nStatistical rigor.\nResults appear single-seed. Small-n suites like AIME/AMC are noisy. Please add multi-seed (≥5) results with 95% CIs (paired bootstrap for Pass@k) and significance tests for headline numbers; report standardized effect sizes where appropriate.\n\nData hygiene and verifier robustness.\nTraining uses OpenR1-Math-46k-8192 plus extra R1 trajectories; evaluation uses Math-Verify and OAT-Grader. Please report exact/near-duplicate removal between training and test problems, and provide at least a small human-audited calibration of grader errors. A robustness check with an alternative grader or injected noise would help, since tail-emphasizing supervision can amplify mislabels.\n\nMicro-group sensitivity and usage statistics.\nThe thresholds and ratios are reasonable, but the paper does not show sensitivity curves or a histogram of actual prefix usage by difficulty/time. This would distinguish “smarter guidance” from “more guidance,” and quantify how often the full-trajectory prefix is triggered.\n\nBaseline coverage and consistency.\nTRAPO beats LUFFY and ReLIFT, but closely related “single-stage SFT+RL” or “prefix-guided SFT” methods (e.g., Prefix-RFT, AMFT, SRFT, HPT, UFT) are not compared head-to-head under the same codebase and budgets. Given conceptual proximity, a direct comparison would clarify novelty/attribution. Also, unify the RL variant description (GRPO vs Dr.GRPO) and ensure all baselines use the same variant and decoding settings.\n\nLLM-usage disclosure inconsistency.\nAppendix C.1 uses an LLM to count reasoning behaviors (backtracking/backward-chaining), while Section F says LLMs were used only for language polishing. Please align these statements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PEvhpodAiR", "reviewer": "Reviewer_Hux5", "summary": "This paper proposes TRAPO, a hybrid framework that unifies external supervision and self-exploration through alternating optimization within each training instance. Concretely, it employs TrSFT, an SFT method that modulates optimization strength via a trust region to eliminate the \"mode-covering\" defect of standard SFT, and calculates RL loss on the model’s self-generated completions to preserve exploratory ability. Experiments on 5 mathematical reasoning and 2 general-domain reasoning benchmarks demonstrate that TRAPO outperforms standard SFT, pure RL, and SFT-then-RL pipelines.", "questions": "1. How does TRAPO perform on newer, harder benchmarks such as AIME25 and GPQA?\n\n2. How robust is TRAPO to low‑quality or noisy expert trajectories in SFT data? For example, with wrong or meaningless prefixes, does performance degrade gracefully, and what mitigation strategies work best, such as prefix filtering or confidence weighting?\n\n3. How dependent is TRAPO on the Expert Model? What happens if the Expert is weaker or partially misaligned? Specifically, if the Expert Model is identical to the Policy Model, how do performance and stability change?\n\n4. How should these hyperparameters be tuned, such as the number and sizing of micro-groups, the prefix ratios and return thresholds, and the TrSFT trust‑region alpha? Are there recommended defaults or automatic tuning strategies?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow. The method is clearly defined, and the figures are clear and helpful.\n\n2. The ablation results support the design. Micro-group sampling alone improves over GRPO, while replacing TrSFT with standard SFT causes a clear drop. This suggests the trust region brings stability, and the adaptive prefixes provide useful guidance.", "weaknesses": "1. This paper only validates TRAPO’s performance on established benchmarks, including 5 mathematical reasoning benchmarks and 2 general-domain reasoning benchmarks, while excluding newer, harder datasets like AIME25 and GPQA.\n\n2. The micro-group sampling of TRAPO requires sequential processing for each prompt’s micro-groups within a mini-batch. This sequential logic for micro-groups is far less parallelizable than baselines like pure GRPO. Additionally, the training time is not provided, so the tradeoff between TRAPO's performance and efficiency remains unproven.\n\n3. TRAPO relies on multiple hyperparameters that are difficult to tune, including micro-group-specific ones and the TrSFT trust-region parameter alpha. While Figure 7(b) shows accuracy variations of Qwen2.5-Math-7B on mathematical benchmarks with different alpha values, it does not demonstrate stable or optimal performance across parameter adjustment; instead, it reflects that alpha is sensitive to performance, let alone the combined tuning complexity of all hyperparameters.\n\n4. TRAPO feels overengineered: it combines staged micro‑group scheduling with per‑prompt SFT prefix adjustment, which together make the pipeline complex and difficult to control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UG0z0lzL2t", "reviewer": "Reviewer_Ux2e", "summary": "This paper proposes TRAPO, a one-stage post-training framework that interleaves SFT and RL at the instance level. For each prompt, the model receives an expert prefix (possibly empty), then rolls out its own completion; training applies TrSFT (a trust-region variant of SFT) on the expert tokens and an RL loss on the self-generated suffix. The authors argue TrSFT curbs the mode-covering tendency of forward-KL SFT by clipping per-token gradient weights outside a probability \"trust region,\" effectively shifting behavior toward mode-seeking (reverse-KL-like) updates that are friendlier to RL. They also introduce micro-group sampling, which adaptively increases the prefix length only when earlier unguided (or shorter-guided) rollouts underperform. Empirically, on five math-reasoning benchmarks, TRAPO outperforms SFT, pure RL, and the common SFT-then-RL pipeline. On average it improves over SFT and GRPO by and beats SFT-then-RL.", "questions": "1. If you add back a standard KL penalty to the RL objective, how do performance and stability change relative to TrSFT alone?\n2. How does TRAPO perform if the expert prefixes come from a weaker trajectory model, or from noisy/partial solutions? Does TrSFT still help, or does it overfit to poor guidance?\n3. Does the dynamic prefix selection strategy incur computation overhead? If so, by how much?\n4. Can you show ablations on code or multimodal reasoning, where the expert prefixes may differ stylistically from the target model's distribution?\n5. You randomly pick one expert trajectory per micro-group. Would prioritizing trajectories (e.g., by diversity or difficulty) further help?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The per-instance coupling, SFT on prefixes + RL on suffixes, neatly addresses the two-stage inconsistency the authors highlight. This method directly targets the lack of exploration space for SFT'ed policies, which is a concrete and important problem in RL today. The micro-group schedule is clear and practical.\n2. Clipping the token-level gradient weight with a threshold α seems to be an intuitive way to prevent outsized updates on low-probability expert tokens. Relevant hyperparameter study is inlcuded.\n3. The pseudocode for the adaptive sampling and the combined gradient update is helpful for me to understand the method in general.", "weaknesses": "1. One (or a type of) baseline I consider missing from the paper is existing work that directly improve sampling diversity during rollout in RL. Simple baseline might include increasing temperature, or other algorithms that encourage diverse samples. I recognize the the authors' argument that SFT may decrease sampling diversity, but there is also simple entropy control mechanisms like [1] (and possibly some others).\n2. Training relies on OpenR1-Math expert trajectories (DeepSeek-R1), and the core evaluations are all math-reasoning; stronger non-math results would better support claims of broader reasoning gains from TRAPO.\n3. The RL baseline is implemented with a GRPO variant and no KL penalty; the paper should clarify whether this favors TRAPO (which already regularizes via TrSFT).\n\n[1] He, Jujie, et al. \"Skywork open reasoner 1 technical report.\" arXiv preprint arXiv:2505.22312 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2HfhGsBaJ", "reviewer": "Reviewer_3vHn", "summary": "This paper introduces TRAPO (Trust-Region Adaptive Policy Optimization), a novel one-stage hybrid framework that unifies Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) for post-training large language models (LLMs). The authors argue that the conventional two-stage pipeline—first SFT, then RL—creates a mismatch: SFT enforces rigid imitation that suppresses exploration and induces catastrophic forgetting, thereby weakening RL’s learning potential.\n\nTo overcome this, TRAPO interleaves SFT and RL within each training instance. It performs:\n- SFT loss on expert prefixes, to absorb expert reasoning patterns.\n- RL loss on model rollouts, to maintain exploration and self-improvement.\n\nA key innovation is Trust-Region SFT (TrSFT), which minimizes forward KL divergence inside a trust region but attenuates optimization outside, effectively shifting toward reverse KL to encourage mode-seeking rather than mode-covering. This ensures stable and meaningful updates beneficial for RL. An adaptive prefix-selection mechanism dynamically adjusts expert prefix length according to model performance, balancing guidance with exploration.\n\nExperiments across five mathematical reasoning benchmarks (AIME2024, AMC, MATH-500, Minerva, OlympiadBench) show TRAPO outperforms SFT, RL, and the SFT-then-RL pipeline by +2–6 points, and achieves the highest average accuracy of 56.6%. TRAPO also generalizes well to non-mathematical tasks (ARC-c, MMLU-Pro) and exhibits superior test-time scaling properties.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Conceptual Integration\nThe paper’s central insight—recasting SFT and RL as interleaved processes at the instance level—is original and well-motivated. By linking expert prefix imitation with RL rollouts in a trust region, TRAPO presents a principled path toward more synergistic post-training.\n\n2. Theoretical Clarity\nThe authors diagnose a specific instability—distribution blending due to the mode-covering nature of forward KL—and formally derive how the clipped gradient in TrSFT converts the optimization dynamics toward reverse KL’s mode-seeking behavior. This analysis provides strong theoretical grounding rarely seen in empirical LLM post-training work.\n\n4. Comprehensive Evaluation\n\nThe paper offers convincing empirical validation:\n- Clear improvement over both pure RL baselines (GRPO, PRIME-Zero, etc.) and hybrid methods (LUFFY, ReLIFT).\n- Extensive ablation studies show the distinct contribution of micro-group sampling and TrSFT.", "weaknesses": "1. Limited Scope of Evaluation\n\nAll major experiments focus on mathematical reasoning benchmarks. While the generalization test on ARC-c and MMLU-Pro is included, more diverse domains (e.g., commonsense reasoning, coding, dialogue) would better validate TRAPO’s universality.\n\n2. Lack of Cost and Efficiency Analysis\n\nTRAPO’s training procedure—especially with multiple micro-groups and prefix sampling—appears more computationally expensive than standard RL. The paper does not quantify overhead (e.g., training time, token cost, GPU hours), which is important for practical adoption.\n\n3. Trust-Region Parameter Sensitivity\n\nAlthough the trust-region boundary parameter α is analyzed in the appendix, the main paper provides limited discussion on its tuning sensitivity and stability under different model scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)