Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Authors: Zishang Jiang
Jinyi Han
tingyun li
Xinyi Wang
Sihang Jiang
Zhaoqian Dai
Ma Shuguang
Fei Yu
Jiaqing Liang
Yanghua Xiao
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Group Relative Policy Optimization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to  provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.

Published as a conference paper at ICLR 2026
SELECTIVE EXPERT GUIDANCE FOR EFFECTIVE AND DIVERSE EXPLORATION IN REINFORCEMENT LEARN- ING OF LLMS
Zishang Jiang', Jinyi Han', Tingyun Li', Xinyi Wang', Sihang Jiang', Zhaoqian Dai*, Shuguang Ma', Fei Yu', Jiaqing Liang'; Yanghua Xiao*
School of Data Science, Fudan University
Shanghai Institute of Artificial Intelligence for Education, East China Normal University
*College of Computer Science and Artificial Intelligence, Fudan University
4 "Ant Group
(zajiang24, xinywang24 @m. fudan. cn, (sihangjiang, liangjiaqing, s fudan edu on (jinyihan099, 1itinyun0715, feiyu. fyyu) com, (daizhaogian.dkq, liangxiao. mwgmmaattgroup.ccommmmm Lianggiao..ssseantrrrou...pp com
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating ex- pert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance at critical deci- sion points rather than the entire reasoning path. Based on this insight, pro- pose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experi- ments show that MENTOR enables models capture the essence of expert strate- gies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). It has significantly improved models' performance in solving challenging mathematics and programming problems, as evidenced by models such as OpenAI-ol (Jaech al., 2024), DeepSeek-RI (Guo et al., 2025), and Kimi- 1.5 (Team et al., 2025). These improvements are largely attributed to the models' ability to generate detailed chains of thought (CoT) before giving final answers (Wei al., 2022), which is termed test-time scaling (Muennighoff et al.. 2025).
However, the effectiveness RLVR strongly depends on the capability of base models. It has been observed that when applied to models with limited parameters, RLVR fails to reproduce the remarkable gains observed on powerful base models (Guo et al., 2025).
This issue arises because RLVR requires the model to have sufficient capability to perform high- quality exploration, which involves both effectiveness and diversity. Specifically, when the task overly challenging for the model, it often struggle to discover any correct reasoning trajectory (Yue et al., 2025), resulting in ineffective exploration that hinders training (Yu al., 2025). Furthermore,
"Corresponding author. https//gghhna.cmmm//ngs1118///

---

Published as a conference paper at ICLR 2026
Update Policy
Ch-policy Advantage
On-policy Rellouts
On-policy Solutions
Reward
Problem
Policy
Model
p Cell Enplirttion
Expariiaa
cthisi find..-/hirre
The final asseer is Voowed(42)
find
Mixed-policy Solutions
<hhrloTo find. Weitt
Export-Policy Model
Mixed- palicdpolicy in (6)
0 0
The
&a (boxed]31)
Mixed-policy Rollouts
To verify, ./thrdo
is Mb
Maxed- Advantage
Figure 1: Illustration of MENTOR framework. By providing expert guidance only at critical deci- sion points, MENTOR steers reasoning trajectories while preserving the policy's s exploration, thereby avoiding the constraints of fixed expert trajectories and achieving more effective and diverse exploration in RL training.
when correct solutions found, limited diversity of reasoning trajectories often leads the model to rapidly converge to a narrow set of solutions (Song et al., 2025), which reflected in entropy collapse (Cui al., 2025) and ultimately traps it in suboptimal solutions (Song et al., 2025).
Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. While such imitation reduces ineffective exploration (Yan et al., 2025; Zhang et al., 2025a;b; Liu et al., 2025; Li et al., 2025), it forces the model to follow to fixed expert trajectories, thereby restricting the diversity of exploration and accelerating entropy collapse (Yan et al., 2025). In addition, the reduction of diversity is further accelerated by gradient imbalance (Huang et al., 2025), which drives the model to quickly overfit expert trajectories, espe- cially when expert reasoning patterns diverge substantially from those of the policy model (Zhang al., 2025a). Although some works attempt to mitigate by reweighting tokens in expert trajecto- ries (Yan et al., 2025; Zhang et al., 2025a), the relief remains superficial, as the exploration space is still fundamentally restricted by the fixed expert trajectories.
To achieve better exploration, we argue that the expert only needs to provide guidance at critical decision points rather than the entire reasoning trajectory. Expert guidance is indeed essential for steering the model toward correct solutions, but blindly imitating full expert trajectories restricts the exploration space. Since tokens contribute unequally to reasoning trajectories (Wang et al., 2025), introducing guidance at critical decision points enables the model to best leverage expert knowledge while preserving exploration diversity. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, framework that in- jects expert guidance only at critical decision points to perform effective and diverse exploration. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby sustaining high-quality exploration and achieving superior overall performance.
Our contributions can be summarized as follows:
We provide a formal analysis of RLVR and demonstrate that effective policy improvement critically depends on high-quality exploration, which requires not only discovering correct solutions but also maintaining sufficient diversity to prevent entropy collapse and avoid being trapped in suboptimal solutions.
We are the first to propose leveraging expert knowledge only at critical decision points in RLVR training rather than imitating entire expert trajectories, thereby enabling models to achieve both effective and diverse exploration in RLVR.
We conduct extensive experiments showing that MENTOR delivers consistent improve- ments on six challenging math benchmarks and out-of-domain tasks, with gains stable across diverse model families. Further analysis reveals that it mitigates entropy collapse in RLVR training and broadens the capability boundary of base models, and case studies demonstrate it can selectively absorb expert knowledge rather than superficial imitation.
2

---

Published as a conference paper at ICLR 2026
2
WHAT IS HIGH-QUALITY EXPLORATION IN RLVR?
Exploration is fundamental to reinforcement learning, it enables models to discover more warding strategies and thereby avoid being trapped suboptimal behaviors. In this section, we investigate the necessary conditions of high-qualitg exploration in RLVR.
2.1 PRELIMINARY
Let denote the space of all possible token sequences over the LLM's vocabulary, and my denote a L.M with parameters 0. Given question space 'D C S and input q E 'D the model generates sequences т autoregressively according to a conditional distribution n
Definition 2.1 (Exploration Support Set). Given probability threshold 8, and a question q, define the exploration support (=19) that excludes negligible-probabili sequences:
supp(re(((g)) =(- S (ellg))) 5/). }
Although softmax guarantees that every sequence has strictly positive probability, a limited sam- pling budget makes extremely low- now-probability sequences practically unreachable. Therefore, supp(ro(-Ig) characterizes the effective exploration space of the model for a given question q.
Fine-tuming LLM using RL with a reward function R(-) involves repeatedly sampling sequences from the current policy, rewarding the LLM for correct sequences and penalizing for the wrong ones, in order to maximize the expected reward:
J(A) B [R(g, r)].
(2)
In practice, this objective is commonly optimized with Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which has demonstrated strong performance across tasks and enables effective scaling the RLVR paradigm. GRPO leverages the reward scores of G sampled solutions for given question g to estimate advantages, thereby eliminating the need for an additional value model. Formally, let No and To denote the policy before and after the update, each representing distribution over tokens every position. Given question g, set of sampled solution sequences т, from Tha and a reward function R(-), GRPO computes the advantage A, by normalizing rewards within the group,
JGRPO (0) B to (-19)
Ir min (o))(0)). clip (rt,(0), + Eaip) Ar.)) DTo((Toll
(3)
Σ
1
where
no (Ty, 9. Ti,ce)
A R. 1--mmen((1)))) (1B))
(0)
(4)
Te (Ti,t q. Ti,e))
std([
2.2 THE NECESSARY CONDITIONS OF IIGIN-QUALITY EXPLORATION IN RLVR
Definition 2.2 (Explorable Optimal Trajectory Subset). For given question q, the optimal trajectory set within the exploration support (upp(=e(-1g)) is defined
T {I E supp(me(-l4)) R(g. r) Rmaxx())
(S)
where Rmax (q) = supres R(g. T) denotes the maximal achievable reward for question g.
Intuitively. T is subset of the globally optimal trajectories, representing the portion of optimal solutions that the model can actually sample during rollouts. Under the training objective in Eq. (2),
the support of π. will progressively contract toward T eventually concentrating its probability mass on this set. This convergence yields the optimal policy Topt which maximizes the expected reward while maintaining the highest possible output diversity (see Appendix A.1 for proof).
3

---

Published as a conference paper at ICLR 2026
Effectiveness issue. However, key insight is that if the model lacks the ability discover any op- timal trajectory, then T becomes empty. and the reinforcement learning process can no longer make progress. For example, under GRPO, when correct solutions are absent, the normalized advantages A in Eq. (4) tend to approach zero. Consequently, the update term in Eq. (3) becomes ineffective, preventing any policy improvement. Therefore, a necessary condition for high-quality exploration that the policy must be able to discover at least one optimal trajectory within its support.
Diversity issue. During reinforcement learning, policy entropy tend to rapidly collapse, leading to reduced diversity in model outputs and limiting the exploration of a wider range of possible tra- jectories. Some studies have found that the decline in exploratory diversity can hinder performance improvements on unsolved problems (Song et al., 2025). The following theorem formalizes this diversity issue (a detailed proof is provided in the appendix A.
Theorem 2.1 (Entropy Upper-Boune Decay with Increasing Expected Reward). In the binary- reward case R E {0, 1}. let T be the set of optimal trajectories with K = [7*), , M = |S \ TT), N + M, where . denotes the cardinality of set. For any expected reward C' (0, 1), the policy entropy H is upper-houme hopedd- by Hnb (C), given by
H S Hc)(C) (C) + C'log K + (11 C) log M.
(6)
where HD(C) = Coloo C - (1 - C*) log (1 - C). For 02 > C1 with ej larger than the expected reward under the uniform policy on ( 4)) (i.e., Ci > K). the entropy upper bound satisfies the single inequality
0 < Han(ci) - (c2) = (c2 ci) K + Ha(ci) Ho(cc,
The entropy upper bound necessarily decreases as the expected reward increases, with the amount of inversely proportional the size K of the optimal trajectory T
This theorem shows that to prevent a rapid collapse of diversity, high-quality exploration must ensure the discovery of multiple, diverse optimal trajectories. When the set T contains only few optimal solutions, increasing expected reward necessarily forces the policy to concentrate probability mass more aggressively, causing its entropy upper bound to drop rapidly and thus accelerating diversity collapse. In contrast, a larger T can slow down entropy collapse and thus preserve more explo- ration diversity, thereby enabling the policy ultimately achieve higher final performance. Therefore, another necessary condition for high-quality exploration is that the policy must discover multiple distinct optimal trajectories, so that exploration diversity can be preserved during reward improve- ment.
Highlights
In summary, avoid suboptimal convergence under limited exploration budgets, high-quality exploration is indispensable. Specifically, it must satisfy two necessary conditions: effec- tiveness and diversity. If either of these conditions missing, the model will converge to suboptimal solution.
3 MENTOR: MIXED-POLICY EXPERT NAVIGATION FOR TOKEN-LEVEL OPTIMIZATION OF REASONING
As discussed in Section 2, high-quality exploration in RLVR requires both effectiveness and diver- sity. However, existing methods that incorporate expert solutions improve effectiveness but overlook diversity, leading entropy collapse (Zhang et al., 2025a). To address this, propose MENTOR. a framework that balances effectiveness and diversity through two components: Mixed-policy Roll- out, which introduces expert guidance only at critical decision points, and Mixed-poliee GRPO, which integrates these guided rollouts into on- on-policy RL with modified advantage estimation. The overall framework is illustrated in Figure
4

---

Published as a conference paper at ICLR 2026
3.1 MIXED-POLICY ROLLOUT
Existing expert-guided methods, in order to obtain reasoning trajectories beyond the capability of the base model. typically sample full trajectories from the expert model x where every token is generated according to y: ~ (- Vee). and the base model is then trained to imitate each token this sxpert-generated trajectory equally.
However, recent studies show that tokens contribute unequally to reasoning trajectories (Wang et al., 2025). some (e.g., high-entropy tokens) determine critical decision forks, while others only serve as deterministic following. The latter often vary across models in stylistic ways, but such differences have little impact on reasoning process. Entire expert trajectories inevitably contain many these low-impact tokens, which distract the model from learning the key reasoning decisions. To mitigate this problem, we introduce expert guidance only where it is truly needed.
Definition 3.1 MMxed-policy Distribution) At each decoding step t. define token-level mixed- policy distribution that interpolates between the on- on-policy distribution To and the expert distribution n². The expert distribution x² is derived from stronger reference model with the same vocabulary V, such as a larger model or a domain domain-adapted model (Du et al., 2024). Formally, given question and prefix yeu, the sampling distribution for token y: is:
mii (- q. M<er) (1 wex) no(: q, Vet) + wy nt(: (: q, Mee),
(8)
where w2 min(1, Hn/7n) is the interpolation weight determined by the token-level entropy H; Ey (y 4, Meee) log (v g, Vet). and 7 denotes the p-quantile of entropies across tokens in the batch. Thus, high-entropy tokens receive stronger expert guidance, while low-entropy tokens remain closer to the on-policy distribution IT..
By sampling trajectories from this mixed-policy distribution, exploration achieves a balance be- tween effectiveness and diversity. Effectiveness is enhanced because expert guidance is injected uncertain decision points, increasing the probability of discovering correct trajectories. Diversity is preserved because expert guidance restricted only few positions, ensuring that the exploration space remains exponentially large and avoiding collapse to a fixed expert solution. At the same time, selective guidance enables models to focus on learning the core reasoning strategies from the expert.
Accelerating Mixed-policy Rollout. Although miix introduces expert guidance only at critical tokens, standard auto-regressive sampling from Tmix still requires forward computation of both the policy model To and the expert x* every step to determine whether guidance is required, which substantially increases rollout cost and consequently reduces the efficiency of training, especially when the expert has large number of parameters.
Since Tiix deviates from the policy distribution To only on few tokens, while at the remaining po- sitions Tmix is close to Ty. Based on this positional sparsity, we propose an accelerated mixed-policy rollout method based on Speculative Sampling (Chen et al., 2023). Speculative Sampling is un- biased acceleration method that Ict the draft model propose multiple tokens and then verifying them with the target model in parallel. Its acceleration effect depends on the draft acceptance rate, making it naturally suitable for mixed-policy rollout where most tokens align with the policy distribution.
We first let the policy model mo regr-seegrsssvvly generate K candidate tokens U1.K, while record- ing the corresponding sampling distributions (-1g. F<t) at each step t. Next, the expert model computes the distributions " (-14, (-<)) in parallel. Based these results, construct the mixed- policy distribution Tmix (4, Fece) as defined in (8) Each candidate token ÿt is then validated with the acceptance probability
Tmix (Te q, (ce)) min (1,
To q. (et)
If ůr is accepted, the process continues to the next candidate until either a rejection occurs or all K candidates are accepted.
When a candidate rejected, it is resampled from the residual distribution
(Tmix 9: (et))= - To(: q. (:Dc.)).
(10)
where (f(v))+ ((x) m.x/000, (0,f(()))), w V.
5

---

Published as a conference paper at ICLR 2026
This process is repeated to generate complete sequences, enabling substantially faster sampling from the mixed policy while remaining unbiased with Eq.(8), see Appendix A.2 for proof. The detailed algorithm is summarized in Algorithm 1.
Algorithm Accelerating Mixed-policy Rollout with Modified Speculative Sampling
Given lookahead K, entropy threshold "p and maximum response length T', Given expert model n², and on-policy model wa. question sequence q-
Initialize n = O.
while < T do
for /. = : K do
Sample candidate tokens from the policy model ÿe ~ no (-11 V/nn Fct) Compute the token-level entropy H, from the on-policy distribution Te (-1g Usn. Fet) Compute weight W/ < min(1, Ht/7))
end for
In parallel, compute K sets of logits from candidate tokens 91....,
....... ýK
n²" (-1g, Vgn), x² (+1g, Vsn. 31) ,1...... nT" (-19, Vsn UcK)
for 1= 1 K do Sample U10, 1] from a uniform distribution.
Compute Tmix (-14, Mkn) ) + (1 (1- (x) + ua u Vsn)
if < min Tuia (ii (ovvs)) then
Set 3n+11 1/1 and 1 else
sample Vn+1 ~ (Tmix (-19, Vga) TT(.19, vsn)) and exit for loop.
end if end for end while
3.2 MIXED-POLICY GRPO
To effectively integrate samples generated by the mixed-policy rollout into GRPO, we extend the algorithm with a modified advantage function. Specifically, for each query we collect two sets of trajectories: (i) on-policy rollouts 9on (7) sampled from the policy model my, and (ii) mixed- policy rollouts Gmix (r) sampled from the mixed-policy Tmix- Then optimizes the policy model by maximizing the following objective:
N [m
Imised (0)
min (ren()AA,, (r.,))))))))))) 1 -- -, A..)
(11)
Σ
No+No
ITal
On-policy advantages. self-improvement:
For T E Gon- we retain GRPO's group-wise standardization to promote
A R; mcan ( (r,g.))
T Gon-
(12)
std (t (,,.,)
Mixed-policy advantages. For Gmix, we aim to encourage exploration rather than penalize failures. To this end, we define its advantage function the positive excess of its reward over the mean reward of on-policy rollouts:
Ar,(T) a R, - mean ((Rglr,Egm)]4 T E Gmix-
(13)
Reange
where [x]+ max(x, 0) ensures that only above-average exploration is rewarded while failures are ignored, and Prange is a fixed reward span (e.g., the global maximum-minimun reward range) used to normalize rewards into [o, 1] for numerical stability. And a is weighting coefficient that balances the contribution of samples from the mixed-police In our setting, is additionally sched- uled to gradually decay, thereby shifting the policy from guided exploration to self-driven exploration training progresses.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 SETUP
Datasets and Models. We conduct experiments on two model families: Qwen2.5 (Team, 2024) and LLaMA3.1 (Dubey et al., 2024). For Qwen2.5, we use the Qwen2.5-7B-Base and Qwen2.5-3B. Base for experiments. And ing to problems with difficulty levels 3-5 and removing any instances overlapping with the test set to prevent data leakage, total 8,889 training examples. For LLaMA3.1, Base for experiments. However, the MATH dataset is too difficult for this model, such that vanilla GRPO fails to train successfully. To enable comparison between GRPO and other baselines, construct a simplified dataset from dataset for LLaMA3.1. Further dataset and expert model details are provided in the Appendix C.
the MATH dataset (Hendrycks al.) as training dataset, restrict-
use the LLaMA3. 1-BB-
(Hugging Face, 2025) as the training
Evaluations. We evaluate the models along two categories. (i) In-domain performance. We assess the in-domain performance on mathematics benchmarks, including MATH (Hendrycks et al.), AIME24, AIME25, and AMC (Li et al., 2024). (ii) Out-of-domain performance. To examine whether post-tuning affects general reasoning ability beyond mathematics, further evaluate the out-of-domain performance in MMLU-Pro (Wang el al., 2024) and GPQA-diamond (Rein et al.). For AIME24, AIME25, and AMC, report avg @ 32 temperature 0.6 the test set relatively small, while for the other benchmarks, we report pass @ at temperature O.
Baselines. We compare MENTOR with several representative baselines, including: (1) Base: The base model without any fine-tuning. (2) On-policy RL: Standard GRPO without expert guidance, enhanced with token-level loss and the Clip-Higher in DAPO (Yu al., 2025) serve stronger baseline. (3) LUFFY (Yan et al., 2025): A method that integrates full expert trajectories within the GRPO rollout groups. (4) QuestA (Li et al., 2025): A method that provides the first half of expert trajectories as hints for the model to follow. Hyper-parameter and training details of different methods can be found in Appendix C.
4.2
MAIN RESULTS
MENTOR achieves consistent improvements across different models. Table 1 shows that MENTOR outperforms the on-policy RI. baseline across all three backbones. On Qwen2.5-7B, for example, MENTOR lifts the average score on the MATH benchmark from 76.8 81.4, and yields notable relative gains of +4.1, +7.4, and +7.1 points on AIME24, AIME25, and AMC, re- spectively. Similar trends are observed on Qwen2.5-3B and LLaMa3. Importantly, these gains are not confined to in-domain reasoning. MENTOR also delivers clear improvements on out-of- domain benchmarks, demonstrating that the reasoning abilities learned under expert guidance can effectively generalize to out-of-domain tasks.
MENTOR achieves a better trade-off between expert guidance and autonomous exploration. Compared to on-policy RL, LUFFY introduces full expert trajectories but achieves only limited im- provements across all models, indicating that directly imitating expert solutions does fully lever- age expert knowledge. This is likely because full trajectories overly constrain the exploration space, causing the model to overfit superficial expert patterns and fall into suboptimal strategies. QuestA, which provides partial expert trajectories as hints, alleviates over-imitation to some extent but its effectiveness strongly depends on model capacity: it yields clear gains (+1.3) on Qwen2.5-7B, only minor improvement (+0.8) on Qwen2.5-3B. and even a negative effect (-1.6) on LLaMa3.1 1-BB. This because, in the absence of subsequent guidance, the weaker model struggles to explore correct so- lutions, and the excessive hints further disrupt its exploration. In contrast, MENTOR consistently outperforms across different models, achieving a better balance between leveraging expert knowl- edge and maintaining autonomous exploration, thereby achieving significant improvements.

---

Published as a conference paper at ICLR 2026
Table I: MENTOR other baselines. Compared the On-policy RL, MENTOR achieves an average performance improvement of 3.2 %, 4.3 % and 3.9 % on the three models, respectively. The best results are highlighted in bold, and the second best results are underlined.
In-Domain Performance
Out-of-Domain
Methods
Avg
MATH AIME24
AIME25
AMC
Minerva Olympiad
GPQA
ARC
MMLU-Pro
LLaMa3. 1-8B-Base
Base
10.6
0.1
0.0
1.8
4.4
2.1
0.0
0.0
0.1
2.1
On-policy RL
24.0
0.4
0.4
8.0
13.6
6.4
25.8
70.7
35.7
20.6
LUFFY
25.2
0.5
0.4
8.4
14.0
7.1
27.8
74.9
34,9
21.5
QuestA
20.6
0.1
0.2
5.3
8.8
4.0
25.3
72.5
33,9
19.0
MENTOR
30.2
1.2
0.6
10.4
16.2
8.9
30.3
77.3
39.1
23.8
Qwen2.5-3B-BB
Base
47.4
2.4
1.9
17.7
19.9
19.0
3.0
23.6
19.4
17.1
On-policy RL
65.8
3.3
2.5
32.2
25.4
29.8
17.7
72.1
30.6
31.0
LUFFY
64.0
5.2
4.2
32.8
25.0
30.1
15.2
72.5
30.8
31.1
QuestA
66.4
7.9
2.9
34.1
27.6
29.8
16.2
70.3
30.9
31.8
MENTOR
69.8
8.3
3.8
34.2
26.5
35.2
22.7
80.8
36.8
35.3
Qwen2.5-771 Base
Base
62.4
5.4
2.9
26.5
16.9
28.9
11.1
70.4
42.9
29.7
On-policy RI.
76.8
14.2
9.1
46.0
34.2
41.5
29.3
86.0
48.0
42.8
LUFFY
77.0
12.9
10.4
46.4
35.3
40.8
26.8
86.0
49.7
42.8
QuestA
78.8
14.6
13.3
47.4
33.5
41.5
30.3
86.7
51.0
44.1
MENTOR
81.4
18.3
16.5
53.1
34.9
45.2
30.8
89.6
50.2
46.7
Queen2 5-38 MENTOR
Queen2 5-38 Gn-policy
Quen2 5-78 MENTOR
Quen2 5-78 On-policy
1230
MATH500 Accracy
Response Length
Entropy
6.6
2:3
Steps
Steps
Steps
Figure 2: Training dynamics of MENTOR compared with On-policy RL. MENTOR mitigates en- tropy collapse, and its response length dynamics reflect shift from learning to understanding, thereby achieving higher performance.
4.3 TRAINING DYNAMICS
Entropy dynamics. Figure 2 compares the training dynamics of On-policy RL and MENTOR in terms of validation accuracy, entropy and response length. Under On-policy RL, entropy collapses rapidly, indicating that the support of the policy exploration space shrinks prematurely subset of trajectories. MENTOR enhances exploration diversity through selective expert guidance, thereby slowing down entropy collapse and enabling more persistent exploration throughout train- ing. More importantly, the entropy eventually converges to a slightly higher level than On- On-policy RI., indicating that the final support set discussed Section expanded, which directly translates into stronger final performance.
narrow
Response Length dynamics. In the early training stage, MENTOR's 's responses grow in length compared with GRPO. By analyzing rollout samples during training, we find that this rapid growth
8

---

Published as a conference paper at ICLR 2026
stems from adopting expert-style reasoning forks such verify and wait, the occurrence of which extends the reasoning chain. However, as training progresses, MENTOR'S s response length gradually declines, consistent with the scheduled reduction of expert advantage. We find that the model starts to distinguish useful tokens (e.g.. verify) from redundant ones (e.g.. wait), reflecting a shift from expert-guided self-driven exploration. Through this selective absorption, the model achieves more efficient final reasoning pattern, as shown in Appendix G.
4.4 THE ANALYSIS OF REASONING PATTERN
To better understand the reasoning patterns induced by different training methods, Figure 3 reports the occurrence rate high-frequency reasoning tokens, defined as the proportion trajectories in which the token appears at least once, computed from 500 trajectories on MATH500, which provides a more reliable perspective than individual cases. Detailed case studies are provided in Appendix G.
Expert-pollcy
On- policy RL
LUFPY
MENTOR
condider
give
thorefore
Word
Redundancy
atenaotiveyy
dacc
Essence
00
02
0.4
066
05
30 00.00
02
0.6
0.6
0.5
10
0.2
0.6
0.6
0.
10 0.0
0.2
0.4
06
08
10
Occurrence rate
Figure 3: The occurrence rate of high-frequency reasoning tokens under different training meth- ods. MENTOR absorbs the essence of expert trajectories such as verify,, while avoiding over- imitation of redundant tokens like okay or wait.
MENTOR achieves selective absorption of expert knowledge. As shown in Figure 3, although LUFFY successfully incorporate expert knowledge compared with on-policy RL, tends to imi- tate indiscriminately. For example, excessively adopts tokens such as okay and wait, which leads to overly redundant reasoning. In contrast, MENTOR exhibits a more selective learning process, adopting valuable reasoning tokens such as verify and check while avoiding preserving redundant ones. This selective learning shows that MENTOR goes beyond surface imitation, effectively ab- sorbing the essence of expert guidance while discarding the redundancy, resulting an efficient reasoning pattern.
4.5 THE ANALYSIS OF REASONING DIVERSITY
To further quantify the impact of different methods on reasoning diversity, we adopt pass @ as the evaluation metric, which is widely used measure reasoning di- versity (Song et al., 2025: Chen et al., 2025). As shown in Figure 4, Pass @ 32 of On-policy RL stagnates even declines compared to the Base model, as it can only re- shape behaviors within the original capability, resulting in reduced reasoning diversity. By introducing exter- nal expert trajectories, LUFFY and QuestA expand the model's 's capability boundary and raise pass@k. How- ever, these methods are limited in achieving further im- provements in reasoning diversity due to excessive imi- tation. In contrast, by balancing expert guidance with autonomous exploration. MENTOR achieves 9.2% average gain in @2,, indicating clear enhance- ment in reasoning diversity.
Oypelly
#ENTOR
50
330
533223
233 23.
AIMEZ4
AIME25
AMC
Pass@32 performance of Qwen2.5-7B under different methods. MENTOR improves the model's rea- soning diversity beyond other baselines.
Figure 4:

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Learning for Large Language Models Reinforcement learning has recently made significant progress in enhancing the reasoning abilities of LLMs (Jaech et al., 2024; Guo al., 2025; Team et al., 2025). A central development is Reinforcement Learning from Verifiable Rewards (RLVR), which replaces human feedback signals (Kirk al., 2024) with automatically checkable objectives such as mathematical verification (Shao et al., 2024) and program execution (Pennino et al., 2025). Such automatically verifiable signals provide reliable supervision and reduce the risk of reward hacking, thereby enabling stable reinforcement learning for complex reasoning tasks (Guo et al., 2025). However, studies also reveal that the gains RLVR are closely tied to the capability of the base model. For instance, DeepSeck-RR RI reports that while RLVR yields remarkable improve- ments for powerful base models, its benefits become much less pronounced when applied to models with more limited capacity (Guo al., 2025).
Reinforcement
On-Policy Learning under Expert Guidance To improve the effectiveness of RIVR, a line of work incorporates expert trajectories into on-policy RL. training. Some approaches directly mix en- tire expert rollouts with policy rollouts (Yan et al., 2025; Zhang al., 2025a), while others provide partial prefixes of expert trajectories as hints for continued generation (Liu et al., 2025; Zhang et al., 2025b; Li et al., 2025). These strategies have proven effective in reducing unproductive exploration and stabilizing training. However, imitation of fixed expert trajectories restricts exploration, accel- erates entropy collapse (Yan et al., 2025), and ultimately undermines the diversity of reasoning tra- jectories. In addition, the reduction of diversity is further accelerated by gradient imbalance (Huang al., 2025), which drives the model to quickly overfit expert trajectories, especially when their rea- soning patterns diverge substantially from those of the policy model (Zhang et al., 2025a). Although token-level reweighting has been proposed alleviate this issue (Yan al., 2025; Zhang et al., 2025a), the fundamental limitation remains: the exploration is still constrained by the fixed expert trajectories.
LLM reasoning under guidance Generating detailed chains of thought (CoT) has become cen- tral strategy for improving LLM problem-solving performance (Wei et al., 2022). This strategy form of test-time compute (Muennighoff al., 2025), where allocating more
can be viewed inference-time FLOPs leads to better performance. Since the quality of the CoT strongly influences final accuracy, a growing body of work focuses ooptimizing the model's 's reasoning process. Some approaches leverage the model's own confidence or self- cefllaaalttion signals to select higher-value reasoning paths (Yao et al., 2023; Fu al., 2025: Razghandi et al., 2025). Another line introduces process-reward models that help the model progressively search the output space for more promis- ing CoT trajectories during inference (Snell et al., 2025; Setlur et al., 2024; Zhang et al., 2024; Chen et al., 2024). While these methods improve reasoning by searching within the model's own distribution, their exploration remains inherently bounded by the model's capability. In contrast, our work employs guidance from more capable expert model, enabling exploration beyond the policy model's 's native reasoning space and thus providing a stronger mechanism for discovering qughet-quality reasoning trajectories.
CONCLUSION
In this paper, we introduced MENTOR, a powerful framework that enables effective and diverse exploration through selective expert guidance at critical decision points. MENTOR avoids super- ficial imitation and allows policy model to internalize the essence of expert reasoning strategies. Across challenging benchmarks, our method consistently outperforms strong baselines and signif- icantly improves @ performance on complex tasks. These results demonstrate the potential of selective expert guidance to enhance RLVR and suggest promising directions for future research, such as extending the framework to multimodal reasoning or investigating how expert guidance can be provided more effectively.
ACKNOWLEDGMENTS
This work was supported by Ant Group
10

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects animal experi- mentation was involved. All datasets used, such MATH and OpenRI-MATH-22000 were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discriminatory outcomes in our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy security con- cerns. We are committed to maintaining transparency and integrity throughout the research process.
9 REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and datasets have been made publicly available an anonymous repository to facilitate repli- cation and verification. The experimental setup, including training steps, model configurations, and hardware details, is described in detail in the paper. Furthermore, we will also release the model checkpoints from main experiments facilitate future research. The public datasets used in the paper, such as MATH, OpenR1-MATHH-2206 are publicly available, ensuring consistent and reproducible evaluation results.
REFERENCES
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, can-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/8, 2023.
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Alphamath almost zero: process supervision without process. Advances in Neural Information Processing Systems, 01227669-27722 2024.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass @ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint ar Xiv: 10751 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617 2025.
Chengyu Du. Jinyi Han, Yizhou Ying, Aili Chen, Qianyu He, Haokun Zhao, Sirui Xia, Haoran Guo, Jiaqing Liang, Zulong Chen, al. Think thrice before you act: Progressive thought refinement in large language models. arXiv preprint arXiv:2410. 13413,,, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Yichao Fu, Xuewei Wang. Yuandong Tian, and Jiawei Zhao. Deep think with confidence. arXiv preprint 2r02:22508. 15260, 2025.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12948. 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu. Edoardo M Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679, 2025.
11

---

Published as a conference paper at ICLR 2026
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: // //qithub. wwt.c///w/ggggnaccccoocoo
oppn-rll.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, 2024. URL https:// wttp:eewwwwweeee/eunnntette
net. Forummiidiioo 2
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, and Jingzhao Zhang. Questa: Expanding reasoning capacity in Ilms via question augmentation. arXiv preprint arXiv:2507. 13266 2025.
Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning. arXiv preprint 2XXi::2507.10628, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 2025.
Federico Pennino, Bianca Raimondi, Massimo Rondelli, Andrea Gurioli, and Maurizio Gabbrielli. From reasoning to code: Grpo optimization for underreepeesented languages. arXiv preprint arXiv:2506. 1 1027, 2025.
Ali Razghandi, Seyed Mohammad Hadi Hosseini, and Mahdich Soleymani Baghshah. Cer: Confi- dence enhanced reasoning Ilms. arXiv preprint arXiv:2502.1463: 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.08146 2024.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03330. 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng. Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https::/ hwwp///eeeeeeeeeeeeeeeeee .eeteeiiewiieeeeeeeeeeem.....eet forum? 1.d.FAFWW.......
12

---

Published as a conference paper at ICLR 2026
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for Ilm reasoning. arXiv preprint arXiv:2509.0665 0999,, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang. Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501. 2025.
Qwen Team. Qwen2.5: A party foundation models, September 2024, URL httpt::::: github.ic /wwllo///wee.......
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0199 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 112952666-52990 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of- thouhtt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning reason under off- off-policy guidance. arXiv preprint WrXiv:2504149453 2025.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliherate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1809-11822, 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity llms beyond the base model? arXiv preprint arXiv:2504. 200224444.377... 2025.
Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing gpt-4 level mathematical ol olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.0P7 07394, 2024.
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang. Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. arXiv preprint arXiv:2508.11 1408, 2025a.
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak. Bread: Branched rollouts from expert anchors bridge sft rl for reasoning. arXiv preprint MrXiv:2506.17211 2025b.
Yaowei Zheng, Shenzhi Wang Junting Lu, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https: http::///ithhbb com/
ELyouga/EasyRR. 2025.
APPENDIX
A THEORETICAL PROOF
A.1 PROOF OF EXPLORATION DIVERSITY
Lemma 2.1 (Policy Distribution under the Expected-Rewe Reward Constraint). For a fixed question q.
let S, supp(mo(- 4)), T (r S, R(T) R.ax). Based the Maximum Entropy
13

---

Published as a conference paper at ICLR 2026
Principle, the policy distribution that attains the largest entropy under the expected- reward constraint Ep [R] = takes the Gibbs form
Pa(T) = (XP(AR(F)} Z(A)
Z(2) Σ exp(AR(7')) r'ES,
As the reward constraint C approaches its maximal value Rmax, the corresponding multiplier diverges, and all probability mass concentrates on the optimal set T*:
TET,
Px(T)
T
0,
T T
Proof. Since the learning objective in (2) is to maximize expected reward but the exact optimal distribution is unknown, we adopt Maximum Entropy Principle (Jaynes, 1957). Specifically, optimize over all probability mass functions Sg - [0, 1] with Eies, P(T) =
max H(P)
S.L. E P() Σ P(T)=1 TES,
(14)
TES,
where H(P) = - P(T) P(T) and C is the target expected reward. A standard La- grangian calculation yields the unique Gibbs-form solution
(XR())(T) Z(A)
Pa(T)
Z(A) Σ exp(AR(7)))
(15)
r'ES
for some multiplier À > 0 chosen such that EP, =
Define ((A) = Σ- Px(r) Then e'(2) Varp, [ 2 0, so ((A) non-decreasing. More- over, lim1 (()) = Rmaax- Hence as C' Rmaax we must have À - co. and for any $ T and 7* T*
PA(T) Px(+*)
R((T))
(0
(16)
Thus all probability mass concentrates on T in the limit.
Theorem 2.1 (Entropy Upper-Bound Decay with Increasing Expected Reward). In the binary- reward case R(T) E (0, 1), let T be the set of optimal trajectories with K M |S, \ TT), N K + M. For any expected reward E (0, 1), the policy entropy is upper-bounded by
HCb(C) = HC) Colog + (1 (-)) log M.
where HC) -C'log C C*) log(1 C*).
For > C1 with larger than the expected reward under the uniform policy on supp(Tel( g))
(i.e., C1 > N ), the entropy upper bound satisfies the single inequality
0 <
Hab(ei) - (c) = (c2 c1) log K
+
The entropy upper bound necessarily decreases as the expected reward increases, with the amount of inversely proportional the size K of the optimal trajectory T
Proof. Let S, denote supp(πT(- q)). Assume R(T) E {0,, 1} for all r E S, and write
T = (T E S, R(() = 1}, K K-,,% M = |S, T K
For fixed target expected reward C e (0, 1), in the binary case the Gibbs distribution Eq. (15) equivalent to
C
TET,
Te(T)
K'
(17)
1-C
TTT.
M
14

---

Published as a conference paper at ICLR 2026
Thus the mookimum-entrope solution is uniform correct trajectories and uniform incorrect ones, with total mass and C, respectively.
The entropy of ne is
(Cb(C) = Σ
(18)
C - (1- 1-C
-Cloog
(19)
M Hb(C) log + (1 Io M,
K
(20)
where Hn(C) -olog C C) log(1 C?) is the binary entropy. Treating H(C) function of C, we have
C1-CKK
(C) log C' log(1 C) log log M log
(21)
CM
The critical point satisfies H., = which gives
C)K =1
K
C
(22)
CM
K
K+M M
N
i... the expected reward under the uniform distribution on Sq. Moreover, H (C) < 0 whenever C> N so Hub (C) is strictly decreasing for C
K
N-
Now take e₁ c2 with oi $ Since Hub is strictly decreasing on ( 1), we obtain
AHab(K) Hub(ci) Hub(c2) 0.
Thus the entropy necessarily drops when the expected reward increases from ei to in this regime. Next, for fixed c₁, 02 and N, the explicit expression
N - K
(Hab(K) = (c)( (C2) = (H)( (c2) + (e2 (1) log
(23)
K
shows that all dependence on K = |T through the factor log N Differentiating with respect to K yields
i
A (K) = (02 - (1) (om)==N - K <0,
(24)
0K
so (K) is strictly decreasing in K. Hence, for the same reward increase + a larger optimal [T*] always leads a smaller entropy drop. In this sense, the entropy loss scales inversely with the size of T and entropy collapse is slower when the optimal set is larger.
A.2 PROOF OF UNBIASEDNESS FOR MIXED-POLICH ROLLOUT
The unbiasedness of speculative sampling is well established in prior work. For completeness, include concise proof specialized to mixed policy Tmixx confirming that the validation procedure remains unbiased our setting.
Let the token space be V. and fix prefix (g, V<e) at step t. Denote the base policy by
p+(-) To(: q, (.ee),
(25)
and let (-)-) (. M...) be the expert policy. The mixed policy is obtained by a deterministic ensemble of (pt, St),
(-(-) m (.1 g. H..))) ((p.(.).sx.))).
(26)
where M denotes any tokenwise mixing operator that yields valid distribution on V (e.g., convex mixing). The validation procedure only depends on dt-
At step f, a candidate token ÿt is first sampled from Pt. is accepted with probability
(
ar( = min
(27)
ps (T.)
15

---

Published as a conference paper at ICLR 2026
If rejection occurs, a new token is drawn from the residual distribution on V, defined for the dummy variable 2 E V by
(q((2) pr(z))+
re(2) = (xe)(a(< px())))+
(w) = max O3.
(28)
For any possible token v E V, the probability that it becomes the committed token is therefore
P(y = u) = px(x) min (1, pu(e) + P(reject) rx())-
(29)
The first term equals min (n The rejection probability is
P(reject) = 1 - Σ pz(2) min 11, zEV
[c(x) p.(z)
1 Σ =EV
(i)(
E(x((z)
(z()))+
sEV
(30)
which coincides with the denominator rt(.). Consequently, the second term contributes exactly (#x()) +.)))++. Combining the two contributions yields
P(ye = v) = (p)(p. ((()) + (g:(e) - Pa (v)) = q((v).
(31)
Thus the distribution of the validated token is exactly the mixed policy Qr-
To extend the result entire speculative sequences, note that y1- Suppose inductively that the joint distribution of the prefix y<r is II @ (3y)) Conditioning on such prefix, the above calculation shows that ye ~ (+)). Hence, by induction.
the marginal distribution is
T
P g)
II ge (we)
II Xmix q (.B.)),
(32)
(=1
which is identical to direct autoregressive sampling from the mixed policy.
A.3 PROOF OF AUTOMATIC FILTERING OF MISLEADING EXPERT GUIDANCE
We show that the mixed-policy objective intrinsically filters out misleading or low-quality expert guidance, thereby ensuring robustness when the expert is weak. For clarity, rewrite the mixed-policy objective of Eq. (11) in its equivalent expectation form (for analytical convenience, omit the clipping)
R(T) R
[R(T) R1+
Imixed (0) = E, (-11)
+ DD r-mmm (-19)
(33)
std(R)
Prange
where R denotes the average reward obtained by on-policy rollouts on the same query q. and [x]] = max(x,0) 0).
The first expectation corresponds to standard GRPO without expert guidance. Thus, we focus on the second term, which represents the contribution expert guidance. The key observation is that the choice of [+ induces implicit rejection sampling effect. In typical reasoning tasks with binary outcome rewards (correct yields 1, incorrect yields 0). we have
[((T)= R, if T is correct, 10,
[R(T) R]+
(34)
otherwise.
Consequently, any trajectory, which results in incorrect answer because of unsuitable mis- leading expert guidance, obtains zero advantage and thus contributes no gradient signal, ensuring that such erroncous expert signals are automatically filtered out. We further equivalently rewrite the second term as
[[(TT) R2}
EynD, (-)
(35)
Riange
RR(7) R
Tmix (T|g) dr +
0 Tmix (T(g) dr
(36)
Teorrect
Rrange
Tincomact
R(T) R
EgnDD D, Tirsuna (naaaall))) (-14), is correct
(37)
Brange
16

---

Published as a conference paper at ICLR 2026
where Teorrect and Tinoorectt denote, for given query q, the sets of trajectories that yield correct and incorrect outcomes, respectively.
(37) shows that the algorithm learns exclusively from effective expert-guided trajectories. Fur- thermore, the term (R(T) - R) measures the improvement provided by expert guidance over the model's own reasoning, which allows the algorithm distinguish whether success comes from the model itself or from the expert guidance. Only those expert-guided trajectories that provide genuine improvement beyond the model's baseline ability yield a positive advantage and consequently reinforced, while guidance that offers no real benefit results in negligible.
In summary, the mixed-policy objective:
completely suppresses gradient contributions from incorrect expert-guided trajectories, thereby preventing interference from misleading guidance.
only reinforces expert guidance when it provides measurable improvement over the model's self- celf-generated rollouts.
Even in the extreme where the expert can provide only misleading guidance, and correct trajectory can he sampled under such guidance, our method still guarantees a performance lower bound equivalent standard GRPO, since the second expectation in Eq.(33) becomes zero and thus has no effect on the update.
B ALGORITHMIC PROCEDURE OF MENTOR
To complement the main-text description, provide the full algorithmic procedure of MENTOR in Algorithm 2. The algorithm outlines how mixed-policy expert navigation is integrated into on on-policy GRPO training, including the construction of the mixed policy, the dynamic update of the entropy threshold, and the computation of group-wise advantages. For clarity, the pseudocode explicitly separates on-policy rollouts from expert-guides mixed rollouts and highlights how the mixed-policy GRPO objective is optimized at each step.
Algorithm Mixed Mixed-policy Expert Navigation for Token-level Optimization of Reasoning
Given initial policy model Tha expert policy model n², task prompts D. Given hyperparameters N1, N2, P, μa, number of total training steps M. Initialize policy model πтe TTm. Initialize entropy threshold "pp <- inf. for step = M do
Sample batch Dr from D.
Update old policy model Nu <- Tp. Define mixed policcy Tmix in Eq. (8) with Tew xt and 7/p
For each question q E Db, sample outputs
Gon = N ~ Teaa ( Gmix = ( Compute and update the entropy threshold 7p from trajectories in Gon.
No Tmix (: q).
Compute rewards for each trajectory in Gon U 9mix-
Compute advantages A,, e for Gan and Gmix. using Eq. (12) and Eq. (13), respectively. do Update policy parameters 0 by maximizing the Mixed-policy GRPO objective in Eq. (11). end for end for return Te
for mini step
C EXPERIMENTAL DETAILS
Platform. All of our experiments are conducted on workstations equipped with eight NVIDIA A100 GPUs with 80GB memory, running Ubuntu 22.04.4 LTS and CUDA 12.4.
17

---

Published as a conference paper at ICLR 2026
System Prompt. All models trained under MENTOR and other baselines, except QuestA, share the same system prompt for both training and inference:
System
You are a helpful AI Assistant that provides well-reasoned and detailed responses. You FIRST think about the reasoning process as internal monologue and then provide the final answer. The reasoning process MUST BE enclosed within </thin>>tas The final answer MUST BE put in { User {QUESTION]
Assistant
For QuestA, we additionally append "## Hint: Partial Solution" after the QUESTION as hint section.
Reward Setting. For outcome reward, we employ Math-Verify to automatically check whether the final answer inside the "<think>... </think>... \boxed[]" format matches the ground truth, assigning +1 correct and otherwise. In addition, we introduce format reward that grants +1 when the response adheres this format, and if not. The same reward design is applied to MENTOR and all baselines to ensure fairness. For Qwen2.5-7B and Qwen2.5-3B, the weights of outcome reward and format reward are set to 9:1. For LLaMa3. 1-8B, however, this ratio is adjusted to 8:2, since the original weighting did not sufficiently enforce format adherence.
problems from the MATH dataset with difficulty levels 3-5, removing all instances that overlap with the test sets to avoid data leakage. This yields total of 8,889 training examples. However, for LLaMA3.1-8B, this dataset is too difficult, making the vanilla GRPO algorithm hard to apply. To address this issue, we constructed an easier training set from OpenR MeenRR-MMth-22000 by selecting problems with response lengths shorter than 4K tokens, on which the model could be successfully trained using GRPO. All subsequent methods on LLaMA3. I-BB were trained using this simplified dataset. For each problem, the fixed expert trajectory used in LUFFY and QuestA is generated by DeepSeek-R1.
Dataset Details. For Qwen2.5-7B and Qwen2.5-3B.
Export Model Details. For Qwen2.5, We adopt OpenR 1-Qwen-7B² the expert model in MEN- TOR, which is trained on a distilled dataset generated by DeepSeek-R1. 1. For 1.aMA3.1, the expert model in MENTOR obtained by further fine-tuning LLaMA3. 1-8B-Instruct under the same dataset and setting used for OpenRI-Qwent
Training Details. We conduct all experiments using the EasyR1* (Zheng et al., 2025) framework, which employs Verl (Sheng et al., 2024) as the RL training engine and vLLM (Kwon et al., 2023) the rollout engine. The training setup includes rollout batch of 128, learning rate of x 10 generation temperature of 1.0, and a higher-clip of 0.28. Each response sequence is up to 8k tokens in length. We perform 8 rollouts per prompt and do not apply KL divergence or entropy regularization (KL Coeff = O, entropy loss 0). The mini-batch size is set to 64. For important parameters of MENTOR, a is initialized to and annealed to O with cosine schedule over 120 steps, enabling smooth transition from expert guidance to autonomous exploration. The number of mixed-policy rollouts is set to 4. For Yp- is chosen as 0.95, corresponding to the 95-th percentile of token-level entropies within each batch. As special case, 7p is initialized to 999 at the first step.
D EXPLORING ALTERNATIVE FORMS OF EXPERT GUIDANCE
Beyond the entropy-based guidance introduced in the main text, further investigate several alternative ways of determining where and how expert guidance should be injected during mixed- policy rollout.
http:://haggiigaaaccccwooooorrpp 1-Qwen-7B hhtgss//githuh.cm//hiyyggggguue
18

---

Published as a conference paper at ICLR 2026
(1) Random guidance. We begin with simple baseline that injects expert guidance uniformly random throughout decoding, without relying on any uncertainty signal or contextual criterion. At each step, the model routes the next-token decision to the expert policy n with probability 0.2, and to the base policy Ty with probability 0.8. In expectation, this stochastic routing yields the following mixed distribution:
mix (ye, xeee)= 0.8 (V x<x) + 0.2 (va xeee).
(38)
(2) Perplexity- guidance. Token-level perpexxxy measures how confused the model about generating a particular next token. For token Y/ with predicted probability po(u: x<e)), the perplexity defined as
1
PPL(() = exp ( log po [<ee)) po (yt (re))"
(39)
Higher perplexity indicates that the model is more confused about predicting the next token and more likely to make error. To leverage this signal, route the top 20% highest-perplexit tokens to the expert policy. Concretely, let r denote the 80th-percentile threshold of token-level perplexity within the sequence, then the mixed policy is defined
(T"
PPL(() > T,
Tmix (yl xZt))=
(40)
(Te(y x<e) otherwise.
To provide a direct illustration how these guidance mechanisms differ in practice, we further ana- lyze the critical tokens generated by expert. Concretely, we use Qwen2.5-7B B-Base as the base policy and O --Qwen-BB as the expert policy, matching the setup used our main experiments. For each AIME24 query, we decode with temperature (T 1.0) and apply the three guidance strate- gies during generation. By aggregating the guidance tokens generated by the expert ' under each strategy, visualize their distributions in Figure 5.
for For
equction his into Thuchers far such
placed
find bak Returh
withsole
Written TeT
Next
fereda
log
South omaga
mon this
which
find
left
dangle
where
problem
A
retrok
Such
ABWn
ast
Twoeect
Written
agid
Bot
However
orie
find
give
whet
solution
t must.
numbere
Ynno Moruubeer Lightnumber
int
intteent
ifrrom
intormation
drow
emingged
This
user
shorpee
Wastort
Pall
For
area
Let
Wort for However
Spants
on
must problem
this
need
(a) Random guidance
(b) Perplexity-haset guidance
(c) Entropy-based guidance
Figure 5: Word-cloud visualizations of expert-generated guidance tokens under different selection strategies.
Compared with random and perplexity-based guidance, cntropy-based guidance generates many log- ical connectors (e.g., wait, however) that, in our experiments, often trigger new reasoning branches and lead to trajectories whose style and structure differ substantially from the model's own reason- ing without guidance. By contrast, random and perplexity-base guidance rarely introduce such branching points, and the resulting reasoning trajectories remain close to those produced by the base model alone.
To further validate the downstream impact of different guidance strategies, we follow the main train- ing setup and compare random guidance, perplexity- based guidance, and entropy-based guidance on Owen2.5-7B-Bass.
As shown in Table 2, both random guidance and perplexity-basee guidance provide only limited improvement GRPO, with the latter even occasionally degrading performance. In contrast, entropy-based guidance delivers substantial gains on both MATH and AIME24, indicating that ex- pert guidance is more effective when applied at high-entropy positions.
19

---

Published as a conference paper at ICLR 2026
Setting
MATH
AIME24
GRPO
76.8
14.2
MENTOR (Random guidance)
77.6
14.8
MENTOR (Perplexity-basee guidance)
77.0
13.3
MENTOR (Entropy-basee guidance)
81.4
18.3
Table Impact of different guidance on MENTOR performance.
E ABLATION STUDY
E. ABLATION OF METHOD COMPONENTS
We analyze the contributions of each component our methodology. as detailed in Table The observed improvements demonstrate the effectiveness of these components in RL training, with each contributing performance gains on MATH.
Method
MATH
AIME24
Qwen2.5-7B-Bau
62.4
5,4
GRPO
76.8
14.2
+Mixed-policy Rollout
79.4
14.6
+Mixed-policc GRPO
81.4
18.3
Table 3: Main results of progressive components applied to MENTOR
E.2 ABLATION OF EXPERT WEIGHT
We also study the effect of the expert weight a, comparing the default decaying schedule (from to 0) with several fixed-weight baselines. As shown in Table 5, MENTOR consistently outperforms standard GRPO under all settings, indicating that the framework remains stable and effective under various parameter configurations. However, different values of induce distinct patterns in how the model acquires and utilizes expert knowledge.
Setting
MATH
AIME24
GRPO (equiv. to 0)
76.8
14.2
MENTOR (fixed 1.0)
78.2
13.9
MENTOR (fixed a = 0.5)
80.4
16.1
MENTOR (decay
81.4
18.3
Table 4: Effect of expert weights on MENTOR performance.
Introducing expert knowledge consistently improves model performance across all hyperpa- rameter settings. Across all hyperparameter configurations, MENTOR consistently surpasses GRPO (a 0), demonstrating that incorporating expert guidance effectively broadens the model's exploration and improves learning stability. This confirms that absorbing expert knowledge is fun- damentally beneficial for the training process.
Beyond injecting expert information, the model must also consolidate and internalize that knowledge. The experiments reveal that using lower fixed weight (a 0.5) yields stronger performance than an overly high weight (a = 1). This indicates that retaining a degree autonomy allows the model to selectively reinforce the parts of expert knowledge that truly useful, rather than relying on it indiscriminately. In other words, preserving autonomy is necessary for genuine understanding rather than rote imitation.
20

---

Published as a conference paper at ICLR 2026
The decaying schedule achieves the best balance between them. Early in training, high mixing weight accelerates learning by leveraging expert guidance; later, as the weight decreases, the model shifts toward autonomous optimization, refining its strategy and filtering expert signals more effectively. This dynamic adjustment enables the model to both learn from experts and ultimately surpass them, producing the strongest overall performance.
E.3 ABLATION OF ENTROPY THRESHOLD "/p
To assess the sensitivity of MENTOR to the entropy threshold Np, we conduct an ablation study by varying the high-entropy quantile p.
Setting
MATH
AIME24
MENTOR (p = 0.8)
80.8
17.0
MENTOR 0.9)
80.2
17.7
MENTOR (p = 0.95)
81.4
18.3
Table 5: Effect of entropy threshold "/p MENTOR performance.
MENTOR's final performance remains stable across different 7/-- As shown in Table 5, the final performance is largely insensitive to the choice of threshold, indicating that MENTOR remains robust across reasonable range of 7p-
F EFFICIENCY ANALYSIS
To provide a deeper comparison between MENTOR and a range of baselines, including on on-policy RL algorithms (GRPO, DAPO) and expert-guided methods (LUFFY, QuestA), conduct detailed efficiency analysis during 200 training steps on 1a-22--7-B-Bsee using the same hyperparameters in the main experiments. For each method, report the average sequence lengths and the aver- age stage runtimes. Additionally, because different RL methods produce responses of substantially different lengths, we further define an throughput metric to ensure fair comparison across methods, which is computed as the average number of tokens that produce gradients per step divided by the average per-step time. The results are shown in Table 6.
Sequence Length
Stage Time (s)
Method
Total Time (s)
Throughput (tokens/s)
Prompt
Response
Gen
Old
Update
On-policy RL
GRPO
153
828
133
24
87
244
3474
DAPO
153
833
307
25
92
424
2011
Expert-guided RL
LUFFY
153
2902
270
60
230
560
5306
QuestA
510
711
142
31
117
290
2510
MENTOR
153
1751
404
48
175
627
2860
Table 6: Efficiency analysis of different methods. Here, Gen, Old and Update denote respectively the generation (rollout) phase, the computing of the logits of Tola and the model update phase in the Verl framework.
MENTOR achieves the highest performance with only moderate and acceptable additional training overhead. Since different methods generate responses of different lengths, mainly rely on throughput for a fair comparison. Compared with on on-policy RL methods, MENTOR reaches 2860 tokens/s, between GRPO (3474) and DAPO (2011), because DAPO often performs two or three full generation phases to collect enough samples, while MENTOR's s mixed-polic rollouts are more efficient than repeated full generations. For expert-guided methods, LUFFY shows high throughput
21

---

Published as a conference paper at ICLR 2026
partly because it mixes in a full offline expert trajectory of about 6k tokens, which increases the num- ber of processed tokens. From the perspective of the algorithmic design, the throughput of LUFFY's newly generated rollout data should be close to that of GRPO (3474). QuestA concatenates expert segments into the input, creating longer prompts that slightly reduce training throughput. Compared with these approaches, MENTOR achieves the highest final performance, and although relies expert guidance during the rollout stage, the additional overhead remains acceptable.
G
CASE STUDY
To complement the aggregate analysis in Figure 3, we provide representative trajectory-Ievel cases in this section. These examples illustrate how different training methods influence the emergence of reasoning tokens such as verify, check, and wait. By examining full reasoning traces, we highlight qualitative differences between methods, offering additional context to the quantitative analysis in main text.
We find that the incorporation of expert policy indeed shapes the final reasoning patterns of the models, with different methods leading to distinct outcomes.
From a stylistic perspective, QuestA and MENTOR exhibit reasoning traces more aligned with on-policy RI, whereas LUFFY shows a closer resemblance to the expert's style. This difference arises because LUFFY emphasizes direct imitation of expert demonstrations, often reproducing surface-level tokens such as wait and okay, while QuestA and MEN- TOR encourage the model to explore autonomously while using expert guidance, thereby preserving a degree of its own stylistic characteristics.
From content perspective, MENTOR and LUFFY successfully acquire specific reason- ing modes from the expert, such as the use of verify,, which are absent in the base model. In contrast, QuestA, despite leveraging expert policy, does not exhibit the adoption of verify, Our analysis suggests this limitation stems from the design choice of using only the first half of the expert trajectory hints, while tokens like verify typically appear toward the end of trajectories, thereby restricting their adoption.
An example of On-policy RI.
QUESTION
Let æ1, Z2, za V1, y2, and ys be real numbers such that
(x1 x²2²² (3) y2)² 9,
(xr xa) (y) y/1²² 16,
(x2 xz3² + (y2 ya)² 25.
Find
21
T2 1/2
V/3
RESPONSE
<think>
F1
1
To solve for
F2
1/2
we start by defining the determinant of the matrix:
V/3
1
y/1
Ø2
1/2
D3
!/3
The determinant D can be expanded follows:
D
2/2
322
+
F2
y2
2/
F3
I3
y/
22

---

Published as a conference paper at ICLR 2026
This expands to:
D = (1/2 1/)) y1 (22 z3) + (#23/ T312).
So, we have:
D 11222 313/ -2 F1 + 4233 Z3322
We need find D2 To do this, will the given distances between the points (11,g1), (I2, V2), and (ra, ys). The distances are:
(æ1 x²22² + (y1 y2)² = 9,
(x) xa)² (yl y3)² 16,
(I22 x3) (y22 y3)² 25.
These distances suggest that the points (I1, v1). (I2, 3/2), and (Iz, Ms) form a right triangle with sides of lengths 3, 4, and 5. Without loss generality, we can place the points coordinate system where the distances satisfied. One possible configuration is:
(21, v1) O), (#2, 12) (3, 0), (I3, Us) = (0, 4).
Now, we calculate the determinant D for these points:
1
D 3 0 1
1
Expanding this determinant, we get:
0 1 D D=
3
+
0- 12.
4 1
0
1
0
4
So, D = 12, and we need to find D
D = 122 144
Thus, the answer is:
144
</think>
144
example of MENTOR
QUESTION
Let 31, 32, 3a V1, U2, and ys be real numbers such that
(x) )2 + (3/1 =32)² 9,
(x) zz) + (3) ya)² 16,
(x2 x²)² (y22 ya)² 25.
Find
J1 1/1 2
33 //3
RESPONSE
<think>
2
1/1
To find
we start by recalling that the determinant of the given matrix is the
.33
area of the triangle formed by the points (æ), vi), (x2, v2), and (æg, Va) multiplied by 2.
23

---

Published as a conference paper at ICLR 2026
21 !1
Let's denote the determinant by D, D
22
V2
Then, D2 is the square of the area Va
1.
23
of the triangle.
The area A of the triangle can be found using the formula for the area of a triangle with vertices (01,9)). (2/2, 122, and (23, wa):
A = 1/2 | (1/2 vs) + x2(13- vn + 2 y2)l
Thus, D 4.4.. To find A we need to the given distances between the points. The distances are given by the equations:
(x) - x²)² + (y) y2)² 9,
(x) xzg (y1 -Va) 16,
(22 n²)² (U2 ys)² 25.
These equations represent the squared lengths of the sides of the triangle. Let's denote the points as A = (I1, V1). B = (zg, 12), and C = (Iz, 3s)). The distances AB = 3, AC = 4,
and BC' Notice that 3 4 = 52 which suggests that triangle ABC is right triangle with LA = 90".
To verify, let's use the coordinates check if the points form a right triangle. Assume A (1), B (I2, 92), and C' (I3, 1/3). The squared distances are:
AB (x1 -x2)² + (11 1/22 = 9,
AC (x1 zz)² + (y1 ys)² 16,
BC (I2 - x33) 2 + (32 ya)² 25.
Since AB = 3, AC = 4, and BC = 5, and 3 + 42 = 52, the triangle is indeed right triangle with /A 900
The area A of right triangle is given by half the product of the legs:
A = x x AB x AC = 1 x 3 x = 6. = x 6 = 144.
Therefore, D = 4
So, the answer is </think>
144
1/1
To find
22
1/2
we start by recalling that the determinant of the matrix is the area
F3
1/3
of the triangle formed by the points (æ1, V1 ), (x2, U2), and (I3,, Us) multiplied by 2. Let's
x1
/1
1
denote the determinant by D, D
22
U2
1
Then, D is the square of the area of
E3
y3
1
the triangle.
Given the distances between the points:
(x) x2) 2 + (y) y2)² 2 9,
(x1 x3)) 2 (y1 yy)) 16.
(222 n:3) (y/ ys)² 25.
we recognize these as the squared lengths of the sides of triangle with sides AB 3,
AC 4, and BC' = 5. Since 32 + 42 = 5², the triangle is a right triangle with <A = 90". The area A of right triangle is given by:
A= AB x AC 1/2 x 4 6.
The determinant D is twice the area of the triangle:
D 2A 12.
Therefore,
D = 12 144
Thus, the final answer is:
144
24

---

Published as a conference paper at ICLR 2026
example of LUFFY
QUESTION
Let 21, 22, 3a, M1, M2, and ys be real numbers such that
(x) x2)² (y) - -y2)² 9,
(x1 nz)² ) + (y11 - yys) 16.
(x2 xa)² 2 + (3/2 ya)² 25.
Find
T1
T2 1/2
V3
RESPONSE
<think>
Okay, so I've got this problem here about finding the square of the determinant matrix given some conditions the coordinates. Let me try to break it down step by step. First, the problem gives me six real numbers: I1, 22, V/, V3- It also provides three equations involving these numbers:
1. (22) + -22) I need find the square of the determinant of the matrix:
9 2. (x1-x3)² + ya)² 163. (22- ya) = 25
1/1
1
T2
1/2
y/3
1
Hmm, okay. Let me recall that the determinant of a 3x3 matrix can be calculated using the formula:
det = xz (1/2 1 1 ys 1) - 1/1 (222 D3 1) + 1 ( Tz32)
Simplifying that, it becomes:
det = in (y2 - y3) - y1 x2)) + 323/ 4332
So, the square of the determinant is:
det [x1 Us) V1 (x2 233) + 22ト'/ 23/222
But wait, maybe there's a better way to approach this. remember that the determinant of a matrix formed by coordinates of points in plane is related to the area of the triangle formed by those points. Specifically, the absolute value of the determinant twice the area of the triangle. So. perhaps can use that.
Let me denote the points as A(11, B(12, 92), and C(r3, Vs). The determinant is twice the area of triangle ABC. So, if I find the area, can square The area of triangle ABC can be found using the formula:
Area
2 Ian (y2 ya) L2 v1) x3( y2)l
But wait, that's the same as the determinant divided by 2. So, the determinant is twice the area. Therefore, the square of the determinant four times the square of the area. So, I need to find the area of triangle ABC. Let me denote the area as A. Then, the square of the determinant is 44².
Now, the area A can be found using the distance formula. The three given equations are the squared distances between the points. Let me denote the points as A, B, and C. So, AB = 9, AC 16, and 2 25. Therefore, AB 3, AC 4. and BC 5. Hmm, interesting. So, the sides of the triangle are 3, 4, and 5. That's a right triangle because 32 442 52. So, the area of right triangle is base height. In this case, the base and height are 3 and 4, the area is 2 3 4 6. Therefore, the square of the area 62 36. So, the square of the determinant is 36 144.
25

---

Published as a conference paper at ICLR 2026
Wait, let me double-check. The area is 6, so the determinant squared 4 x 6 = 4 x 36 144. That seems correct.
But let me make sure didn't 't make a mistake in the area calculation. The sides are 3, 4, 5, which form a right triangle. The is indeed 2 3 x 6. So, the square is 36, and the determinant squared is 144.
I think that's So, the final answer 144.
</thing >
The square of the determinant is
An example of QuestA
QUESTION
Let 21, 22, 2a. yn, U/, and Ys be real numbers such that
(x) x²)² (3ı 1/2² 9,
(x1 n;3²² 2 (y1 ys)² 16,
( maa 2 (3/2 y3)² 25.
Find
21 V1
T2
23 V/
RESPONSE
<think>
Given the equations:
(21 T2) 2 (y1 y2) 2
(x1 F3) (y) y3)² 16
(x2 x3) (y2 y3)² 25,
Ø1
1/1
need to find
22
1/2
D3
Let's denote the determinant by D:
1/1
D
32
1/2
F3
1/3
The square of the determinant D2 can be found using the formula for the square deter- minant of a 3 x 3 matrix. However, we can use geometric interpretation to simplify the problem.
Consider the points A (#1, 31), B (I2, V2). and C (Zz, ya) in the coordinate plane. The given equations represent the squared distances between these points:
AB 9.
2C 16,
BC² 25.
These distances suggest that the points A, B, and C form a right triangle with BC as the hypotenuse. This is because 9 + 16 25, which satisfies the Pythagorean theorem.
Without loss of generality, let's assume is the right angle, so A. B, and C' form a right triangle with BC' as the hypotenuse. This assumption simplifies The area of the triangle ABC can be calculated using the distances given. The area K of triangle with vertices (æ11 V1), (x2,, V2), and (Za, ya) can also be found using the determinant formula for the area of a triangle:
calculations.
K
x (22 Vs) + I2lVV v1) + +.alV1 M2)1-
2
26

---

Published as a conference paper at ICLR 2026
The square of the determinant D is related the area of the triangle. Specifically, the square of the determinant of the matrix formed by the coordinates of the points is four times the square of the area of the triangle:
= 4K=
To find K2, we use the fact that the points form right triangle. The legs of the triangle are the distances AB and AC, and the hypotenuse is BC. The area of the right triangle can be calculated as:
K 2 .A AC'
V v
3
12 6.
2
2
Thus, the square of the area K2 is:
k² = 6² = 36.
Therefore, the square of the determinant D is:
D 4k² 36 144.
</think>
The final answer is:
144
H LLM USAGE
Large Language Models (LLMs) were used to aid the writing and polishing of the manuscript. Specifically, we used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
is important to note that the LLM was not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with involvement in the scientific content data analysis.
developed and conducted by
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres ethical guidelines and does not contribute to plagiarism or scientific misconduct.
27


Reviews:
[{"review_id": "6DKFYqODPe", "reviewer": "Reviewer_uaif", "summary": "This paper introduces MENTOR, a reinforcement learning framework that aims to enhance both the effectiveness and diversity of exploration in RLVR for LLMs. Inject expert guidance only at critical decision points (high-entropy tokens) rather than imitating entire expert trajectories (such as LUFFY). In-Domain and OOD performance evaluation (LLaMa3.1-8B-Base, Qwen2.5-3B-Base and Qwen2.5-7B-base) demonstrate MENTOR's effectiveness.", "questions": "What would happen if the token points for guidance were selected randomly or uniformly? \n\nHow robust is the proposed method to the expert’s capability — for instance, if the expert is weaker, can the model filter out misleading guidance? \n\nIn the baselines, LUFFY and QuestA belong to expert imitation. Are there any more dynamic or flexible expert guidance baselines for comparison?\n\nIf the authors can provide more detailed verification of the method’s effectiveness and robustness, I will increase my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Clear problem framing: Rollout's effectiveness and diversity are important for RL. Paper's section details description this aspect.\n- Critical point guidance: By leveraging token entropy, providing expert guidance only at critical decision points, steers reasoning trajectories while preserving the policy’s own exploration, rather than entirely imitating expert.\n- Empirical Results: Based on three backbone models, proposed method MENTOR compared to Vanilla GRPO, LUFFY and QuestA, demonstrate the method's effectiveness.", "weaknesses": "- Experiment: The Table 1's experiment appear insufficient, given that the paper’s pipeline involves multiple combinations of hyperparameters and strategies. \n- Critical point selection: The selection strategy for critical decision points could be elaborated in more detail — for example, why is it based on token-level entropy rather than other metrics (e.g., higher-level semantic)?\n- Typos: Line 322 QuestA appear twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lcsdqesyJb", "reviewer": "Reviewer_Kxjd", "summary": "This paper introduces MENTOR (Mixed-policy Expert Navigation for Token-level Optimization of Reasoning), a GRPO+ training method for improving the reasoning capabilities. By providing guidance on critical tokens during inference, MENTOR achieves effective improvement of reasoning while preserving the diversity of group-based sampling. Experiments on math and science benchmarks show that MENTOR consistently outperforms over baselines like GRPO, LUFFY, and QuestA.", "questions": "1. Could the authors provide a more detailed comparison between MENTOR and DAPO? Although DAPO may require additional time for resampling, it does not incur the extra GPU memory overhead associated with loading a separate expert model. In this respect, DAPO could be a more resource-efficient alternative to MENTOR.\n\n2. Could the authors provide concrete case studies to better illustrate MENTOR's impact on sampling diversity? While Figure 3 presents keyword statistics, a more direct qualitative analysis or examples would offer clearer insight into the diversity achieved through MENTOR.\n\n3. Could the authors provide an explanation about changing the selected expert model rather than OpenR1-Qwen-7B? How does performance change if the expert is weaker or from a different architecture/family?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the paper is reasonable, while full expert guidance could lead to the reduction of sampling diversity, MENTOR implements a relaxed guidance that guides the sampling only on critical tokens, leading to the balance of training efficacy and sampling diversity.\n\n2. The MENTOR also involves an accelerated components that address the efficiency problem of the expert-guidance method, which is also a unique contribution.\n\n3. This paper does comprehensive experiments on both in-domain and out-of-domain benchmarks, showing the efficacy and generalizability of MENTOR. In addition, the analysis results of entropy show that MENTOR can preserve the generation diversity during training.", "weaknesses": "1. One of the main weaknesses is that the mechanism of \"MENTOR only guides the decoding on critical tokens\". This paper shows MENTOR's decoding strategy in Sec. 3.1, but does not compare this strategy with prior methods of fully expert-guided decoding. A more detailed introduction and explanation of the decoding strategy is required.\n\n2. The motivation for using an expert-guided method is not entirely clear. If a powerful expert model is already available, it seems straightforward to use it directly for reasoning tasks. Why is it necessary to further train and distill its knowledge into a smaller model? I would appreciate a more detailed discussion on the advantages and rationale behind training a smaller model with expert guidance.\n\n3. The expert-guided method costs more training resources compared with vanilla GRPO, since it requires loading a bigger expert model for decoding, which will occupy a large amount of GPU resources. Even though this paper proposes an accelerated strategy, the extra cost of GPU computation and memory can not be overlooked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3leLkHui8K", "reviewer": "Reviewer_ugNa", "summary": "MENTOR addresses the challenge of balancing effectiveness and diversity in Reinforcement Learning with Verifiable Rewards (RLVR). The method mixes the policy and expert distributions at the token level using entropy-based weights, and introduces an “accelerated mixed-policy rollout” to improve sampling efficiency. \nExperiments on Qwen2.5 and LLaMA3.1 show consistent gains across in-domain and out-of-domain reasoning benchmarks.", "questions": "3.1 Can the authors clarify how the “accelerated rollout” avoids computing π* at every step when wt is small? \n\n3.2 Did you test other mixture strategies (e.g., thresholded expert calls or adaptive gating) to validate that entropy weighting is optimal?\n\n3.3 How sensitive are results to γp and α scheduling?\n\n3.4 Can you provide more seeds for the results?\n\n3.5 Could you report results with a non-zero KL coefficient to show how it affects stability and performance?  Also, please clarify the reasoning behind setting the KL term to 0 during MENTOR fine-tuning.\n\nI am willing to increase my score if the authors adress my concerns/questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.1 The motivation is clear and well-grounded. The paper directly tackles the exploration–effectiveness trade-off in RLVR.\n\n1.2 The proposed PMix mechanism (Eq.6) is conceptually interesting, using entropy as a proxy for critical decision points.\n\n1.3 Results are consistently strong across multiple model scales and benchmarks, showing stable gains.", "weaknesses": "2.1 The “accelerating mixed-policy rollout” (Eq.7) appears inconsistent. Even with a small wt (e.g., 0.01), the expert π* must still be evaluated to compute the mixed distribution, so the method does not actually avoid the cost of expert inference. A gating mechanism for small weights would be necessary for genuine speed-up.\n\n2.2 The claimed parallelization (L216–220) is unclear. Eq.7 explicitly depends on π*, so it cannot be fully parallelized in the autoregressive decoding process. The efficiency argument based on speculative sampling seems overstated.\n\n2.3 Missing ablations for critical parameters (γp, α) and no exploration of alternative PMix strategies.\n\n2.4 In Appendix A.2, the authors set the KL regularization coefficient to 0 during MENTOR fine-tuning. \nThis is concerning, as the KL term is typically used to constrain the policy from drifting too far from the base model. Disabling it may weaken stability and allow distributional drift, which the paper does not quantify or justify empirically.\n\n2.5 Results appear to be based on single runs, with no reported variance or standard deviations. This limits statistical reliability and makes it difficult to assess the robustness or significance to the provided results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HnNFKJ52EH", "reviewer": "Reviewer_QY8s", "summary": "This paper proposes MENTOR, a reinforcement learning framework for large language models that enhances reasoning by providing expert guidance only at critical decision points rather than on entire reasoning paths. This approach promotes both effective and diverse exploration, enabling models to capture strategic essence rather than superficial imitation, which leads to superior performance compared to existing RLVR methods.", "questions": "- Why do you still adopt trajectories from the current model in Eq. 9? Any insights here?\n- In Eq. 11, you write $A_i = \\dfrac{[R\\_i - mean(R\\_j)]\\_+}{R\\_{\\rm range}}$ rather than $\\dfrac{[R\\_i - mean(R\\_j)]\\_+}{{\\rm std}(R\\_j)}$, why? Why only normalize the advantages from the mixed policy into [0, 1] but use Eq. 10 directly from GRPO?\n- Can the proposed MENTOR still work when the base LLM is strong?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is easy to read and easy to follow\n- This paper focuses on an important topic in large language model reasoning, aiming at enhancing the exploration capabilities of existing LLMs via some specially designed approaches\n- The authors provide codes in the main text, which enhances the reliability of the results reported in this paper\n- The experiments are carried out using both LLaMa models and Qwen models with different model sizes, which shows the generality of the proposed method to some extent", "weaknesses": "- (major) I do not see the necessity of Definitions 2.1 and 2.2. If the authors would like to claim that the optimal trajectory can be rarely covered, and hence raises two critical issues in the RLVR training, there is no need to state Definition 2.1 and 2.2 since the readers can generally get that. The organization of this paper can be improved\n- (major) No ablation study and parameter study can be found either in the main text or the appendix. This makes it hard to figure out which component (mixed-policy rollout and mixed-policy GRPO) contributes most to the performance of MENTOR. It is also hard to check how sensitive MENTOR is to the introduced hyperparameters, e.g., the initial value of $\\alpha$ in Eq. 11, how different decay schedules of $\\alpha$ affect the performance, how to set $p$ in $\\gamma_p$, etc.\n- (major) Despite the fact that the authors propose a method to accelerate mixed-policy rollout, the entire process of MENTOR is still both time-consuming and memory-consuming compared to other methods like QuestA. The authors ought to show the training efficiency of MENTOR against baseline methods\n- (major) It is somewhat strange to me that the authors use different expert models for different base LLMs, e.g., OpenR1-Qwen-7B model for Qwen models and finetuned LLaMA3.1-8B-Instruct model for LLaMa models. Is there any reason that you do not use a fixed expert model? Why do you use the Instruct model for LLaMa here? It is unclear whether the proposed method would fail if one adopts other LLMs as the base model, e.g., deepseek-R1 as the expert model. \n- (major) It is also unclear whether the proposed method still works if the expert model is only marginally stronger than the base model, or if the expert model is too strong (e.g., Qwen3-max), and whether the proposed method can still work. Note that QuestA conducts experiments using some strong base LLMs. The authors commented that the advantages of QuestA rely heavily on the vanilla performance of the base models, while it raises questions on whether MENTOR can work if it is adopted upon some strong base models (e.g.,  Nemotron-1.5B)\n- (minor) The related work part can be significantly improved. LLM reasoning and LLM reasoning under guidance are fast-growing research areas, and there are numerous papers that do similar things. The authors should include more discussion in the manuscript\n- (minor) No full pseudo-code for MENTOR can be found. Algorithm 1 just states part of the MENTOR algorithm\n- (minor) There are numerous minor issues in the manuscript. I just list some of them below, please double-check your manuscript,\n  - Line 019, *we argue that the expert only needs to provide guidance only at critical*, two *only* in this sentence\n  - Line 322, two QuestA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)