Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs

## Abstract

Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR : Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online 1 1 1 https://github.com/Jiangzs1028/MENTOR .

## 1 Introduction

Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). It has significantly improved models’ performance in solving challenging mathematics and programming problems, as evidenced by models such as OpenAI-o1 (Jaech et al., 2024), DeepSeek-R1 (Guo et al., 2025), and Kimi-1.5 (Team et al., 2025). These improvements are largely attributed to the models’ ability to generate detailed chains of thought (CoT) before giving final answers (Wei et al., 2022), which is termed test-time scaling (Muennighoff et al., 2025).

However, the effectiveness of RLVR strongly depends on the capability of base models. It has been observed that when applied to models with limited parameters, RLVR fails to reproduce the remarkable gains observed on powerful base models (Guo et al., 2025).

This issue arises because RLVR requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Specifically, when the task is overly challenging for the model, it often struggle to discover any correct reasoning trajectory (Yue et al., 2025), resulting in ineffective exploration that hinders training (Yu et al., 2025). Furthermore, even when correct solutions are found, limited diversity of reasoning trajectories often leads the model to rapidly converge to a narrow set of solutions (Song et al., 2025), which reflected in entropy collapse (Cui et al., 2025) and ultimately traps it in suboptimal solutions (Song et al., 2025).

Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. While such imitation reduces ineffective exploration (Yan et al., 2025; Zhang et al., 2025a, b; Liu et al., 2025; Li et al., 2025), it forces the model to follow to fixed expert trajectories, thereby restricting the diversity of exploration and accelerating entropy collapse (Yan et al., 2025). In addition, the reduction of diversity is further accelerated by gradient imbalance (Huang et al., 2025), which drives the model to quickly overfit expert trajectories, especially when expert reasoning patterns diverge substantially from those of the policy model (Zhang et al., 2025a). Although some works attempt to mitigate it by reweighting tokens in expert trajectories (Yan et al., 2025; Zhang et al., 2025a), the relief remains superficial, as the exploration space is still fundamentally restricted by the fixed expert trajectories.

To achieve better exploration, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning trajectory. Expert guidance is indeed essential for steering the model toward correct solutions, but blindly imitating full expert trajectories restricts the exploration space. Since tokens contribute unequally to reasoning trajectories (Wang et al., 2025), introducing guidance at critical decision points enables the model to best leverage expert knowledge while preserving exploration diversity. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that injects expert guidance only at critical decision points to perform effective and diverse exploration. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby sustaining high-quality exploration and achieving superior overall performance.

Our contributions can be summarized as follows:

- •
We provide a formal analysis of RLVR and demonstrate that effective policy improvement critically depends on high-quality exploration, which requires not only discovering correct solutions but also maintaining sufficient diversity to prevent entropy collapse and avoid being trapped in suboptimal solutions.
- •
We are the first to propose leveraging expert knowledge only at critical decision points in RLVR training rather than imitating entire expert trajectories, thereby enabling models to achieve both effective and diverse exploration in RLVR.
- •
We conduct extensive experiments showing that MENTOR delivers consistent improvements on six challenging math benchmarks and out-of-domain tasks, with gains stable across diverse model families. Further analysis reveals that it mitigates entropy collapse in RLVR training and broadens the capability boundary of base models, and case studies demonstrate it can selectively absorb expert knowledge rather than superficial imitation.

Figure: Figure 1: Illustration of MENTOR framework. By providing expert guidance only at critical decision points, MENTOR steers reasoning trajectories while preserving the policy’s own exploration, thereby avoiding the constraints of fixed expert trajectories and achieving more effective and diverse exploration in RL training.
Refer to caption: x1.png

## 2 What is high-quality exploration in RLVR?

Exploration is fundamental to reinforcement learning, as it enables models to discover more rewarding strategies and thereby avoid being trapped in suboptimal behaviors. In this section, we investigate the necessary conditions of high-quality exploration in RLVR.

### 2.1 preliminary

Let $\mathcal{S}$ denote the space of all possible token sequences over the LLM’s vocabulary, and let $\pi_{\theta}$ denote a LLM with parameters $\theta$. Given a question space $\mathcal{D}\subseteq\mathcal{S}$ and a input $q\in\mathcal{D}$, the model generates sequences $\tau$ autoregressively according to a conditional distribution $\pi_{\theta}(\cdot|q)$.

Definition 2.1 (Exploration Support Set). Given a probability threshold $\delta_{p}$ and a question $q$,
define the exploration support of $\pi_{\theta}(\cdot|q)$ that excludes negligible-probability sequences:

$$ $\text{supp}(\pi_{\theta}(\cdot|q))=\Big\{\tau\in\mathcal{S}\big|\pi_{\theta}(\tau|q)>\delta_{p}\Big\},$ (1) $$

Although softmax guarantees that every sequence has strictly positive probability, a limited sampling budget makes extremely low-probability sequences practically unreachable. Therefore, $\text{supp}(\pi_{\theta}(\cdot|q))$ characterizes the effective exploration space of the model for a given question $q$.

Fine-tuning LLM $\pi_{\theta}$ using RL with a reward function $R(\cdot)$ involves repeatedly sampling sequences from the current policy, rewarding the LLM for correct sequences and penalizing for the wrong ones, in order to maximize the expected reward:

$$ $J(\theta)=\mathbb{E}_{q\sim\mathcal{D},\tau\sim\pi_{\theta}(\cdot\mid q)}[R(q,\tau)].$ (2) $$

In practice, this objective is commonly optimized with Group Relative Policy Optimization (GRPO) (Shao et al., 2024), which has demonstrated strong performance across tasks and enables effective scaling in the RLVR paradigm. GRPO leverages the reward scores of $G$ sampled solutions for a given question $q$ to estimate advantages, thereby eliminating the need for an additional value model. Formally, let $\pi_{\theta_{\text{old}}}$ and $\pi_{\theta}$ denote the policy before and after the update, each representing a distribution over tokens at every position. Given a question $q$, a set of sampled solution sequences $\tau_{i}$ from $\pi_{\theta_{\text{old}}}$, and a reward function $R(\cdot)$, GRPO computes the advantage $A_{i}$ by normalizing rewards within the group,

$$ $\displaystyle\mathcal{J}$ ${}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\left\{\tau_{i}\right\}_{i=1}^{G}\sim\pi_{\theta_{\text{old }}}(\cdot\mid q)}$ $\displaystyle{\left[\frac{1}{\sum_{i=1}^{G}\left|\tau_{i}\right|}\sum_{i=1}^{G}\sum_{t=1}^{\left|\tau_{i}\right|}\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\operatorname{clip}\left(r_{i,t}(\theta),1\pm\varepsilon_{\text{clip}}\right)\hat{A}_{i,t}\right)-\beta D_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})\right]}$ (3) $$

where

$$ $r_{i,t}(\theta)=\frac{\pi_{\theta}\left(\tau_{i,t}\mid q,\tau_{i,<t}\right)}{\pi_{\theta_{\text{old }}}\left(\tau_{i,t}\mid q,\tau_{i,<t}\right)},\quad\hat{A}_{i,t}=\frac{R_{i}-\operatorname{mean}(\left\{R_{i}\right\}_{i=1}^{G})}{\operatorname{std}(\left\{R_{i}\right\}_{i=1}^{G})}.$ (4) $$

### 2.2 The necessary conditions of high-quality exploration in RLVR

Definition 2.2 (Optimal Trajectory Set).
For a given question $q$, the optimal trajectory set within the exploration support $\text{supp}(\pi_{\theta}(\cdot|q))$ is defined as

$$ $\mathcal{T}^{\star}=\left\{\tau\in\text{supp}(\pi_{\theta}(\cdot|q))\;\middle|\;R(q,\tau)=R_{\max}(q)\right\},$ (5) $$

where $R_{\max}(q)=\sup_{\tau\in\text{supp}(\pi_{\theta}(\cdot|q))}R(q,\tau)$ denotes the maximal achievable reward within the exploration support $\text{supp}(\pi_{\theta}(\cdot|q))$ .

If rollouts were sufficient to cover the entire $\mathcal{T}^{\star}$ defined in Definition [2.2](https://arxiv.org/html/2510.04140v1#S2.SS2), training under the objective in Eq.([2](https://arxiv.org/html/2510.04140v1#S2.E2)) would progressively shrink the support of $\pi_{\theta}$ toward $\mathcal{T}^{\star}$, eventually concentrating its probability mass on $\mathcal{T}^{\star}$. This convergence corresponds to the optimal policy $\pi_{\text{opt}}$, where the policy assigns non-negligible probability only to sequences that achieve the maximal reward (see Appendix [A.1](https://arxiv.org/html/2510.04140v1#A1.SS1) for proof).

However, since the exploration space remains exponentially large, while only a limited number of trajectories can be sampled during the rollout phase in reinforcement learning, $\mathcal{T}^{\star}$ can rarely be fully covered (Wu et al., 2025), which raises two critical issues in the RLVR training.

#### Effectiveness issue.

With a limited number of rollouts, the sampled sequences may fail to contain any correct solutions. One reason is that low-probability correct trajectories are missed under the restricted exploration budget. A more serious reason is that $\mathcal{T}^{\star}$ itself contains not any correct solution due to the limited capability of models, in which case no amount of additional sampling could help. When correct solutions are absent, the normalized advantages $\hat{A}_{i,t}$ in Eq. ([4](https://arxiv.org/html/2510.04140v1#S2.E4)) tend to approach zero. As a result, the update term in Eq. ([3](https://arxiv.org/html/2510.04140v1#S2.E3)) becomes ineffective and policy improvement cannot be achieved.

#### Diversity issue.

Even when correct sequences are sampled, they often come from a narrow high-probability region of $\mathcal{T}^{\star}$, while low-probability but correct alternatives remain largely unexplored. As these frequent outputs are consistently reinforced, the support of policy quickly shrinks to this narrow subset, manifesting as entropy collapse. Although this premature concentration yields short-term gains, it may ultimately undermine long-term performance. In addition, some studies have found that the shrinkage of the exploration space on solved problems will propagate to unsolved ones, thereby constraining the ultimate performance of reinforcement learning (Song et al., 2025).

## 3 MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning

As discussed in Section [2](https://arxiv.org/html/2510.04140v1#S2), high-quality exploration in RLVR requires both effectiveness and diversity. However, existing methods that incorporate expert solutions improve effectiveness but overlook diversity, leading to entropy collapse (Zhang et al., 2025a). To address this, we propose MENTOR, a framework that balances effectiveness and diversity through two components: Mixed-policy Rollout, which introduces expert guidance only at critical decision points, and Mixed-policy GRPO, which integrates these guided rollouts into on-policy RL with modified advantage estimation. The overall framework is illustrated in Figure [1](https://arxiv.org/html/2510.04140v1#S1.F1).

### 3.1 Mixed-policy Rollout

Tokens contribute unequally to reasoning trajectories (Wang et al., 2025). some (e.g., high-entropy tokens) determine critical decision forks, while others only serve as deterministic following. The latter often vary across models in stylistic ways, but such differences have little impact on reasoning process. Entire expert trajectories inevitably contain many of these low-impact tokens, which distract the model from learning the key reasoning decisions. To mitigate this problem, we introduce expert guidance only where it is truly needed.

Definition 3.1 (Mixed-policy Distribution)
At each decoding step $t$, we define a token-level mixed-policy distribution that interpolates between the on-policy distribution $\pi_{\theta}$ and the expert distribution $\pi^{*}$. The expert distribution $\pi^{*}$ is derived from a stronger reference model with the same vocabulary $\mathcal{V}$, such as a larger model or a domain-adapted model (Du et al., 2024).
Formally, given question $q$ and prefix $y_{<t}$, the sampling distribution for token $y_{t}$ is:

$$ $\pi_{\text{mix}}(\cdot\mid q,y_{<t})=(1-w_{t})\,\pi_{\theta}(\cdot\mid q,y_{<t})+w_{t}\,\pi^{*}(\cdot\mid q,y_{<t}),$ (6) $$

where $w_{t}=\min\!\left(1,{H_{t}}/{\gamma_{p}}\right)$ is the interpolation weight determined by the token-level entropy $H_{t}=-\sum_{y}\pi_{\theta}(y\mid q,y_{<t})\log\pi_{\theta}(y\mid q,y_{<t})$, and $\gamma_{p}$ denotes the $p$-quantile of entropies across tokens in the batch. Thus, high-entropy tokens receive stronger expert guidance, while low-entropy tokens remain closer to the on-policy distribution $\pi_{\theta}$.

By sampling trajectories from this mixed-policy distribution, exploration achieves a balance between effectiveness and diversity. Effectiveness is enhanced because expert guidance is injected at uncertain decision points, increasing the probability of discovering correct trajectories. Diversity is preserved because expert guidance is restricted to only a few positions, ensuring that the exploration space remains exponentially large and avoiding collapse to a fixed expert solution. At the same time, selective guidance enables models to focus on learning the core reasoning strategies from the expert.

#### Accelerating Mixed-policy Rollout.

Although $\pi_{\text{mix}}$ introduces expert guidance only at critical tokens, standard auto-regressive sampling from $\pi_{\text{mix}}$ still requires forward computation of both the policy model $\pi_{\theta}$ and the expert $\pi^{*}$ at every step to determine whether guidance is required, which substantially increases rollout cost and consequently reduces the efficiency of training, especially when the expert has a large number of parameters.

Since $\pi_{\text{mix}}$ deviates from the policy distribution $\pi_{\theta}$ only on a few tokens, while at the remaining positions $\pi_{\text{mix}}$ is close to $\pi_{\theta}$. Based on this positional sparsity, we propose an accelerated mixed-policy rollout method based on Speculative Sampling (Chen et al., 2023). Speculative Sampling is an unbiased acceleration method that let the draft model propose multiple tokens and then verifying them with the target model in parallel. Its acceleration effect depends on the draft acceptance rate, making it naturally suitable for mixed-policy rollout where most tokens align with the policy distribution.

We first let the policy model $\pi_{\theta}$ auto-regressively generate $K$ candidate tokens $\tilde{y}_{1:K}$, while recording the corresponding sampling distributions $\pi_{\theta}(\cdot|q,\tilde{y}_{<t})$ at each step $t$. Next, the expert model computes the distributions $\pi^{*}(\cdot|q,\tilde{y}_{<t})$ in parallel . Based on these results, we construct the mixed-policy distribution $\pi_{\text{mix}}(\cdot|q,\tilde{y}_{<t})$ as defined in Eq.([6](https://arxiv.org/html/2510.04140v1#S3.E6)). Each candidate token $\tilde{y}_{t}$ is then validated with the acceptance probability

$$ $\min\left(1,\frac{\pi_{\text{mix}}(\tilde{y}_{t}\mid q,\tilde{y}_{<t})}{\pi_{\theta}(\tilde{y}_{t}\mid q,\tilde{y}_{<t})}\right).$ (7) $$

If $\tilde{y}_{t}$ is accepted, the process continues to the next candidate until either a rejection occurs or all $K$ candidates are accepted.

When a candidate is rejected, it is resampled from the residual distribution

$$ $\big(\pi_{\text{mix}}(\cdot\mid q,\tilde{y}_{<t})-\pi_{\theta}(\cdot\mid q,\tilde{y}_{<t})\big)_{+}.$ (8) $$

where $(f(v))_{+}=\max(0,f(v))\,/\,\sum_{v}\max(0,f(v)),\quad v\in\mathcal{V}$.

This process is repeated to generate complete sequences, enabling substantially faster sampling from the mixed policy while remaining unbiased with Eq.([6](https://arxiv.org/html/2510.04140v1#S3.E6)), see Appendix [A.2](https://arxiv.org/html/2510.04140v1#A1.SS2) for proof. The detailed algorithm is summarized in Algorithm [1](https://arxiv.org/html/2510.04140v1#alg1).

Figure: Algorithm 1 Accelerating Mixed-policy Rollout with Modified Speculative Sampling

### 3.2 Mixed-policy GRPO

To effectively integrate samples generated by the mixed-policy rollout into GRPO, we extend the algorithm with a modified advantage function. Specifically, for each query $q$, we collect two sets of trajectories: (i) on-policy rollouts $\mathcal{G}_{on}=\{\tau\}^{N_{1}}$ sampled from the policy model $\pi_{\theta}$, and (ii) mixed-policy rollouts $\mathcal{G}_{\text{mix}}=\{\tau\}^{N_{2}}$ sampled from the mixed-policy $\pi_{\text{mix}}$. Then optimizes the policy model by maximizing the following objective:

$$ $\mathcal{J}_{\text{mixed}}(\theta)=\frac{1}{\sum_{i=1}^{N_{1}+N_{2}}\left|\tau_{i}\right|}\sum_{i=1}^{N_{1}+N_{2}}\sum_{t=1}^{\left|\tau_{i}\right|}\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\operatorname{clip}\left(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right)$ (9) $$

#### On-policy advantages.

For $\tau\in\mathcal{G}_{\text{on}}$, we retain GRPO’s group-wise standardization to promote self-improvement:

$$ $\hat{A}_{i,t}(\tau)=\frac{R_{i}-\operatorname{mean}\!\big(\{R_{j}\}_{\tau_{j}\in\mathcal{G}_{\text{on}}}\big)}{\operatorname{std}\!\big(\{R_{j}\}_{\tau_{j}\in\mathcal{G}_{\text{on}}}\big)},\quad\tau\in\mathcal{G}_{\text{on}}.$ (10) $$

#### Mixed-policy advantages.

For $\tau\in\mathcal{G}_{\text{mix}}$, we aim to encourage exploration rather than penalize failures. To this end, we define its advantage function as the positive excess of its reward over the mean reward of on-policy rollouts:

$$ $\hat{A}_{i,t}(\tau)=\alpha\cdot\frac{\big[R_{i}-\operatorname{mean}\big(\{R_{j}\}_{\tau_{j}\in\mathcal{G}_{\text{on}}}\big)\big]_{+}}{R_{\text{range}}},\quad\tau\in\mathcal{G}_{\text{mix}}.$ (11) $$

where $[x]+=\max(x,0)$ ensures that only above-average exploration is rewarded while failures are ignored, and $R_{\text{range}}$ is a fixed reward span (e.g., the global maximum–minimum reward range) used to normalize rewards into $[0,1]$ for numerical stability. And $\alpha$ is a weighting coefficient that balances the contribution of samples from the mixed-policy. In our setting, $\alpha$ is additionally scheduled to gradually decay, thereby shifting the policy from expert-guided exploration to self-driven exploration as training progresses.

## 4 Experiments

### 4.1 Setup

#### Datasets and Models.

We conduct experiments on two model families: Qwen2.5 (Team, 2024) and LLaMA3.1 (Dubey et al., 2024). For Qwen2.5, we use the Qwen2.5-7B-Base and Qwen2.5-3B-Base for experiments. And we use the MATH dataset (Hendrycks et al.,) as training dataset, restricting to problems with difficulty levels 3–5 and removing any instances overlapping with the test set to prevent data leakage, total 8,889 training examples. For LLaMA3.1, we use the LLaMA3.1-8B-Base for experiments. However, the MATH dataset is too difficult for this model, such that vanilla GRPO fails to train successfully. To enable comparison between GRPO and other baselines, we construct a simplified dataset from OpenR1-MATH-220K(^2^22https://huggingface.co/datasets/open-r1/OpenR1-Math-220k) (Hugging Face, 2025) as the training dataset for LLaMA3.1. Further dataset and expert model details are provided in the Appendix [B](https://arxiv.org/html/2510.04140v1#A2.SS0.SSS0.Px3).

#### Evaluations.

We evaluate the models along two categories. (i) In-domain performance. We assess the in-domain performance on mathematics benchmarks, including MATH (Hendrycks et al.,), AIME24, AIME25, and AMC (Li et al., 2024). (ii) Out-of-domain performance. To examine whether post-tuning affects general reasoning ability beyond mathematics, we further evaluate the out-of-domain performance in MMLU-Pro (Wang et al., 2024) and GPQA-diamond (Rein et al.,). For AIME24, AIME25, and AMC, we report avg@32 at temperature 0.6 as the test set is relatively small, while for the other benchmarks, we report pass@1 at temperature 0.

#### Baselines.

We compare MENTOR with several representative baselines, including: (1) Base: The base model without any fine-tuning. (2) On-policy RL: Standard GRPO without expert guidance, enhanced with token-level loss and the Clip-Higher in DAPO (Yu et al., 2025) to serve as a stronger baseline. (3) LUFFY (Yan et al., 2025): A method that integrates full expert trajectories within the GRPO rollout groups. (4) QuestA
(Li et al., 2025): (4) QuestA (Li et al., 2025): A method that provides the first half of expert trajectories as hints for the model to follow. Hyper-parameters and training details of different methods can be found in Appendix [B](https://arxiv.org/html/2510.04140v1#A2.SS0.SSS0.Px5).

### 4.2 Main Results

**Table 1: MENTOR vs. other baselines. Compared to the On-policy RL, MENTOR achieves an average performance improvement of 3.2%, 4.3% and 3.9% on the three models, respectively. The best results are highlighted in bold, and the second-best results are underlined.**
| Methods | In-Domain Performance | Out-of-Domain | Avg |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MATH | AIME24 | AIME25 | AMC | Minerva | Olympiad | GPQA | ARC | MMLU-Pro |  |  |
| LLaMa3.1-8B-Base |  |  |  |  |  |  |  |  |  |  |
| Base | 10.6 | 0.1 | 0.0 | 1.8 | 4.4 | 2.1 | 0.0 | 0.0 | 0.1 | 2.1 |
| On-policy RL | 24.0 | 0.4 | 0.4 | 8.0 | 13.6 | 6.4 | 25.8 | 70.7 | 35.7 | 20.6 |
| LUFFY | 25.2 | 0.5 | 0.4 | 8.4 | 14.0 | 7.1 | 27.8 | 74.9 | 34.9 | 21.5 |
| QuestA | 20.6 | 0.1 | 0.2 | 5.3 | 8.8 | 4.0 | 25.3 | 72.5 | 33.9 | 19.0 |
| MENTOR | 30.2 | 1.2 | 0.6 | 10.4 | 16.2 | 8.9 | 30.3 | 77.3 | 39.1 | 23.8 |
| Qwen2.5-3B-Base |  |  |  |  |  |  |  |  |  |  |
| Base | 47.4 | 2.4 | 1.9 | 17.7 | 19.9 | 19.0 | 3.0 | 23.6 | 19.4 | 17.1 |
| On-policy RL | 65.8 | 3.3 | 2.5 | 32.2 | 25.4 | 29.8 | 17.7 | 72.1 | 30.6 | 31.0 |
| LUFFY | 64.0 | 5.2 | 4.2 | 32.8 | 25.0 | 30.1 | 15.2 | 72.5 | 30.8 | 31.1 |
| QuestA | 66.4 | 7.9 | 2.9 | 34.1 | 27.6 | 29.8 | 16.2 | 70.3 | 30.9 | 31.8 |
| MENTOR | 69.8 | 8.3 | 3.8 | 34.2 | 26.5 | 35.2 | 22.7 | 80.8 | 36.8 | 35.3 |
| Qwen2.5-7B-Base |  |  |  |  |  |  |  |  |  |  |
| Base | 62.4 | 5.4 | 2.9 | 26.5 | 16.9 | 28.9 | 11.1 | 70.4 | 42.9 | 29.7 |
| On-policy RL | 76.8 | 14.2 | 9.1 | 46.0 | 34.2 | 41.5 | 29.3 | 86.0 | 48.0 | 42.8 |
| LUFFY | 77.0 | 12.9 | 10.4 | 46.4 | 35.3 | 40.8 | 26.8 | 86.0 | 49.7 | 42.8 |
| QuestA | 78.8 | 14.6 | 13.3 | 47.4 | 33.5 | 41.5 | 30.3 | 86.7 | 51.0 | 44.1 |
| MENTOR | 81.4 | 18.3 | 16.5 | 53.1 | 34.9 | 45.2 | 30.8 | 89.6 | 50.2 | 46.7 |

#### MENTOR achieves consistent improvements across different models.

Table [1](https://arxiv.org/html/2510.04140v1#S4.T1) shows that MENTOR outperforms the on-policy RL baseline across all three backbones. On Qwen2.5-7B, for example, MENTOR lifts the average score on the MATH benchmark from 76.8 to 81.4, and yields notable relative gains of +4.1, +7.4, and +7.1 points on AIME24, AIME25, and AMC, respectively. Similar trends are observed on Qwen2.5-3B and LLaMa3.1-8B. Importantly, these gains are not confined to in-domain reasoning. MENTOR also delivers clear improvements on out-of-domain benchmarks, demonstrating that the reasoning abilities learned under expert guidance can effectively generalize to out-of-domain tasks.

#### MENTOR achieves a better trade-off between expert guidance and autonomous exploration.

Compared to on-policy RL, LUFFY introduces full expert trajectories but achieves only limited improvements across all models, indicating that directly imitating expert solutions does not fully leverage expert knowledge. This is likely because full trajectories overly constrain the exploration space, causing the model to overfit superficial expert patterns and fall into suboptimal strategies. QuestA, which provides partial expert trajectories as hints, alleviates over-imitation to some extent but its effectiveness strongly depends on model capacity: it yields clear gains (+1.3) on Qwen2.5-7B, only minor improvement (+0.8) on Qwen2.5-3B, and even a negative effect (-1.6) on LLaMa3.1-8B. This is because, in the absence of subsequent guidance, the weaker model struggles to explore correct solutions, and the excessive hints further disrupt its exploration. In contrast, MENTOR consistently outperforms across different models, achieving a better balance between leveraging expert knowledge and maintaining autonomous exploration, thereby achieving significant improvements.

### 4.3 Training dynamics

Figure: Figure 2: Training dynamics of MENTOR compared with On-policy RL. MENTOR mitigates entropy collapse, and its response length dynamics reflect a shift from learning to understanding, thereby achieving higher performance.
Refer to caption: x2.png

#### Entropy dynamics.

Figure [2](https://arxiv.org/html/2510.04140v1#S4.F2) compares the training dynamics of On-policy RL and MENTOR in terms of validation accuracy, entropy and response length. Under On-policy RL, entropy collapses rapidly, indicating that the support of the policy exploration space shrinks prematurely to a narrow subset of trajectories. MENTOR enhances exploration diversity through selective expert guidance, thereby slowing down entropy collapse and enabling more persistent exploration throughout training. More importantly, the entropy eventually converges to a slightly higher level than On-policy RL, indicating that the final support set discussed in Section [2](https://arxiv.org/html/2510.04140v1#S2) is expanded, which directly translates into stronger final performance.

#### Response Length dynamics.

In the early training stage, MENTOR’s responses grow in length compared with GRPO. By analyzing rollout samples during training, we find that this rapid growth stems from adopting expert-style reasoning forks such as verify and wait, the occurrence of which extends the reasoning chain. However, as training progresses, MENTOR’s response length gradually declines, consistent with the scheduled reduction of expert advantage. We find that the model starts to distinguish useful tokens (e.g., verify) from redundant ones (e.g., wait), reflecting a shift from expert-guided to self-driven exploration. Through this selective absorption, the model achieves a more efficient final reasoning pattern, as shown in Appendix [C](https://arxiv.org/html/2510.04140v1#A3).

### 4.4 The Analysis of Reasoning Pattern

To better understand the reasoning patterns induced by different training methods, Figure [3](https://arxiv.org/html/2510.04140v1#S4.F3) reports the occurrence rate of high-frequency reasoning tokens, defined as the proportion of trajectories in which the token appears at least once, computed from 500 trajectories on MATH500, which provides a more reliable perspective than individual cases. Detailed case studies are provided in Appendix [C](https://arxiv.org/html/2510.04140v1#A3).

Figure: Figure 3: The occurrence rate of high-frequency reasoning tokens under different training methods. MENTOR absorbs the essence of expert trajectories such as verify, while avoiding over-imitation of redundant tokens like okay or wait.
Refer to caption: x3.png

#### MENTOR achieves selective absorption of expert knowledge.

As shown in Figure [3](https://arxiv.org/html/2510.04140v1#S4.F3), although LUFFY successfully incorporate expert knowledge compared with on-policy RL, it tends to imitate indiscriminately. For example, it excessively adopts tokens such as okay and wait, which leads to overly redundant reasoning. In contrast, MENTOR exhibits a more selective learning process, adopting valuable reasoning tokens such as verify and check while avoiding preserving redundant ones. This selective learning shows that MENTOR goes beyond surface imitation, effectively absorbing the essence of expert guidance while discarding the redundancy, resulting in an efficient reasoning pattern.

### 4.5 The Analysis of Reasoning Diversity

Figure: Figure 4: Pass@32 performance of Qwen2.5-7B under different methods. MENTOR improves the model’s reasoning diversity beyond other baselines.
Refer to caption: x4.png

To further quantify the impact of different methods on reasoning diversity, we adopt pass@k as the evaluation metric, which is widely used to measure reasoning diversity (Song et al., 2025; Chen et al., 2025). As shown in Figure [4](https://arxiv.org/html/2510.04140v1#S4.F4), Pass@32 of On-policy RL stagnates or even declines compared to the Base model, as it can only reshape behaviors within the original capability, resulting in reduced reasoning diversity. By introducing external expert trajectories, LUFFY and QuestA expand the model’s capability boundary and raise pass@k. However, these methods are limited in achieving further improvements in reasoning diversity due to excessive imitation. In contrast, by balancing expert guidance with autonomous exploration, MENTOR achieves a 9.2% average gain in pass@32, indicating a clear enhancement in reasoning diversity.

## 5 Related Work

#### Reinforcement Learning for Large Language Models

Reinforcement learning has recently made significant progress in enhancing the reasoning abilities of LLMs (Jaech et al., 2024; Guo et al., 2025; Team et al., 2025). A central development is Reinforcement Learning from Verifiable Rewards (RLVR), which replaces human feedback signals (Kirk et al., 2024) with automatically checkable objectives such as mathematical verification (Shao et al., 2024) and program execution (Pennino et al., 2025). However, studies also reveal that the gains of RLVR are closely tied to the capability of the base model. For instance, DeepSeek-R1 reports that while RLVR yields remarkable improvements for powerful base models, its benefits become much less pronounced when applied to models with more limited capacity (Guo et al., 2025).

#### On-Policy Learning under Expert Guidance

To improve the effectiveness of RLVR, a line of work incorporates expert trajectories into on-policy RL training. Some approaches directly mix entire expert rollouts with policy rollouts (Yan et al., 2025; Zhang et al., 2025a), while others provide partial prefixes of expert trajectories as hints for continued generation (Liu et al., 2025; Zhang et al., 2025b; Li et al., 2025). These strategies have proven effective in reducing unproductive exploration and stabilizing training. However, imitation of fixed expert trajectories restricts exploration, accelerates entropy collapse (Yan et al., 2025), and ultimately undermines the diversity of reasoning trajectories. In addition, the reduction of diversity is further accelerated by gradient imbalance (Huang et al., 2025), which drives the model to quickly overfit expert trajectories, especially when their reasoning patterns diverge substantially from those of the policy model (Zhang et al., 2025a). Although token-level reweighting has been proposed to alleviate this issue (Yan et al., 2025; Zhang et al., 2025a), the fundamental limitation remains: the exploration is still constrained by the fixed expert trajectories.

## 6 Conclusion

In this paper, we introduced MENTOR, a powerful framework that enables effective and diverse exploration through selective expert guidance at critical decision points. MENTOR avoids superficial imitation and allows policy model to internalize the essence of expert reasoning strategies. Across challenging benchmarks, our method consistently outperforms strong baselines and significantly improves pass@k performance on complex tasks. These results demonstrate the potential of selective expert guidance to enhance RLVR and suggest promising directions for future research, such as extending the framework to multimodal reasoning or investigating how expert guidance can be provided more effectively.

## References

- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Team et al. (2025)
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al.
Kimi k1. 5: Scaling reinforcement learning with llms.
*arXiv preprint arXiv:2501.12599*, 2025.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
s1: Simple test-time scaling.
*arXiv preprint arXiv:2501.19393*, 2025.
- Yue et al. (2025)
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
*arXiv preprint arXiv:2504.13837*, 2025.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Song et al. (2025)
Yuda Song, Julia Kempe, and Remi Munos.
Outcome-based exploration for llm reasoning.
*arXiv preprint arXiv:2509.06941*, 2025.
- Cui et al. (2025)
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al.
The entropy mechanism of reinforcement learning for reasoning language models.
*arXiv preprint arXiv:2505.22617*, 2025.
- Yan et al. (2025)
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang.
Learning to reason under off-policy guidance.
*arXiv preprint arXiv:2504.14945*, 2025.
- Zhang et al. (2025a)
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou.
On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting.
*arXiv preprint arXiv:2508.11408*, 2025a.
- Zhang et al. (2025b)
Xuechen Zhang, Zijian Huang, Yingcong Li, Chenshun Ni, Jiasi Chen, and Samet Oymak.
Bread: Branched rollouts from expert anchors bridge sft & rl for reasoning.
*arXiv preprint arXiv:2506.17211*, 2025b.
- Liu et al. (2025)
Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu.
Ghpo: Adaptive guidance for stable and efficient llm reinforcement learning.
*arXiv preprint arXiv:2507.10628*, 2025.
- Li et al. (2025)
Jiazheng Li, Hong Lu, Kaiyue Wen, Zaiwen Yang, Jiaxuan Gao, Hongzhou Lin, Yi Wu, and Jingzhao Zhang.
Questa: Expanding reasoning capacity in llms via question augmentation.
*arXiv preprint arXiv:2507.13266*, 2025.
- Huang et al. (2025)
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov.
Blending supervised and reinforcement fine-tuning with prefix sampling.
*arXiv preprint arXiv:2507.01679*, 2025.
- Wang et al. (2025)
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al.
Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.
*arXiv preprint arXiv:2506.01939*, 2025.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Wu et al. (2025)
Zongqian Wu, Baoduo Xu, Tianyu Li, Zhu Sun, Xiaofeng Zhu, and Lei Feng.
Mitigating strategy-selection bias in reasoning for more effective test-time scaling.
*arXiv preprint arXiv:2509.17905*, 2025.
- Du et al. (2024)
Chengyu Du, Jinyi Han, Yizhou Ying, Aili Chen, Qianyu He, Haokun Zhao, Sirui Xia, Haoran Guo, Jiaqing Liang, Zulong Chen, et al.
Think thrice before you act: Progressive thought refinement in large language models.
*arXiv preprint arXiv:2410.13413*, 2024.
- Chen et al. (2023)
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper.
Accelerating large language model decoding with speculative sampling.
*arXiv preprint arXiv:2302.01318*, 2023.
- Team (2024)
Qwen Team.
Qwen2.5: A party of foundation models, September 2024.
URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/).
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pages arXiv–2407, 2024.
- (23)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
In *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)*.
- Hugging Face (2025)
Hugging Face.
Open r1: A fully open reproduction of deepseek-r1, January 2025.
URL [https://github.com/huggingface/open-r1](https://github.com/huggingface/open-r1).
- Li et al. (2024)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al.
Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.
*Hugging Face repository*, 13(9):9, 2024.
- Wang et al. (2024)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
*Advances in Neural Information Processing Systems*, 37:95266–95290, 2024.
- (27)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
In *First Conference on Language Modeling*.
- Chen et al. (2025)
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi.
Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models.
*arXiv preprint arXiv:2508.10751*, 2025.
- Kirk et al. (2024)
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu.
Understanding the effects of RLHF on LLM generalisation and diversity.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=PXD3FAVHJT](https://openreview.net/forum?id=PXD3FAVHJT).
- Pennino et al. (2025)
Federico Pennino, Bianca Raimondi, Massimo Rondelli, Andrea Gurioli, and Maurizio Gabbrielli.
From reasoning to code: Grpo optimization for underrepresented languages.
*arXiv preprint arXiv:2506.11027*, 2025.
- Jaynes (1957)
Edwin T Jaynes.
Information theory and statistical mechanics.
*Physical review*, 106(4):620, 1957.
- Zheng et al. (2025)
Yaowei Zheng, Shenzhi Wang Junting Lu, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong.
Easyr1: An efficient, scalable, multi-modality rl training framework.
[https://github.com/hiyouga/EasyR1](https://github.com/hiyouga/EasyR1), 2025.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.

## Appendix

## Appendix A Theoretical Proof

### A.1 Support Contraction to 𝒯 ⋆ \mathcal{T}^{\star}

We now provide a short proof that, under the reward-maximization objective in Eq.([2](https://arxiv.org/html/2510.04140v1#S2.E2)), the optimal distribution places all probability mass on $\mathcal{T}^{\star}$.

For a fixed question $q$, write $R(\tau)\equiv R(q,\tau)$ on the discrete set $\mathcal{S}_{q}=\operatorname{supp}(\pi_{\theta}(\cdot\mid q))$. Let $R_{\max}=\sup_{\tau\in\mathcal{S}_{q}}R(\tau)$ and denote the set of maximizers by $\mathcal{T}^{\star}=\{\tau\in\mathcal{S}_{q}:R(\tau)=R_{\max}\}$.

Since the learning objective in Eq.([2](https://arxiv.org/html/2510.04140v1#S2.E2)) is to maximize expected reward but the exact optimal distribution is unknown, we adopt a Maximum Entropy Principle (Jaynes, 1957). Specifically, we optimize over all probability mass functions $P:\mathcal{S}_{q}\to[0,1]$ with $\sum_{\tau\in\mathcal{S}_{q}}P(\tau)=1$:

$$ $\max_{P}\;H(P)\quad\text{s.t.}\quad\sum_{\tau\in\mathcal{S}_{q}}P(\tau)R(\tau)=C,\ \ \sum_{\tau\in\mathcal{S}_{q}}P(\tau)=1,$ (12) $$

where $H(P)=-\sum_{\tau\in\mathcal{S}_{q}}P(\tau)\log P(\tau)$ and $C$ is the target expected reward. A standard Lagrangian calculation yields the unique Gibbs-form solution

$$ $P_{\lambda}(\tau)=\frac{\exp\{\lambda R(\tau)\}}{Z(\lambda)},\qquad Z(\lambda)=\sum_{\tau^{\prime}\in\mathcal{S}_{q}}\exp\{\lambda R(\tau^{\prime})\},$ (13) $$

for some multiplier $\lambda>0$ chosen such that $\mathbb{E}_{P_{\lambda}}[R]=C$.

Define $\phi(\lambda)=\sum_{\tau}P_{\lambda}(\tau)R(\tau)$. Then $\phi^{\prime}(\lambda)=\operatorname{Var}_{P_{\lambda}}[R]\geq 0$, so $\phi(\lambda)$ is non-decreasing. Moreover, $\lim_{\lambda\to\infty}\phi(\lambda)=R_{\max}$. Hence as $C\uparrow R_{\max}$, we must have $\lambda\to\infty$, and for any $\tau\notin\mathcal{T}^{\star}$ and $\tau^{\star}\in\mathcal{T}^{\star}$,

$$ $\frac{P_{\lambda}(\tau)}{P_{\lambda}(\tau^{\star})}=\exp\{-\lambda\,(R_{\max}-R(\tau))\}\ \longrightarrow\ 0\quad(\lambda\to\infty).$ (14) $$

Thus all probability mass concentrates on $\mathcal{T}^{\star}$ in the limit.

### A.2 Proof of Unbiasedness for Mixed-Policy Rollout

The unbiasedness of speculative sampling is well established in prior work. For completeness, we include a concise proof specialized to our mixed policy $\pi_{\text{mix}}$, confirming that the validation procedure remains unbiased in our setting.

Let the token space be $\mathcal{V}$, and fix a prefix $(q,y_{<t})$ at step $t$. Denote the base policy by

$$ $p_{t}(\cdot)=\pi_{\theta}(\cdot\mid q,y_{<t}),$ $$

and let $s_{t}(\cdot)=\pi^{*}(\cdot\mid q,y_{<t})$ be the expert policy. The mixed policy is obtained by a deterministic ensemble of $(p_{t},s_{t})$,

$$ $q_{t}(\cdot)=\pi_{\text{mix}}(\cdot\mid q,y_{<t})=\mathcal{M}\!\big(p_{t}(\cdot),s_{t}(\cdot)\big),$ $$

where $\mathcal{M}$ denotes any tokenwise mixing operator that yields a valid distribution on $\mathcal{V}$ (e.g., convex mixing). The validation procedure only depends on $q_{t}$.

At step $t$, a candidate token $\tilde{y}_{t}$ is first sampled from $p_{t}$. It is accepted with probability

$$ $\alpha_{t}(\tilde{y}_{t})=\min\!\Big(1,\frac{q_{t}(\tilde{y}_{t})}{p_{t}(\tilde{y}_{t})}\Big),$ $$

If rejection occurs, a new token is drawn from the residual distribution on $\mathcal{V}$, defined for the dummy variable $z\in\mathcal{V}$ by

$$ $r_{t}(z)=\frac{(q_{t}(z)-p_{t}(z))_{+}}{\sum_{z^{\prime}\in\mathcal{V}}(q_{t}(z^{\prime})-p_{t}(z^{\prime}))_{+}},\qquad(u)_{+}=\max\{u,0\}.$ $$

For any possible token $v\in\mathcal{V}$, the probability that it becomes the committed token is therefore

$$ $\mathbb{P}(y_{t}=v)=p_{t}(x)\min\!\Big(1,\tfrac{q_{t}(v)}{p_{t}(v)}\Big)+\mathbb{P}(\text{reject})\,r_{t}(v).$ $$

The first term equals $\min\{p_{t}(v),q_{t}(v)\}$. The rejection probability is

$$ $\mathbb{P}(\text{reject})=1-\sum_{z\in\mathcal{V}}p_{t}(z)\min\!\Big(1,\tfrac{q_{t}(z)}{p_{t}(z)}\Big)=1-\sum_{z\in\mathcal{V}}\min\{p_{t}(z),q_{t}(z)\}=\sum_{z\in\mathcal{V}}(q_{t}(z)-p_{t}(z))_{+},$ $$

which coincides with the denominator of $r_{t}(\cdot)$. Consequently, the second term contributes exactly $(q_{t}(v)-p_{t}(v))_{+}$. Combining the two contributions yields

$$ $\mathbb{P}(y_{t}=v)=\min\{p_{t}(v),q_{t}(v)\}+(q_{t}(v)-p_{t}(v))_{+}=q_{t}(v).$ $$

Thus the distribution of the validated token is exactly the mixed policy $q_{t}$.

To extend the result to entire speculative sequences, note that at $t=1$ the marginal distribution is $q_{1}$. Suppose inductively that the joint distribution of the prefix $y_{<t}$ is $\prod_{j<t}q_{j}(y_{j})$. Conditioning on such a prefix, the above calculation shows that $y_{t}\sim q_{t}(\cdot)$. Hence, by induction,

$$ $\mathbb{P}(y_{1:T}\mid q)=\prod_{t=1}^{T}q_{t}(y_{t})=\prod_{t=1}^{T}\pi_{\text{mix}}(y_{t}\mid q,y_{<t}),$ $$

which is identical to direct autoregressive sampling from the mixed policy.

## Appendix B Experimental Details

#### Platform.

All of our experiments are conducted on workstations equipped with eight NVIDIA A100 GPUs with 80GB memory, running Ubuntu 22.04.4 LTS and CUDA 12.4.

#### System Prompt.

All models trained under MENTOR and other baselines, except QuestA, share the same system prompt for both training and inference:

For QuestA, we additionally append “## Hint: Partial Solution” after the QUESTION as a hint section.

#### Reward Setting.

For outcome reward, we employ Math-Verify to automatically check whether the final answer inside the “<think>… </think>… \boxed{}” format matches the ground truth, assigning +1 if correct and 0 otherwise. In addition, we introduce a format reward that grants +1 when the response adheres to this format, and 0 if not. The same reward design is applied to MENTOR and all baselines to ensure fairness. For Qwen2.5-7B and Qwen2.5-3B, the weights of outcome reward and format reward are set to 9:1. For LLaMa3.1-8B, however, this ratio is adjusted to 8:2, since the original weighting did not sufficiently enforce format adherence.

#### Dataset Details.

For Qwen2.5-7B and Qwen2.5-3B, we use problems from the MATH dataset with difficulty levels 3–5, removing all instances that overlap with the test sets to avoid data leakage. This yields a total of 8,889 training examples. However, for LLaMA3.1-8B, this dataset is too difficult, making the vanilla GRPO algorithm hard to apply. To address this issue, we constructed an easier training set from OpenR1-Math-220K by selecting problems with response lengths shorter than 4K tokens, on which the model could be successfully trained using GRPO. All subsequent methods on LLaMA3.1-8B were trained using this simplified dataset. For each problem, the fixed expert trajectory used in LUFFY and QuestA is generated by DeepSeek-R1.

#### Export Model Details.

For Qwen2.5, We adopt OpenR1-Qwen-7B(^3^33https://huggingface.co/open-r1/OpenR1-Qwen-7B) as the expert model in MENTOR, which is trained on a distilled dataset generated by DeepSeek-R1. For LLaMA3.1, the expert model in MENTOR is obtained by further fine-tuning LLaMA3.1-8B-Instruct under the same dataset and setting used for OpenR1-Qwen-7B.

#### Training Details.

We conduct all experiments using the EasyR1(^4^44https://github.com/hiyouga/EasyR1) (Zheng et al., 2025) framework, which employs Verl (Sheng et al., 2024) as the RL training engine and vLLM (Kwon et al., 2023) as the rollout engine. The training setup includes a rollout batch of 128, a learning rate of $1\times 10^{-6}$, a generation temperature of 1.0, and a higher-clip of 0.28. Each response sequence is up to 8k tokens in length. We perform 8 rollouts per prompt and do not apply KL divergence or entropy regularization (KL Coeff = 0, entropy loss = 0). The mini-batch size is set to 64. For important parameters of MENTOR, $\alpha$ is initialized to 1 and annealed to 0 with a cosine schedule over 120 steps, enabling a smooth transition from expert guidance to autonomous exploration. The number of mixed-policy rollouts is set to 4. For $\gamma_{\text{p}}$, p is chosen as 0.95, corresponding to the 95-th percentile of token-level entropies within each batch. As a special case, $\gamma_{\text{p}}$ is initialized to 999 at the first step.

## Appendix C Case Study

To complement the aggregate analysis in Figure [3](https://arxiv.org/html/2510.04140v1#S4.F3), we provide representative trajectory-level cases in this section. These examples illustrate how different training methods influence the emergence of reasoning tokens such as verify, check, and wait. By examining full reasoning traces, we highlight qualitative differences between methods, offering additional context to the quantitative analysis in the main text.

We find that the incorporation of expert policy indeed shapes the final reasoning patterns of the models, with different methods leading to distinct outcomes.

- •
From a stylistic perspective, QuestA and MENTOR exhibit reasoning traces more aligned with on-policy RL, whereas LUFFY shows a closer resemblance to the expert’s style. This difference arises because LUFFY emphasizes direct imitation of expert demonstrations, often reproducing surface-level tokens such as wait and okay, while QuestA and MENTOR encourage the model to explore autonomously while using expert guidance, thereby preserving a degree of its own stylistic characteristics.
- •
From a content perspective, MENTOR and LUFFY successfully acquire specific reasoning modes from the expert, such as the use of verify, which are absent in the base model. In contrast, QuestA, despite leveraging expert policy, does not exhibit the adoption of verify. Our analysis suggests this limitation stems from the design choice of using only the first half of the expert trajectory as hints, while tokens like verify typically appear toward the end of trajectories, thereby restricting their adoption.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: What would happen if the token points for guidance were selected randomly or uniformly? 

How robust is the proposed method to the expert’s capability — for instance, if the expert is weaker, can the model filter out misleading guidance? 

In the baselines, LUFFY and QuestA belong to expert imitation. Are there any more dynamic or flexible expert guidance baselines for comparison?

If the authors can provide more detailed verification of the method’s effectiveness and robustness, I will increase my score.
soundness: 2
strengths: - Clear problem framing: Rollout's effectiveness and diversity are important for RL. Paper's section details description this aspect.
- Critical point guidance: By leveraging token entropy, providing expert guidance only at critical decision points, steers reasoning trajectories while preserving the policy’s own exploration, rather than entirely imitating expert.
- Empirical Results: Based on three backbone models, proposed method MENTOR compared to Vanilla GRPO, LUFFY and QuestA, demonstrate the method's effectiveness.
summary: This paper introduces MENTOR, a reinforcement learning framework that aims to enhance both the effectiveness and diversity of exploration in RLVR for LLMs. Inject expert guidance only at critical decision points (high-entropy tokens) rather than imitating entire expert trajectories (such as LUFFY). In-Domain and OOD performance evaluation (LLaMa3.1-8B-Base, Qwen2.5-3B-Base and Qwen2.5-7B-base) demonstrate MENTOR's effectiveness.
weaknesses: - Experiment: The Table 1's experiment appear insufficient, given that the paper’s pipeline involves multiple combinations of hyperparameters and strategies. 
- Critical point selection: The selection strategy for critical decision points could be elaborated in more detail — for example, why is it based on token-level entropy rather than other metrics (e.g., higher-level semantic)?
- Typos: Line 322 QuestA appear twice.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Could the authors provide a more detailed comparison between MENTOR and DAPO? Although DAPO may require additional time for resampling, it does not incur the extra GPU memory overhead associated with loading a separate expert model. In this respect, DAPO could be a more resource-efficient alternative to MENTOR.

2. Could the authors provide concrete case studies to better illustrate MENTOR's impact on sampling diversity? While Figure 3 presents keyword statistics, a more direct qualitative analysis or examples would offer clearer insight into the diversity achieved through MENTOR.

3. Could the authors provide an explanation about changing the selected expert model rather than OpenR1-Qwen-7B? How does performance change if the expert is weaker or from a different architecture/family?
soundness: 3
strengths: 1. The motivation of the paper is reasonable, while full expert guidance could lead to the reduction of sampling diversity, MENTOR implements a relaxed guidance that guides the sampling only on critical tokens, leading to the balance of training efficacy and sampling diversity.

2. The MENTOR also involves an accelerated components that address the efficiency problem of the expert-guidance method, which is also a unique contribution.

3. This paper does comprehensive experiments on both in-domain and out-of-domain benchmarks, showing the efficacy and generalizability of MENTOR. In addition, the analysis results of entropy show that MENTOR can preserve the generation diversity during training.
summary: This paper introduces MENTOR (Mixed-policy Expert Navigation for Token-level Optimization of Reasoning), a GRPO+ training method for improving the reasoning capabilities. By providing guidance on critical tokens during inference, MENTOR achieves effective improvement of reasoning while preserving the diversity of group-based sampling. Experiments on math and science benchmarks show that MENTOR consistently outperforms over baselines like GRPO, LUFFY, and QuestA.
weaknesses: 1. One of the main weaknesses is that the mechanism of "MENTOR only guides the decoding on critical tokens". This paper shows MENTOR's decoding strategy in Sec. 3.1, but does not compare this strategy with prior methods of fully expert-guided decoding. A more detailed introduction and explanation of the decoding strategy is required.

2. The motivation for using an expert-guided method is not entirely clear. If a powerful expert model is already available, it seems straightforward to use it directly for reasoning tasks. Why is it necessary to further train and distill its knowledge into a smaller model? I would appreciate a more detailed discussion on the advantages and rationale behind training a smaller model with expert guidance.

3. The expert-guided method costs more training resources compared with vanilla GRPO, since it requires loading a bigger expert model for decoding, which will occupy a large amount of GPU resources. Even though this paper proposes an accelerated strategy, the extra cost of GPU computation and memory can not be overlooked.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 3.1 Can the authors clarify how the “accelerated rollout” avoids computing π* at every step when wt is small? 

3.2 Did you test other mixture strategies (e.g., thresholded expert calls or adaptive gating) to validate that entropy weighting is optimal?

3.3 How sensitive are results to γp and α scheduling?

3.4 Can you provide more seeds for the results?

3.5 Could you report results with a non-zero KL coefficient to show how it affects stability and performance?  Also, please clarify the reasoning behind setting the KL term to 0 during MENTOR fine-tuning.

I am willing to increase my score if the authors adress my concerns/questions.
soundness: 3
strengths: 1.1 The motivation is clear and well-grounded. The paper directly tackles the exploration–effectiveness trade-off in RLVR.

1.2 The proposed PMix mechanism (Eq.6) is conceptually interesting, using entropy as a proxy for critical decision points.

1.3 Results are consistently strong across multiple model scales and benchmarks, showing stable gains.
summary: MENTOR addresses the challenge of balancing effectiveness and diversity in Reinforcement Learning with Verifiable Rewards (RLVR). The method mixes the policy and expert distributions at the token level using entropy-based weights, and introduces an “accelerated mixed-policy rollout” to improve sampling efficiency. 
Experiments on Qwen2.5 and LLaMA3.1 show consistent gains across in-domain and out-of-domain reasoning benchmarks.
weaknesses: 2.1 The “accelerating mixed-policy rollout” (Eq.7) appears inconsistent. Even with a small wt (e.g., 0.01), the expert π* must still be evaluated to compute the mixed distribution, so the method does not actually avoid the cost of expert inference. A gating mechanism for small weights would be necessary for genuine speed-up.

2.2 The claimed parallelization (L216–220) is unclear. Eq.7 explicitly depends on π*, so it cannot be fully parallelized in the autoregressive decoding process. The efficiency argument based on speculative sampling seems overstated.

2.3 Missing ablations for critical parameters (γp, α) and no exploration of alternative PMix strategies.

2.4 In Appendix A.2, the authors set the KL regularization coefficient to 0 during MENTOR fine-tuning. 
This is concerning, as the KL term is typically used to constrain the policy from drifting too far from the base model. Disabling it may weaken stability and allow distributional drift, which the paper does not quantify or justify empirically.

2.5 Results appear to be based on single runs, with no reported variance or standard deviations. This limits statistical reliability and makes it difficult to assess the robustness or significance to the provided results.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - Why do you still adopt trajectories from the current model in Eq. 9? Any insights here?
- In Eq. 11, you write $A_i = \dfrac{[R\_i - mean(R\_j)]\_+}{R\_{\rm range}}$ rather than $\dfrac{[R\_i - mean(R\_j)]\_+}{{\rm std}(R\_j)}$, why? Why only normalize the advantages from the mixed policy into [0, 1] but use Eq. 10 directly from GRPO?
- Can the proposed MENTOR still work when the base LLM is strong?
soundness: 2
strengths: - This paper is easy to read and easy to follow
- This paper focuses on an important topic in large language model reasoning, aiming at enhancing the exploration capabilities of existing LLMs via some specially designed approaches
- The authors provide codes in the main text, which enhances the reliability of the results reported in this paper
- The experiments are carried out using both LLaMa models and Qwen models with different model sizes, which shows the generality of the proposed method to some extent
summary: This paper proposes MENTOR, a reinforcement learning framework for large language models that enhances reasoning by providing expert guidance only at critical decision points rather than on entire reasoning paths. This approach promotes both effective and diverse exploration, enabling models to capture strategic essence rather than superficial imitation, which leads to superior performance compared to existing RLVR methods.
weaknesses: - (major) I do not see the necessity of Definitions 2.1 and 2.2. If the authors would like to claim that the optimal trajectory can be rarely covered, and hence raises two critical issues in the RLVR training, there is no need to state Definition 2.1 and 2.2 since the readers can generally get that. The organization of this paper can be improved
- (major) No ablation study and parameter study can be found either in the main text or the appendix. This makes it hard to figure out which component (mixed-policy rollout and mixed-policy GRPO) contributes most to the performance of MENTOR. It is also hard to check how sensitive MENTOR is to the introduced hyperparameters, e.g., the initial value of $\alpha$ in Eq. 11, how different decay schedules of $\alpha$ affect the performance, how to set $p$ in $\gamma_p$, etc.
- (major) Despite the fact that the authors propose a method to accelerate mixed-policy rollout, the entire process of MENTOR is still both time-consuming and memory-consuming compared to other methods like QuestA. The authors ought to show the training efficiency of MENTOR against baseline methods
- (major) It is somewhat strange to me that the authors use different expert models for different base LLMs, e.g., OpenR1-Qwen-7B model for Qwen models and finetuned LLaMA3.1-8B-Instruct model for LLaMa models. Is there any reason that you do not use a fixed expert model? Why do you use the Instruct model for LLaMa here? It is unclear whether the proposed method would fail if one adopts other LLMs as the base model, e.g., deepseek-R1 as the expert model. 
- (major) It is also unclear whether the proposed method still works if the expert model is only marginally stronger than the base model, or if the expert model is too strong (e.g., Qwen3-max), and whether the proposed method can still work. Note that QuestA conducts experiments using some strong base LLMs. The authors commented that the advantages of QuestA rely heavily on the vanilla performance of the base models, while it raises questions on whether MENTOR can work if it is adopted upon some strong base models (e.g.,  Nemotron-1.5B)
- (minor) The related work part can be significantly improved. LLM reasoning and LLM reasoning under guidance are fast-growing research areas, and there are numerous papers that do similar things. The authors should include more discussion in the manuscript
- (minor) No full pseudo-code for MENTOR can be found. Algorithm 1 just states part of the MENTOR algorithm
- (minor) There are numerous minor issues in the manuscript. I just list some of them below, please double-check your manuscript,
  - Line 019, *we argue that the expert only needs to provide guidance only at critical*, two *only* in this sentence
  - Line 322, two QuestA


Decision: Accept (Poster)