Title: R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

efficient reasoning

long context
Abstract: Large Language Models (LLMs) achieve impressive reasoning capabilities at the cost of substantial inference overhead, posing substantial deployment challenges. Although distilled Small Language Models (SLMs) significantly enhance efficiency, their performance suffers as they fail to follow LLMs' reasoning paths. Luckily, we reveal that only a small fraction of tokens genuinely diverge reasoning paths between LLMs and SLMs. Most generated tokens are either identical or exhibit neutral differences, such as minor variations in abbreviations or expressions. Leveraging this insight, we introduce **Roads to Rome (R2R)**, a neural token router that selectively utilizes LLMs only for these critical, path-divergent tokens, while leaving the majority of token generation to the SLM. We also develop an automatic data generation pipeline that identifies divergent tokens and generates token-level routing labels to train the lightweight router. We apply R2R to combine R1-1.5B and R1-32B models from the DeepSeek family, and evaluate on challenging math, coding, and QA benchmarks. With an average activated parameter size of 5.6B, R2R surpasses the average accuracy of R1-7B by 1.6×, outperforming even the R1-14B model. Compared to R1-32B, it delivers a 2.8× wall-clock speedup with comparable performance, advancing the Pareto frontier of test-time scaling efficiency.

R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing

Abstract

Large Language Models (LLMs) achieve impressive reasoning capabilities at the cost of substantial inference overhead, posing substantial deployment challenges. Although distilled Small Language Models (SLMs) significantly enhance efficiency, their performance suffers as they fail to follow LLMs' reasoning paths. Luckily, we reveal that only a small fraction of tokens genuinely diverge reasoning paths between LLMs and SLMs. Most generated tokens are either identical or exhibit neutral differences, such as minor variations in abbreviations or expressions. Leveraging this insight, we introduce Roads to Rome (R2R) , a neural token routing method that selectively utilizes LLMs only for these critical, path-divergent tokens, while leaving the majority of token generation to the SLM. We also develop an automatic data generation pipeline that identifies divergent tokens and generates token-level routing labels to train the lightweight router. We apply R2R to combine R1-1.5B and R1-32B models from the DeepSeek family, and evaluate on challenging math, coding, and QA benchmarks. With an average activated parameter size of 5.6B, R2R surpasses the average accuracy of R1-7B by 1.6 × , outperforming even the R1-14B model. Compared to R1-32B, it delivers a 2.8 × wall-clock speedup with comparable performance, advancing the Pareto frontier of test-time scaling efficiency. Our code is available at https://github.com/thu-nics/R2R .

1 Introduction

Large Language Models (LLMs) demonstrate strong capabilities across a wide range of tasks [1-3]. Building upon the largest and strongest LLMs, test-time scaling has become a prominent way to further boost their abilities on challenging tasks [4-7]. It is typically done by generating extensive Chain-of-Thought (CoT) reasoning before producing the final answer. However, this approach requires massive LLMs with hundreds of billions of parameters to generate thousands of tokens per query [8], resulting in significant inference overhead [9].

Distilled Small Language Models (SLMs), containing only a few billion parameters, offer much higher generation efficiency. Through supervised finetuning on LLM responses, SLMs can mimic LLM reasoning behaviors, making them a popular alternative. However, SLMs may still produce different reasoning paths from their LLM counterparts during inference, causing severe performance degradation. For example, compared to the R1-32B LLM, the R1-1.5B SLM provides different final answers on 45% questions in the AIME benchmark [10], suffering a 4.8 × reduction in accuracy as shown in Table 2.

9999

2

is

hard

, rewrite it...

Okey, let

's

think step by step...

Compute 9999

2

- 9998

×

1000.

question:

LLM:

9999

2

is

99

9801...

Okey, let

us

think step by step...

neutral

divergent

identical

✔

✔

✘

89%

5%

6%

11%

different

identical

neutral

divergent

(a)

(b)

Fortunately, we find that SLMs and LLMs often agree on next-token predictions given the same context. Instead, large performance gaps between them primarily arise from cumulative errors: their reasoning paths increasingly drift apart after some crucial differences in partial responses. To investigate this, we treat each step's LLM partial response as a prefix context and assess whether the next-token prediction from SLM is identical to LLM (Figure 1(a)). Across 2,094 queries totaling 7.6M tokens generated by the 32B LLM, the 1.5B SLM differs on only 11% of tokens-far less frequent than differences observed in final answers. Moreover, some of these differences are merely neutral variations, such as abbreviations or alternative expressions (e.g., let's vs. let us ), which do not affect reasoning outcomes. The key drifts start from a subset of different tokens, which we call divergent tokens. These tokens genuinely alter the meaning, logic, or conclusion of the current sentence, thus diverging the subsequent reasoning path. This observation motivates us to selectively use SLM and LLM for different generation steps. It naturally leads to a critical research question:

Can SLMs follow LLM reasoning paths by replacing only divergent tokens?

If addressed, we could unlock substantial efficiency advantage of SLMs for most generation steps, yet preserving the high-quality reasoning typical of LLMs. This can enable better test-time scaling by advancing the efficiency-performance Pareto frontier.

The main challenge of SLM-LLM mix inference involves two interconnected parts: labeling the preferred model under certain objective, and designing the routing scheme to enforce it during inference. Previous methods typically route at the query level, selecting either SLM or LLM for entire response to maximize human preference win-rate within a cost budget [11, 12]. However, these approaches rely on human annotations and complex router designs, whose data labeling and routing scheme are both too expensive for fine-grained, token-level routing. Alternatively, speculative decoding methods aim for identical outputs between SLM and LLM at the token level [13-16]. They draft outputs with SLMs (or draft models) and periodically verify them with LLMs. While accurate, this strict verification leads to low acceptance rates. Additionally, mid-draft differences invalidate all subsequent tokens, severely restricting the accepted lengths as shown in Figure 2(b).

To address these challenges, we propose Roads to Rome (R2R) , a token-level routing method that selectively utilizes LLMs only for path-divergent tokens during SLM generation. We begin by automating token-level model preference labeling under a path-following objective. Starting from the LLM's reasoning paths, we identify different predictions for SLM and LLM, briefly continue generation from the point of difference, then use another LLM as verifier to determine whether the difference is truly divergent or merely a neutral variation. This labeling approach minimizes the lower bound of LLM usage by allowing neutral SLM-LLM differences. Using the resulting labeled dataset, we train a lightweight neural router to predict and immediately route divergent SLM tokens to the LLM for correction. We further improve routing accuracy by identifying predictive indicators of divergence such as SLM uncertainty and token rarity, available directly during SLM inference. Our contributions are summarized as follows.

Data Labeling Pipeline. We develop an automatic pipeline to label divergent tokens. We formalize the global token-routing optimization problem, then propose a path-following strategy to generate routing labels with highly parallel, local decisions. We validate that SLM can effectively match LLM reasoning quality by following these routing labels.

Token-Router Design. We introduce a token-level routing scheme using a lightweight neural router. We investigate SLM outputs that aid accurate token routing and incorporate them into the router, enabling immediate and more accurate routing of divergent tokens.

LLM:

output:

It's 99

It's 9998

It's hard

It's 9998

✘

hard

It's 999801

It's 999801

✘

It's 99

It's hard,

It's hard

✔

It's

hard

It's hard, re

re

✔

input:

It's

(a)

(b)

,

It's hard, rewrite

write

✔

It's 999801.

It's 999801.

✘

Performance-Efficiency Pareto Frontier. R2R enables more efficient performance scaling at test time. Compared to query-level routing and distilled R1-14B, it delivers 1.1-1.5 × higher AIME accuracy with 1.5-1.6 × lower latency. R2R also provides a 2.8 × wall-clock speedup over R1-32B LLM at similar accuracy, while raising R1-1.5B SLM accuracy by 4.6 × with only 12.9% LLM usage.

2 Related Work

Test-time scaling improves LLM performance at the higher cost of inference, often through the explicit generation of the CoT reasoning paths [4, 6]. For more effective scaling, previous works optimize the length and width of reasoning paths, or reduce the generation overhead of each path [17, 18].

Controlling reasoning paths . Some approaches reduce LLM output lengths. They employ prompting [19], post-training [20], or heuristics [21] to generate concise CoT with fewer decoding steps [19, 21]. Others explore the width of paths. They let LLMs generate multiple reasoning paths in parallel, then select the best outcome with methods like best-of-N voting [22] or external verifiers [23]. Both strategies modify the structure of reasoning paths, which are perpendicular to R2R's focus on reducing the overhead of each path.

Model routing . Model routing reduces generation cost by selecting the most suitable model for each query based on difficulty and budget. Current works explore selection criteria of learned human preferences [11], reward signals [24], query tags [25], and model profiles [12]. Despite simplicity, they enforce the same LLM for each response, yielding suboptimal performance for the common mixed-difficulty generations. In contrast, R2R routes at token level to further improve efficiency.

Speculative decoding . Speculative decoding accelerates generation by fixing the low parallelism in LLM decoding [13-16]. It drafts outputs through SLM sequential decoding, then periodically verifies them with high-parallel LLM prefilling. However, speculative decoding pursues identical output token (distribution) between SLM and LLM, causing low acceptance rate. In addition, it is often that not all tokens generated by SLM within one draft-verify cycle can pass LLM verification. The mid-draft rejection invalidates all subsequent drafts and LLM verifications as shown in Figure 2(b), leading to frequent rollbacks. Expanding the single draft-chain to draft-tree alleviates the problem, but also incurs higher overheads that harm batch serving efficiency [16]. Considering the internality of CoT process, R2R accepts neutrally different output tokens, and immediately corrects all divergent tokens to avoid any rollback.

3 Model Preference Labeling

In Section 3.1, we formalize the token-level routing problem, aiming to minimize generation cost without sacrificing response quality. In Section 3.2, we introduce the path-following strategy for this problem, which assigns model preference labels to each output token, and empirically validate its effectiveness.

3.1 Token-level Routing Formulation

For autoregressive language models, reasoning can be represented as a sequence of next-token predictions. Throughout this paper, we focus on greedy sampling for simplicity:

Here, x i and y i denote the input and output tokens, respectively. For notational simplicity, we define the token sequence at step i as S <i = [ x 0 , . . . , x n -1 , y 0 , . . . , y i -1 ] , where S < 0 is the input tokens. The next-token probability P m i is predicted by model m i ∈ { θ s , θ l } at step i , where θ s and θ l denote the SLM and LLM, respectively.

The essence of the routing strategy is to define a routing function R that selects the model for each decoding step:

Our objective is to minimize the total generation cost while ensuring that the output sequence matches the quality of LLM-only outputs. We define the cost C as the sum of activated model parameters per token over the entire generation process. The quality of a response is evaluated by task-specific criteria, such as correctness for math problems, pass rates for coding tasks, or LLM-based grading for writing tasks. We define V as the verifier function, which returns 1 if and only if two sequences are of equivalent quality.

3.2 Path-following Routing Strategy

Optimally solving the token-level routing problem is computationally prohibitive, especially for large-scale data generation. While better routing sequences-potentially diverging from the LLM's reasoning path-may exist, finding them requires exhaustively searching a vast O (2 n ) space and generating thousands of output tokens for each search.

To overcome this practical limitation, we propose a greedy, sentence-level path-following routing strategy that reduces the search complexity to O ( n ) . Rather than exploring all possible model choices, our approach incrementally aligns mixed-model generation with the reasoning path established by the LLM. At each generation step, the strategy prefers the efficient SLM unless this would cause a meaningful divergence from the LLM's intended reasoning path, as determined by a continuationand-verification mechanism.

Specifically, at each step, we first compare the next-token predictions from the SLM and LLM. If the predictions are identical , we confidently select the SLM, as this does not affect the output sequence. When predictions differ, we must determine whether the difference is neutral or divergent . To do so, we construct two candidate sequences by appending predictions from SLM and LLM to the previous token sequence, respectively. Both sequences are then continued using the LLM until a stopping criterion is met (e.g., EOS token is generated). These continuations reveal how the initial token difference affects subsequent reasoning, measured under optimal yet achievable conditions (i.e., LLM-only continuation). If the first continuation still matches the quality of the second under the verifier function V , the difference is considered neutral ; otherwise, it is divergent and the token is routed to the LLM.

Equations 3-5 formalize the routing strategy. Here, y i ( m i | S <i ) indicates that this output token is generated by model m i given the previous sequence S <i , as a simplified expression of Equation 1. The continuation sequences, respectively generated after SLM and LLM token, are denoted by S s and S l . The operator ⊕ indicates concatenation of token sequences.

When continuation produces the full response by stopping only at the regular EOS token, we call this full path-following routing. By using the quality verifier from Section 3.1, the mixed-generated token sequence is guaranteed to achieve the same quality as its LLM-only counterpart, as it always remains on a path that could achieve LLM-only quality. The formal proof of this quality guarantee is provided in Appendix E. While the resulting model choice sequence M <i = [ m 0 . . . m i -1 ] can be used as labels for router training, full continuation is computationally expensive for large-scale data generation. In addition, the effect of current difference to the final output quality thousands of tokens away is too hard to learn for the neural router to be trained.

In practice, we use sentence-level path-following routing, where the continuation ends at the current sentence, as shown in Figure 3 (step 2). We monitor sentence-ending symbols, like period, during continuation and use existing semantical sentence separators [26, 27] to conclude generation if the sentence truly ends. To verify this local continuation, a capable LLM serves as a sentence-level verifier V ′ , as shown in Figure 3 (step 3). It is prompted to compare the continuations and determine whether the initial token difference introduces a meaningful divergence from the LLM's intended reasoning path, or merely a neutral abbreviation. Instead of verifying the entire generation, this approach checks the reasoning path at the sentence level, greatly improving data labeling efficiency.

We empirically validate the effectiveness of sentence-level path-following routing using Qwen2.572B [28] as the verifier model, with prompts detailed in Appendix F.1. Among 17 AIME-24 questions correctly solved by R1-32B within an 8K-token limit, our path-following strategy achieves comparable accuracy (16 questions correctly answered) while relying on the larger R1-32B model for only 3% of generated tokens.

By locally evaluating token choices through sentence-level path-following routing, we closely align mixed inference with the LLM's high-quality reasoning path, eliminating the prohibitive overhead of global evaluations. However, direct use of this strategy for real-time inference is impractical, as it relies on costly LLM continuation and verification. Instead, the local nature of our strategy simplifies routing decisions, creating an easier learning task for a neural router compared to global routing. We therefore design and train a lightweight neural router that efficiently approximates this strategy, relying solely on SLM outputs to determine when to use the LLM during inference.

4 Token-Level Neural Router

This section describes our methodology for constructing the neural router. Specifically, we detail how routing labels are generated for training using the sentence-level path-following strategy (Section 4.1), identify predictive SLM indicators for the router (Section 4.2), and outline the neural router's architecture along with its routing scheme for inference deployment (Section 4.3).

4.1 Training Data Generation

We use sentence-level path-following routing to generate training labels for the neural router, incorporating several optimizations to control data labeling overhead.

Figure 3 shows our data generation pipeline. Given queries from existing datasets, we first obtain the complete LLM response, either directly from the dataset or via batched LLM inference. Next, we use highly parallel SLM prefilling to efficiently identify tokens where the SLM prediction is identical to the LLM, allowing us to exclude about 90% of tokens from further processing. For the remaining 10% of differing tokens, we perform batched LLM continuations from each SLM prediction. To further improve efficiency, we apply prefix caching in current frameworks [29, 30] to reuse KV-Cache

step0:

LLM response

Compute 9999

2

- 9998

×

1000.

step1:

SLM prefill

Let

's

think

step

by

step

.

99

99

2

is

hard

,

re

write

it

Let

us

think

step

by

step

.

99

99

2

is

99

,

re

write

it

LLM continuation

query

Let

us

Let

's

think

step

by

step

.

99

99

2

is

99

step3:

verify

Let's think step by step

Verify

and

Let

us

think about it step by step

neutral

9999

2

is hard, rewrite it

Verify

and

9999

2

is

99

9801

divergent

think about it step by step.

9801.

LLM

output label

①

②

①

②

①

②

different

different

0

1

2

3

4

Entropy

0

20

40

60

Percentage (%)

Average:

0.45

Average:

1.70

Data Type

Divergent

Non-Divergent

10

6

10

5

10

4

10

3

10

2

Token Frequency (#Occurrence/#Total Token)

0

20

40

60

80

100

Divergence Rate (%)

Data Type

Math

Code

QA

computations for shared context prefixes across multiple continuations (e.g., everything preceding Let in Figure 3). Continuations for the corresponding LLM tokens, S ( θ l ) , are directly extracted from the pre-generated LLM response, eliminating redundant computation. Finally, the verifier model compares both continuations and label routing preference. Further analysis of the robustness of the verifier and the comparison to human experts are provided in Appendix B.1.

Using this pipeline, we efficiently generate 7.6 million routing labels in approximately 2.3 days on 8 A800 GPUs, covering topics of math, coding, and QA with queries from the Bespoke-Stratos [31] dataset. Table 1 summarizes the statistics of the generated training dataset.

4.2 Predictive Indicators of Divergence

We explore predictive indicators that can help identify divergent tokens. To enable immediate routing, we focus on indicators that can be acquired solely during the SLM's next-token predictions. The following analysis is based on 7.6 million tokens in our training set.

SLM logits . As shown in Figure 4(a), divergent tokens exhibit substantially higher entropy in the SLM's output logits, with a 3.8 × mean value over that of non-divergent tokens. We observe similar trends with other uncertainty measures [32], and concurrent work [33] also confirms such observation. These empirical results indicate that increased uncertainty in SLM predictions is strongly correlated with token divergence. Motivated by this, our router takes top-100 SLM logit values as one of its input features.

Token frequency . Figure 4(b) shows that low-frequency tokens in the dataset are more likely to be divergent. This likely arises from the long-tail token distribution in the training data, making rare tokens harder for SLMs to model effectively due to the limited capacity [34]. Given this insight, our router explicitly incorporates token-frequency biases by using the token embedding of as router inputs.

4.3 Router Design and Routing Scheme

Model architecture . Guided by insights in Section 4.2, we design the neural router as a lightweight, six-layer feed-forward network (FFN) with 56M parameters. It takes the SLM's output logits and tokenized embedding, along with its last-layer hidden states for additional semantic context. All inputs are linearly projected, concatenated, and fed into the FFN backbone. The router outputs a binary classification probability, indicating whether the current token diverges from the LLM's reasoning path. Full network architecture detail descriptions are in Appendix A.1.

Training scheme . We train the router with cross-entropy loss using the labeled data described in Section 4.1. To address class imbalance caused by the low divergence rate, we re-weight the loss inversely to class frequency. After training, we use the validation set to select the routing threshold that meets the user-defined LLM usage rate. The full training details are provided in Appendices A.3 and A.2.

Routing scheme . Unlike speculative decoding methods that periodically verify SLM outputs, our routing scheme aims to immediately decide whether to accept each SLM token, eliminating the need for rollbacks. As shown in Figure 2, this approach reduces unnecessary draft and verification computations, which is especially beneficial in computation-intensive batch-serving scenarios. Specifically, the neural router estimates divergence probability at each generation step using SLM outputs. When this probability exceeds a predefined threshold p th, the LLM is invoked to correct the current output token. Following speculative decoding methods [13, 15], we utilize highly parallel prefilling for efficient LLM KV-Cache updates, whose overhead can be further reduced by overlapping them with SLM decoding [35].

5 Experiment

5.1 Setup

Baselines . We use DeepSeek-R1-Distill-Qwen models as baselines, denoted by R1M B, where M indicates the model size in billions. We designate R1-1.5B and R1-32B as SLM and LLM, respectively, while intermediate sizes (7B, 14B) capture distillation scaling behavior. We compare various query-level routing (QR) methods from the RouteLLM framework [11], including similarityweighted ranking (QR-SW), matrix factorization (QR-MF), a BERT-based classifier (QR-BERT), and a Llama3-8B-based classifier (QR-LLM). For speculative decoding, we adopt EAGLE2 [14] and HASS [15] with R1-32B LLM. We use the official HASS draft model, and train the EAGLE2 draft model using its official script, as no pre-trained EAGLE2 draft is provided for R1-32B.

R2R setup . R2R routes between R1-1.5B and R1-32B using a lightweight 56M-parameter FFN router, trained on 7.6M token-level routing labels described in Section 4.1. More details on router architecture, training data, hyperparameters, and implementation are presented in Appendix A. Note that the router weights are fixed for all evaluations. The routing threshold p th is selected for 6B average parameter usage on the validation set. Performance-efficiency trade-offs are controlled solely by adjusting p th, without retraining the router.

Benchmark . We evaluate methods across challenging reasoning benchmarks, including mathematics (AIME 2024-2025 [10]; denoted as AIME), graduate-level question-answering (GPQADiamond [36]; denoted as GPQA), and coding tasks (LiveCodeBench 2024-08-2025-01; denoted as LiveCodeBench [37]). All experiments use a maximum output length of 32K tokens and zero generation temperature to ensure reproducibility.

Efficiency metric . Following previous works [2, 38], we use the average activated parameters per token as a hardware-agnostic efficiency metric, referred to as the average parameter ( ¯ M ) for brevity. For query-level routing, ¯ M is computed as the weighted average of SLM and LLM parameters based on their activation ratios across all outputs. For R2R, ¯ M includes the SLM and router parameters, along with the LLM parameters weighted by the LLM activation ratio. We also report the total Cost ( C ), defined as average activated parameters multiplied by the average output tokens per query. The average parameter and total cost reflect the average decoding speed and total latency, respectively. In addition, we report hardware-specific decoding speed on NVIDIA A800-80GB GPUs using SGLang [29] framework.

1.5

7.0

14.0

32.0

Activated Parameters Per Token (B)

10

20

30

40

50

60

Accuracy (%)

AIME

Ours

Distill

QR-MF

QR-BERT

QR-LLM

QR-SW

LLM

1.5

7.0

14.0

32.0

Activated Parameters Per Token (B)

0

10

20

30

40

50

LiveCodeBench

1.5

7.0

14.0

32.0

Activated Parameters Per Token (B)

0

10

20

30

40

50

GPQA

5.2 Performance

Scaling behavior . Figure 5 shows accuracy scaling with average activated parameters per token. Query-level routing (QR) methods exhibit near-linear accuracy scaling from 1.5B to 32B parameters. Distilled models (R1-7B, R1-14B) achieve superlinear gains with extensive training, reaching 88% of R1-32B's accuracy with just 50% of the parameter size at 14B. By routing only divergent tokens to the LLM, R2R achieves 92% average accuracy with only 17% average parameters, delivers even better scaling at a new Pareto frontier. Moreover, due to reduced output lengths, R2R offers an even better trade-off in terms of accuracy versus total test-time cost C (see Appendix B.7). The routing threshold in R2R also enables flexible, post-training control of this trade-off.

Numerical comparison . Table 2 shows numerical details of model performance around average parameter sizes of 7B and 14B. With an average parameter size of 5.6B, R2R outperforms the best query-level routing methods (in both 7B and 14B) by 1.4-2.4 × and 1.2-1.4 × , respectively. Compared to distilled models, R2R improves accuracy by 1.4-1.7 × over R1-7B and even surpasses R1-14B in average accuracy by 1.1 × . Relative to the extremes, R2R achieves 4.6 × higher accuracy than R1-1.5B and retains 92% of R1-32B's accuracy, while using the LLM for only 11-15% of tokens.

Generalizability . Beyond the R1 family, we further train and evaluate R2R on the Qwen3 series (both dense and MoE variants) spanning 0.6B, 1.7B, 8B, 30B-A3B, and 32B parameter scales. Across all configurations, R2R consistently surpasses both the base Qwen3 models and query-level routing baselines, demonstrating strong adaptability and cross-architecture generalization. We also evaluate on Arena-Hard (dialogue) and MMLU-Redux-Philosophy benchmarks, which are beyond the selected mathematical reasoning, QA, and code generation tasks. Remarkably, R2R continues to outperform

(a)

(b)

R1-14B while maintaining an average activated parameter size of only 6.1-6.7B, confirming its robust generalization across domains and model families (see Appendix B.2.2).

Sampling method extension . Beyond greedy decoding, we extend R2R to nucleus (top-p = 0.95) with temperature (temperature = 0.6). R2R still greatly advances Pareto frontier, with 1.4-1.5x higher AIME score over query-level routing, reaching R1-14B score with only 8.6B average parameters. Implementation details and theoretical analysis are provided in Appendix C.1.

5.3 Efficiency

Wall-clock speedup . Table 3 reports the wall-clock latency and speed for all methods on the AIME benchmark. All baselines use the official, highly efficient SGLang [29] framework and are evaluated with tensor parallelism on two NVIDIA A800-80GB GPUs. R2R uses the same thresholds as in Table 2; query-level routing methods use the 14B version for comparable performance. R2R achieves 1.62 × and 2.76 × generation speed over R1-14B and R1-32B, respectively. Compared to query-level routing, R2R delivers 1.48-1.52 × speedup. It also outperforms highly optimized speculative decoding methods with tree-like drafts, which speedup mostly at the current single-batch setup [16]. Further system-level optimization can be done to yield even greater gains for R2R. Note that, in theory, speculative decoding should exactly match the R1-32B LLM's output. However, we occasionally observe inconsistencies for very long outputs, likely due to cumulative numerical errors. We faithfully report this observation without questioning the equivalence guarantee of speculative decoding.

Computation and memory access . Compared with the LLM baseline, R2R achieves a 5 . 4 × reduction in per-token memory access. The total computation increases only marginally due to the additional SLM and router computation. Because decoding is largely memory-bound, the reduced memory traffic yields higher throughput. This substantial reduction in memory traffic translates into higher throughput that outweighs the minor compute overhead. Compared with speculative decoding approaches such as Eagle2 and HASS, R2R requires about 17 . 0 × less total computation, primarily because its immediate correction prevents the LLM from repeatedly prefilling unused tokens during periodic verification (see Figure 2). At the same time, it reduces memory access by 2 . 4 -2 . 5 × relative to these speculative methods, yielding a more balanced compute-memory trade-off. More details are provided in Appendix B.3.2.

5.4 Ablation Study

Starting from R2R in the first row of Table 4, we evaluate the effectiveness of our design by retraining the router with alternative objectives or reduced inputs. We adjust the routing thresholds p th of ablation variants to match or slightly exceed the LLM rate of our original router. All experiments are conducted on the AIME benchmark with all other settings held constant. Further discussions on the trade-off between the LLM usage rate and recall are provided in Appendix B.5.3.

Routing objective . As discussed in Section 3, we categorize different next-token predictions as either neutral or divergent . R2R improves efficiency by tolerating neutral differences and only routing truly divergent tokens to the LLM. When the router is trained to use the LLM for all different tokens, it fails to reach the original accuracy within the same amount of LLM usage, facing 1 . 4 × accuracy degradation, as shown in the second row of Table 4. This confirms that restricting LLM usage to only divergent tokens is crucial for reducing cost while maintaining high accuracy.

Router input . As discussed in Section 4, both SLM logits and token embeddings are strong indicators of divergence to be used as router inputs. When gradually remove these features, accuracy drops by up to 1 . 3 × , underscoring their importance. Note that while SLM logits can be computed from last-layer hidden states within the router in principle. However, doing so requires the capacity of the 234M-parameter embedding layer, which exceeds the capacity of the 56M-parameter neural router.

SLM-LLM combination . We extend R2R to the Qwen3 family by fixing the LLM and varying the SLM to form three combinations: Qwen3-0.6B+8B, Qwen3-1.7B+8B, and Qwen3-4B+8B. As Figure 10 in Appendix B.5.3 shows, with the LLM fixed, smaller SLM delivers a better Pareto frontier. It indicates that, when selecting SLM-LLM combinations, favoring a smaller SLM is often more efficient for the same accuracy target due to lower cost. We further analyze R2R alongside orthogonal techniques such as MoE and query-level routing in Appendix B.4.

5.5 Routing Result Observation

We analyze the routing behavior of R2R on the AIME benchmark, considering finished responses within the 32K token limit. Figure 6(a) shows the LLM usage rate across response positions. Each response is divided into the thinking process and the subsequent reply, with positions normalized to [0, 1]. The subplot widths reflect their respective average lengths. The results show that R2R routes noticeably fewer tokens to the LLM during the reply phase. It reflects the intuition that after internal thinking, the reply itself is straightforward and less demanding.

Following prior work [21], we further segment the thinking process into sequential thoughts based on tokens such as Wait and Alternatively . Figure 6(b) examines the LLM usage ratio within each thought. It shows that R2R relies more on the LLM at the beginning and end of each thought. This aligns with the expectation that the initial tokens set the direction for the thought, while the concluding tokens determine whether to end the thought, branch into alternatives, or continue deeper reasoning. Notably, these routing patterns are not hand-crafted but naturally emerge from router training. It helps R2R to effectively allocate LLMs for more better test-time scaling.

6 Conclusion

We introduce R2R, a token-level router that lets an SLM track an LLM's reasoning by correcting only path-divergent tokens. We propose a path-following labeling strategy and identify predictive signals that train a neural router for accurate token selection. On challenging benchmarks, R2R outperforms R1-14B with less than 7B average parameters, boosts SLM performance by 4.6 × with under 15% LLM usage, and achieves a 2.8 × wall-clock speedup over the LLM at comparable accuracy.

Limitations . Our routing strategy is chiefly tuned and tested for greedy sampling. While we have validated R2R on a limited set of alternative sampling policies, broader exploration could improve versatility. Further system-level optimizations are also needed to realize its full cost benefits.

Acknowledgments and Disclosure of Funding

This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164, 92364201), Tsinghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We thank Zinan Lin, Xuefei Ning, and Donglin Yang for their valuable discussions and suggestions. We thank Chao Xiong for his support with the SGLang interface. We also thank all the support from Infinigence-AI.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

OpenAI. Openai o3 and o4-mini system card, April 2025. URL https://cdn.openai.com/ pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card. pdf .

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Qwen Team. Qwen3, April 2025. URL https://qwenlm.github.io/blog/qwen3/ .

Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1 .

DeepSeek AI. Day 6: One more thing - deepseek-v3/r1 inference system overview. https:// github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/ day_6_one_more_thing_deepseekV3R1_inference_system_overview.md , 2025. Accessed: 2025-05-03.

Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL https://www.kaggle.com/ datasets/hemishveeraboina/aime-problem-set-1983-2024 .

Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, MWaleed Kadous, and Ion Stoica. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations , 2024.

Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for llm selections. In Proceedings of the International Conference on Learning Representations (ICLR) , 2025. URL https://openreview.net/forum?id=eU39PDsZtT . Poster presentation.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 , 2024.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858 , 2024.

Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. arXiv preprint arXiv:2408.15766 , 2024.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840 , 2025.

Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614 , 2025.

Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419 , 2025.

Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502.18600 , 2025.

Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware llm reasoning. arXiv preprint arXiv:2412.18547 , 2024.

Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585 , 2025.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations , 2023.

Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theorem proving in llms through large-scale synthetic data. arXiv preprint arXiv:2405.14333 , 2024.

Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692 , 2023.

Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. Mixllm: Dynamic routing in mixed large language models. arXiv preprint arXiv:2502.18482 , 2025.

Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028 , 2002.

Fengxiang Sun. jieba: Chinese text segmentation. https://github.com/fxsjy/jieba , 2013. Accessed: 2025-04-01.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2023.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , 2023.

Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation. https://www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-ofreasoning-distillation, 2025. Accessed: 2025-01-22.

Huan Ma, Jingdong Chen, Guangyu Wang, and Changqing Zhang. Estimating llm uncertainty with logits. arXiv preprint arXiv:2502.00290 , 2025.

Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning, 2025.

Rodolfo Zevallos, Mireia Farrús, and Núria Bel. Frequency balanced datasets lead to better language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 7859-7872, 2023.

Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F AbouElhamayed, Yueying Li, and Mohamed S Abdelfattah. Splitreason: Learning to offload reasoning. arXiv preprint arXiv:2504.16379 , 2025.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search, 2024. URL https://arxiv.org/abs/ 2408.08152 .

Dongwon Jung, Wenxuan Zhou, and Muhao Chen. Code execution as grounded supervision for llm reasoning, 2025. URL https://arxiv.org/abs/2506.10343 .

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URL https://arxiv.org/abs/2406.11939 .

Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with mmlu?, 2025. URL https://arxiv.org/abs/2406.04127 .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Fan Yang, Xinhao Yang, Hongyi Wang, Zehao Wang, Zhenhua Zhu, Shulin Zeng, and Yu Wang. Glitches: Gpu-fpga llm inference through a collaborative heterogeneous system. In 2024 IEEE High Performance Extreme Computing Conference (HPEC) , pages 1-7. IEEE, 2024.

Wenheng Ma, Xinhao Yang, Shulin Zeng, Tengxuan Liu, Libo Shen, Hongyi Wang, Shiyao Li, Ke Hong, Zhenhua Zhu, Xuefei Ning, Tsung-Yi Ho, Guohao Dai, and Yu Wang. Cd-llm: A heterogeneous multi-fpga system for batched decoding of 70b+ llms using a compute-dedicated architecture. ACM Trans. Reconfigurable Technol. Syst. , October 2025. ISSN 1936-7406. doi: 10.1145/3771288. URL https://doi.org/10.1145/3771288 . Just Accepted.

Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey on efficient inference for large language models. ArXiv , abs/2404.14294, 2024.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 , 2023.

Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406.14909 , 2024.

Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML) , 2025.

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 , 2025.

Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189 , 2025.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems , 6:87-100, 2024.

Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Proceedings of the 41st International Conference on Machine Learning , pages 28480-28524, 2024.

Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR) , 2025.

Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML) , 2025.

Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv:2505.11594 , 2025.

Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot llms. arXiv preprint arXiv:2505.18610 , 2025.

Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: Enhancing small-model reasoning with large model guidance at inference time. arXiv preprint arXiv:2504.12329 , 2025.

Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specreason: Fast and accurate inference-time compute via speculative reasoning. arXiv preprint arXiv:2504.07891 , 2025.

Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P Xing, Hongyi Wang, and Huaxiu Yao. Citer: Collaborative inference for efficient large language model decoding with token-level routing. arXiv preprint arXiv:2502.01976 , 2025.

Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guided speculative decoding for efficient llm reasoning. arXiv preprint arXiv:2501.19324 , 2025.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims of R2R to let the SLM follows the reasoning path of LLM with the routing of divergent tokens.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The theoretical result and proof are provided in Sections 3.1, 3.2 and Appendix E.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We specify all the training and test details in Section A and our code will be open source soon.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code is contained in implementation materials and will be open source soon.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in Section A

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We set the temperature to zero to allow the reproducibility of the experiment and report the actual latencies and speeds in Section 5.3

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We use NVIDIA A800-80GB GPUs, with eight for data generation and two for inference, as detailed in Section 5.1 and Appendix B.3.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research does not contain potential harmful consequences.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: R2R is a general-purpose method aimed at improving the efficiency of reasoning LLMs. It is not tailored to any specific application domains that might pose clear positive or negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models used are properly cited and clarified in Section 5.1 and Appendix A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is contained in implementation materials and has been open source. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We use the LLM as a verifier when generating the training data. The way of LLM usage is detailed in Sections 3.2 and 4.1.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Additional Experiment Setups

A.1 Router Architecture

Inputs projection . At each decoding step, we use the hidden states of the last layer, the top 100 logits with the highest values and the embeddings of the predicted token from the SLM to generate the routing result. We first apply linear projections to align their dimensions with the hidden states and then concatenate the features from the logits, hidden states, and token embeddings. Finally, we use another linear layer to project the concatenated features to match the input feature dimension of the model backbone.

Neural network backbone . For the router architecture, we adopt a six-layer feed-forward network (FFN) with residual connections between blocks as the backbone, using a hidden size of 1024. The architecture of each FFN follows the common design used in previous LLMs [39]. Each block begins with LayerNorm for input normalization, followed by a pair of linear projections forming an expand-then-contract structure with an expansion factor of 4. Dropout is applied to each linear layer, with a GELU activation function between them. These blocks are connected using residual connections. At the end of the last block, we apply an additional layer normalization and a linear layer to convert the output to a single value, followed by a sigmoid function to produce the normalized prediction of the router. A predefined threshold p th between 0 and 1 is used for generating binary results from the router output. Predictions above the p th are considered that current tokens diverge from the LLM's reasoning path.

A.2 Routing Data

Training dataset . Our training data for the router are sourced from tasks across three distinct scenarios: mathematics, code, and question answering (QA). The mathematics problems are drawn from the American Invitational Mathematics Examination (AIME) [10], covering the years 1983 to 2022. Code and QA problems are sampled from Bespoke-Stratos-17k dataset [31]. We use only the formatted questions from these datasets as prompts and generate responses using DeepSeek-R1Distill-Qwen-32B, with the temperature set to 0 and a maximum generation length of 32,768 tokens. Only responses that contain an end-of-sequence (EOS) token within the specified length are retained as effective samples, which will be used for subsequent stages of our data generation pipeline, as discussed in Section 4.1.

Validation dataset . Our validation dataset are constructed in the exact same way as the training data, but with different queries. The validation dataset comprises all 30 problems from AIME 2023, 69 coding problems from the Bespoke-Stratos-17k dataset that are excluded from the training set, and 60 QA problems selected from the GPQA-Extended [36] dataset.

A.3 Training Scheme

Loss function . Due to the significant class imbalance in the training data, we adopt the weighed BCEWithLogitsLoss as our loss function. The weight of each class is calculated inversely proportional to its frequency in the class, which encourages the model to pay more attention to underrepresented classes.

Training hyperparameters . During training, we employ the AdamW optimizer with hyperparameters β 1 = 0 . 9 and β 2 = 0 . 999 . The learning rate is set to 5 × 10 -5 , with a dropout rate of 0.1 and a weight decay of 5 × 10 -4 . We train the neural network with float32 precision. The router is trained for up to 50 epochs using a batch size of 1024, with early stopping applied based on a patience of 10 epochs. Validation is performed at every epoch. We adopt the checkpoint corresponding to the best-performing epoch on the validation set as the final router used.

Threshold selection . After training, we use the validation dataset to select a preferred threshold. We pass the pre-collected neural router inputs from the validation dataset through the neural router and record the predicted divergence probabilities. By sweeping p th from 0 to 1, we analyze how different thresholds affect the LLM usage rate and average parameter size., as shown in Figure 8. This process is efficient, as all router inputs (SLM logits, token embeddings, and last-layer hidden states) are pre-collected and evaluated in a single pass. During inference, given any user-defined average parameter budget, we set the threshold to meet the target budget accordingly.

A.4 Routing System Implementation

Model initialization . The routing system consists of three components: a SLM (R1-1.5B), a LLM (R1-32B), and a router model. The SLM is loaded onto a single GPU (GPU 1) using the SGLang scheduler, with the mem_fraction_static set to 0.15. The LLM employs tenser-parallel inference distributed across two GPUs (GPU 0 and GPU 1) via SGLang schedulers managed by PyTorch's distributed multiprocessing framework with the NCCL backend, with the mem_fraction_static set to 0.80. The router model is directly loaded onto GPU 0 using standard PyTorch, independent of the SGLang interface. Prior to inference, each of SLM and LLM is individually warmed up using simple inputs to stabilize GPU kernels and caches, ensuring consistent inference latency.

Inference workflow . During each inference step, the workflow begins with the SLM decoding a single token, returning the corresponding last-layer hidden states and output logits. The router generates a divergence probability based on these outputs of SLM. If the probability surpasses the predefined threshold, the LLM is activated to extend the sequence by one token. Specifically, a new request, constructed from the input token IDs, is placed into the LLM's input queue. Subsequently, a new schedule batch is initialized for the LLM, explicitly setting the forward mode to EXTEND and allocating appropriate memory for handling input sequences. The system maintains prefix indices to track processed tokens, enabling efficient token management between models. When the LLM extends a token, it is communicated back through an output queue to replace the SLM's predicted token. A token manager actively tracks the sequence states during the generation process, managing active sequences and handling termination conditions effectively. At each token position, the dynamic routing mechanism assesses model outputs, determines the appropriate routing decision, and updates sequence states accordingly. This iterative process continues until a sequence is completed or reaches the predefined maximum token limit.

B Additional Experiment Results

B.1 Verifier Robustness

B.1.1 LLMVerifier Against Human Verifier

Setup . We conducted a human evaluation to validate the verifier's reliability. Specifically, we recruited four undergraduate students to independently label 1,357 differences as either neutral or divergent . The differences are between R1-1.5B and R1-32B on the first six AIME-2024 questions. Three annotators' labels determined the ground truth for divergence by majority voting (general divergence , positive rate: 24.8%) and unanimous consent (core divergence , positive rate: 11.9%). The fourth annotator's labels, along with several LLM verifiers, were then compared against these ground truths. All annotators and LLM verifiers received identical contexts and instructions.

Results and discussion . Our selected verifier (Qwen2.5-72B) closely matches human expert performance. The detailed results are shown in Table 5.

B.1.2 Impact of Verifier Quality

Setup . Next, we examined the sensitivity of our method to verifier quality. We applied the sentencelevel path-following routing method (Equations 3-5) to the AIME 2024 benchmark across verifiers (Qwen2.5-3B to Qwen2.5-72B) within 8K token budget.

Results and Discussion . Our analysis reveals that even moderate-recall verifiers (e.g., Qwen2.5-3B) still significantly improve reasoning path guidance compared to the distilled 7B baseline. Note that the current verifier prompt is tuned for the Qwen2.5-72B verifier. Empirically, smaller verifiers should benefit from prompts biased towards classifying ambiguous cases as divergent , thereby enhancing recall.

B.2 Generalizability

B.2.1 Data Labeling Method Generalizability

The labeling method of R2R, relying on semantic comparison rather than formal verification, can easily generalize across tasks and achieve high performance even in some out-domain tasks. Unlike methods requiring task-specific external tools (e.g., formal proofs [40], code execution [41]), R2R uses an LLM verifier to identify general semantic divergence in meaning, reasoning, logic, or conclusions F.1.

In practice, we apply the identical labeling strategy and verifier prompt across closed-form math, coding tasks, and open-ended QA tasks. This consistency enables our router to naturally generalize to unseen tasks during inference, as demonstrated later.

For tasks with subjective divergence criteria that are challenging to identify semantically within a single sentence, the continuation length can be extended. This adjustment gradually transitions from sentence-level routing to full routing, the latter of which only requires overall response quality evaluation, which is a feasible requirement commonly met for LLM tasks.

B.2.2 Router Generalizability

Generalize across benchmarks . Our router leverages outputs from the SLM (e.g., logits) to predict token divergence. Benefiting from the inherent generalizability of SLMs, indicators such as logits entropy robustly identify divergent tokens across different tasks. To validate the router's generalizability, we directly apply our router, trained for math, QA, and code, to additional benchmarks: Arena-Hard for Dialog [42], and the Philosophy split from MMLU-Redux [43], which are never included in our training dataset. As table 7 shows, although the cost-accuracy trade-off gains are a bit less pronounced on out-of-domain tasks, R2R nonetheless exhibits strong generalization, which continues to outperform the 14B model with less than 7B parameter size.

Generalize to different LLMs . We test whether a router, trained on data from a specific SLMLLM pair (e.g., 0.6B-32B), can be effectively applied to a different pair (e.g., 0.6B-8B) without retraining. The experiments are conducted with the Qwen3 model family (0.6B, 8B, and 32B), tested on the AIME (2024+2025) benchmark with a target recall of 0.95 for divergent tokens. The results are summarized in Table 8. Our empirical results indicate acceptable performance when directly substituting the LLM of a trained router without additional retraining. This can be attributed to the common divergence patterns shared by strong LLMs paired with the same weaker SLM.

B.3 Efficiency Analysis

B.3.1 Data Generation Overhead

As Table 9 shows, our four-stage pipeline completes in 56 h (448 GPU hours). The LLM response stage dominates runtime (35 h, 280 GPU hours), but it can be mitigated by directly utilizing the responses of LLMs from open-source SFT datasets, provided they were generated by the same LLM used for routing. The SLM prefill step is highly efficient, requiring only 0.1 h of wall clock time. The subsequent LLM continuation and verification stages take 7 h (56 GPU hours) and 14 h (112 GPU hours), respectively. Compared to downstream tasks, the overall data generation pipeline remains relatively lightweight and efficient.

B.3.2 Computation and Memory Access

Following prior work [44-46], we analyze the computational and memory access overhead of R2R against various baselines on the AIME benchmark. Table 10 presents a detailed comparison of total computation (TFLOPs), total memory access (TB), and the average memory access per token (GB).

Given that the decoding process in large language models is predominantly memory-bound, the volume of memory access is a critical factor influencing overall throughput. As shown in the table, our method, R2R, markedly reduces total memory access to just 216 TB, a 4.5x reduction compared to the R1-32B model. This efficiency is achieved with only a modest increase in total computation, which primarily arises from the additional KV-cache updates required when switching between the LLM and SLM.

In contrast, speculative decoding approaches such as Eagle2 and HASS incur substantially higher computational costs-more than 22x that of R1-32B. This overhead is a direct consequence of their

complex tree-structured draft and verification processes, which can involve up to 60 tokens per cycle. Despite this significant computational demand, these methods also achieve considerable throughput improvements by effectively reducing the memory access bottleneck during the decoding phase. R2R, however, provides a more balanced trade-off, achieving significant memory savings without the extreme computational overhead of speculative methods.

B.3.3 Inference Latency Breakdown

We integrated R2R into the production-level SGLang serving infrastructure, which is demonstrated by the high absolute throughput results for our method and the baselines. To assess the latency implications of our token-level routing, we performed a detailed runtime breakdown. The analysis was conducted on the AIME benchmark, averaged over 60 questions, using two A800 GPUs.

The results, shown in Table 11, indicate that the token-level routing decisions made by our 56M parameter router introduce minimal latency overhead (5.96%). Given the small size of the neural router, we anticipate that further runtime improvements can be achieved through additional systemlevel optimizations.

B.3.4 LLMActivation Intervals

The cost efficiency of the routing system largely depends on how frequently the LLM is activated during generation. Since the EXTEND operation of the LLM is inherently compute-bounded when consecutive LLM calls are spaced further apart. In other words, given a fixed LLM activation rate, longer intervals between two LLM invocations lead to better GPU utilization and less scheduling overhead per generated token.

To better understand this effect, we analyze the distribution of the interval between consecutive divergent tokens. Figure 7 illustrates these distributions for both the training dataset and actual inference traces on AIME25. As shown, the actual distribution closely aligns with that observed in training dataset, suggesting that the router generalizes well to real inference scenarios. More importantly, both distributions exhibit a long-tailed shape, indicating that divergence events occur sparsely and irregularly, which allows R2R to sustain high computational efficiency without overusing the LLM.

1

10

20

30

40

50

Interval Between Consecutive Divergent Tokens

0

5

10

15

20

Frequency (%)

Mean: 8.24

Median: 4.00

Std: 9.59

1

10

20

30

40

50

Interval Between Consecutive Reference Tokens

0

5

10

15

20

25

30

35

40

Frequency (%)

Mean: 6.49

Median: 2.00

Std: 94.54

B.4 Comparison and Compatibility with Orthogonal Techniques

B.4.1 R2R and Mixture-of-Experts (MoE)

We compare our method, R2R, with the Mixture-of-Experts (MoE) architecture, another prominent approach for efficient model scaling. This section begins with a conceptual comparison, followed by empirical results, and concludes by discussing how the two methods can be integrated to create a more favorable performance-efficiency trade-off.

Conceptual comparison . As noted, both MoE and R2R leverage partial activation, but they differ in key design aspects, as summarized in Table 12. This conceptual distinction motivates their potential integration, which we explore next.

Empirical comparison . We evaluate the MoE model Qwen3-30BA3B and an R2R model (routing between Qwen3-0.6B and Qwen3-8B) on the AIME benchmark. The 60 questions are split into Easy (28) and Hard (32) subsets based on whether the question ID is less than or equal to 7. The results in Table 13 highlight the complementary strengths of each approach. Thanks to extensive pretraining, MoE models can match dense model performance on challenging tasks. However, their fixed per-token overhead limits efficiency on simpler inputs. In contrast, R2R dynamically adjusts computation based on token-level divergence , achieving lower cost on easier examples, though its lighter training cannot match MoE performance on harder questions given the same activated parameters. Given their complementary advantages, we explore the integration of both methods.

R2R for MoE . Conversely, R2R's principles can enhance future MoE designs. Current MoE methods generally use uniform-sized experts; introducing mixed-sized experts could enable query-adaptive activation. Furthermore, supplementing an MoE's training objective with explicit divergence or difficulty-based routing supervision from R2R may encourage more adaptive usage of larger experts for only the most critical decoding steps. Although retraining MoE models is beyond this paper's scope, this presents an exciting direction for future research.

B.4.2 R2R and Query-level Routing (QR)

Query-level routing (QR) is methodologically orthogonal to R2R. Consequently, the composition of the two is a natural and well-motivated design choice. We investigated combining R2R with query-level routing methods using the Qwen3 model series with varied sizes, evaluated on the AIME benchmark. Specifically, we treated R2R-S (0.6B+8B) and R2R-L (0.6B+32B) as the SLM and LLM, respectively, for query-level routing.

From Table 15, we can conclude that (1) R2R models generally serve as superior SLM/LLM candidates for query-level routing compared to original single-model setups, consistently advancing the Pareto frontier; (2) The interaction between medium-sized R2R models and query-level routing involving smaller or larger R2R-LLMs is empirically non-trivial. This observation suggests exciting opportunities for future research into sophisticated routing strategies, which combine the search space of query and token-level routing methods.

0.0

0.2

0.4

0.6

0.8

1.0

threshold

p

th

0

20

40

60

80

100

Recall (%)

threshold

p

th vs Recall

0.0

0.2

0.4

0.6

0.8

1.0

threshold

p

th

0

5

10

15

20

Param. (B)

threshold

p

th vs Param.

0

5

10

15

20

Param. (B)

0

20

40

60

80

100

Recall (%)

Param. vs Recall

B.5 Additional Discussions and Analysis

B.5.1 Influence of Routing Threshold

Figure 8 visualizes how the routing threshold p th affects the average parameter size on the validation dataset. In our experiments, we select thresholds based on the user-specified average parameter budget (e.g., 6B) measured on the validation set. However, the threshold does not strictly guarantee the same average parameter size during inference, particularly when query difficulty varies significantly from the validation set. Empirically, we observe minimal variance between the target and actual parameter sizes, as the difficulty of our validation and evaluation datasets is generally well aligned. For tasks that are much easier or harder than those considered in this paper, users can construct a tailored validation set using our data labeling pipeline to determine an appropriate threshold.

Beyond analyzing the relationship between threshold and average parameter size, we also examine its effect on divergent token recall. This links the average parameter size to the ability to recall divergent tokens. As shown in Figure 8, recall rises rapidly with increasing average parameter size, demonstrating the strong predictive performance of our router model.

B.5.2 Sentence-level Path-following Routing

The sentence-level path-following assumption, while necessary to manage the vast search space, could be limiting in certain scenarios. For instance, an SLM might rearrange sentences in a paragraph while still preserving a valid reasoning path, yet our sentence-level verifier might incorrectly flag this as a divergence. While such mislabeling of a neutral continuation as divergent does not degrade final performance, it can introduce unnecessary computational overhead by triggering the LLM more often than required.

To empirically evaluate the impact of this assumption, we extended our routing strategy to a more general N-sentence path-following approach. In this setup, the continuation-and-verification process described in Section 3.2 is not stopped by the first sentence-ending token, but continues for N sentences. As N increases, the strategy transitions smoothly from our default sentence-level approach (N=1) towards the full path-following routing defined in Section 3.1, which is guaranteed to find the optimal path (see Appendix E).

We conducted experiments on a subset of the AIME dataset to measure the effect of varying N. We found that increasing N from 1 to 5 only yields a modest reduction in the measured divergence rate, from 2.91% to 2.66%. This corresponds to a marginal decrease of 0.08B in the average activated parameters required to maintain the same level of accuracy.

Given the limited performance gain relative to the substantial increase in data labeling complexity and cost, we empirically selected N=1 (sentence-level routing) as the most practical and efficient default setting for our framework.

B.5.3 Ablation Study Results

To further investigate the impact of different routing strategies, we perform an ablation study that analyzes the relationship between Positive Rate , defined as the fraction of tokens routed to the LLM, and Recall , defined as the proportion of correctly identified positive samples in the training dataset.

Routing objective . As illustrated in Figure 9, R2R (default) consistently achieves higher recall across all positive prediction rates compared to the other variants, demonstrating that its routing policy is more accurate in identifying informative tokens. The different variant, which substitutes the semantic divergence metric with a simpler criterion that only considers mismatched predictions, exhibits a noticeable drop in recall. This degradation highlights the importance of fine-grained semantic alignment in accurately distinguishing divergence tokens and maintaining routing precision.

Routing input . Figure 9 also shows that HS and HS+Token variants, which rely solely on hidden-state distance or only combine it with current token ids, yield similar but lower recall curves, indicating that low-level representation similarity alone is insufficient to capture reasoning divergence and prove the significance of the logits during routing. In contrast, R2R achieves both higher efficiency and better coverage, maintaining near-optimal recall with less than 14% of tokens routed to the LLM.

SLM-LLM combination . To analyze the performance of R2R under difference model pairs, we apply it to the Qwen3 family with a fixed LLM selection (Qwen3-8B) and vary the SLM to form three different SLM-LLM combinations: Qwen3-0.6B+8B, Qwen3-1.7B+8B, and Qwen3-4B+8B. All combinations were tested on AIME with identical settings. Figure 10 shows the relationship between accuracy and the average activated parameters per token: For a fixed LLM, smaller SLMs yield a better Pareto frontier, matching accuracy at lower cost or achieving higher accuracy at the same cost. The intuition is that, for identical tokens that can be predicted by multiple SLMs, routing them to the smaller SLM can preserve correctness of the prediction with less computational cost, thereby improving the overall accuracy-efficiency trade-off.

Interestingly, we found that all seven AIME questions that the SLM answered correctly were also answered correctly by the LLM, indicating that the LLM consistently outperforms the SLM rather than specializing in a different subset of math problems.

These results validate that the semantic divergence criterion used in R2R provides a more discriminative signal for routing, effectively balancing accuracy and efficiency in the overall system. Moreover, appropriate selection of the SLM-LLM combination can further improve the cost-accuracy Pareto frontier.

0

10

20

30

40

50

Positive Rate (%)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Recall

Ablation Variants

R2R (default)

Different

HS+Token

HS

B.6 Systematic Failure Analysis

To identify systematic failure modes, we analyzed the response status of R2R (routing between R1-1.5B and R1-32B without sampling) on the AIME and GPQA benchmarks. As shown in Table 16, the primary failure mode occurs when R2R cannot complete its reasoning within the 32K token limit. This typically happens due to repetitive reasoning patterns that were not encountered during the router's training phase. Additionally, we observe that R2R tends to invoke the LLM slightly

1

2

3

4

5

6

7

8

Activated Parameters Per Token (B)

0

10

20

30

40

50

60

70

80

Accuracy (%)

0.6B+8B

1.7B+8B

4B+8B

Distill

more frequently on the GPQA benchmark. This is particularly noticeable for queries involving rare tokens, such as complex protein or chemical compound names, where the SLM's uncertainty is higher, triggering the router.

B.7 Performance-Cost Trade-off

100

200

300

400

500

Cost (K Tokens × Billion Parameters)

10

20

30

40

50

60

Accuracy (%)

AIME

Ours

Distill

QR-MF

QR-BERT

QR-LLM

QR-SW

LLM

100

200

300

400

500

600

Cost (K Tokens × Billion Parameters)

0

10

20

30

40

50

LiveCodeBench

100

200

300

400

500

Cost (K Tokens × Billion Parameters)

0

10

20

30

40

50

GPQA

Figure 11 illustrates the trade-off between accuracy and the total cost of generation. As defined in Section 5.1, the cost metric is calculated as the average number of output tokens multiplied by the average parameter size, serving as a hardware-agnostic indicator of latency across methods. R2R consistently outperforms both query-level routing methods and distilled LLMs, establishing a new Pareto frontier in performance-cost trade-offs.

C Additional Design Discussions

C.1 Extend R2R to More Sampling Methods

C.1.1 Path-Following Routing Under Sampling

R2R naturally extends to sampling-based decoding by adapting divergence labeling to a probabilistic setting. Specifically, the deterministic continuation sequence pair ( S s , S l ) from Equations 3-5 are replaced by k sampled continuation pairs under a chosen sampling method. Divergence is then evaluated for each sampled pair. The routing decision for each token then depends on whether the overall divergence probability exceeds a predefined threshold:

If setting P threshold to the LLM's selfdivergence probability P ( V ( S l , S l ) = 0) , it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone.

C.1.2 Efficient Data Generation

Due to sampling stochasticity, each mismatched token can branch into multiple continuations. However, sampling multiple continuations for divergence labeling is computationally expensive. Since continuations are sentence-level, we empirically observe that the divergence decision primarily depends on the first differing token between the SLM and LLM samples, rather than the subsequent

continuations. To efficiently approximate probabilistic routing, we therefore only sample for nexttoken generation y i ( θ s | S <i ) and y i ( θ l | S <i ) , while leaving continuations deterministic.

Given the extensive volume of tokens, we set k = 1 and P threshold = 0 . 5 , simplifying the setup. This approximation incurs overhead comparable to our greedy data generation pipeline. In implementation, our data-generation pipeline adapts to the new setup by using sampling-based generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.

Using this modified pipeline, we generate 16.1 million routing labels, covering topics of math, coding, and QA with queries from the Bespoke-Stratos dataset. Table 17 summarizes the statistics of the generated training dataset with the sampling method. Compared with the non-sampled dataset in Table 1, sampling increases the average divergence rate by only 3.1%, which remains low overall, indicating divergence patterns consistent with greedy decoding.

C.1.3 Experiment Results

For preliminary experiments, we use DeepSeek-R1's recommended sampling settings (temperature = 0.6, top-p = 0.95) for both dataset generation and evaluation. The neural router remains unchanged, as it already takes the sampled token's embedding as input.

We evaluate the extended R2R on the AIME benchmark, reporting its pass@1 accuracy over 16 independent samples per problem, and compare it against query-level routing baselines following Section 5.1. As shown in Table 18, R2R continues to advance the Pareto frontier, achieving 1.4-1.5 × higher AIME scores than query-level routing methods and matching the R1-14B performance with only 8.6B average activated parameters. Extending R2R to stochastic decoding preserves its Pareto optimality relative to both distilled models and query-level routing approaches.

C.2 Routing Algorithm Details

As illustrated in Algorithm 1, the objective of our routing algorithm is to identify and correct pathdivergent tokens during inference by using predictions from a large language model (LLM). Both SLM and LLM perform greedy decoding, and a token is considered identical if both models produce the same prediction.

When the SLM and LLM outputs differ, the algorithm triggers a continuation process for each model: the SLM and LLM, respectively, continue generating tokens, starting from their initial divergent prediction, until a special separator (SEP) token is produced. These continuations yield two complete sequences that differ only at the initial divergence point and subsequent tokens.

To assess whether this divergence impacts reasoning, a separate LLM-based verifier is employed. This verifier receives the two generated sequences and outputs a binary decision: 0 if the sequences are semantically neutral, and 1 if they diverge significantly in meaning or logic.

If the verifier outputs 0 (neutral), the router accepts the SLM's prediction. However, if the verifier outputs 1 (divergent), the algorithm corrects the current token by adopting the LLM's prediction, thus preventing further drift from the intended reasoning path.

This approach ensures that the system maintains high alignment with LLM reasoning, while minimizing unnecessary reliance on the LLM by routing to the more efficient SLM whenever possible.

Algorithm 1 Path-Following Routing

D Additional Related Work

D.1 Model-level Compression

Extensive studies have been proposed to accelerate the costly decoding processes of LLM by compressing the models themselves [47]. Prominent techniques include sparse attention mechanisms [48-52] and model quantization [53-58]. In contrast, our R2R method focuses on optimizing inference above the model level, complementing these model compression techniques. Therefore, it can be effectively combined with them to further enhance inference efficiency.

D.2 Concurrent Mix Inference Methods

Given recent rapid advancements in reasoning LLMs, several concurrent studies also explore mix inference strategies that integrate small and large language models. These methods differ primarily in their routing granularity, objectives, and specific routing schemes.

Step-level Methods : Speculative Thinking [59], SplitReason [35], and SpecReason [60] operate at the reasoning step granularity. Speculative Thinking observes that LLMs excel at affirmation and reflection compared to SLMs . Thus, it employs heuristic triggers-such as affirmations ("yeah"), reflections ("wait"), or verification signals ("check")-to invoke the LLM selectively after detecting delimiter tokens (e.g., "\n\n"), enhancing subsequent SLM-generated segments. SplitReason aims to offload difficult reasoning steps to the LLM . It first uses a strong LLM to identify challenging reasoning steps, then trains the SLM to generate a special token (i.e., '<bigmodel>') signaling the LLM to take over these difficult steps. SpecReason adapts speculative decoding to reasoning

step-level . It utilizes the LLM to evaluate steps generated by the SLM, reverting to the LLM only when the score of SLM-generated steps falls below a certain threshold.

Token-level Methods : Unlike step-level methods, CITER [61] and RSD [62] adopt finer-grained token-level routing strategies. CITER formulates the routing problem as a long-horizon reinforcement learning task, optimizing for final answer quality and inference efficiency. Because CITER targets general decoding tasks (e.g., short-form QA), repeatedly generating the complete response to determine token-level preferences remains computationally manageable. In contrast, RSD leverages an existing reward model to dynamically select tokens for LLM generation whenever the SLM-produced tokens exhibit low reward scores. This approach performs well on tasks with clear and definable reward signals.

Distinctiveness of R2R : R2R distinguishes itself from concurrent works by specifically targeting immediate divergence correction at token granularity . Unlike methods focused on offloading complex reasoning steps, R2R addresses the subtle scenario where the SLM and LLM may agree on challenging steps, yet diverge unexpectedly on seemingly straightforward tokens (under human or LLM judgment). Such divergences can significantly alter the subsequent reasoning path, thus requiring immediate correction. Moreover, R2R differs from the speculative decoding scheme, as it does not rely on periodic LLM verification steps to inform routing decisions. Instead, R2R immediately routes divergent tokens to the LLM, effectively preventing divergence without incurring rollback overhead.

Given these distinct objectives and design choices, integrating R2R with these concurrent methods represents a promising direction for future research, enabling even more effective mix inference frameworks.

E Proof of Quality Guarantee for Full Path-Following Routing

E.1 Notations

We summarize the notations used throughout this proof:

θ l , θ s : the large and small language models (LLM, SLM), respectively.

S <i = [ x 0 , . . . , x n -1 , y 0 , . . . , y i -1 ] : the prefix sequence up to, but not including, position i , where S < 0 contains only the input tokens.

y i ( m | S <i ) : the token generated at position i by model m ∈ { θ s , θ l } given prefix S <i .

V ( · , · ) ∈ { 0 , 1 } : the quality verifier function, returning 1 iff the first sequence achieves the same quality as the second.

S ( L ) <i : sequence up to i generated by the LLM only.

S ( M ) <i : sequence up to i generated by the mixed (routed) strategy.

S s , S l : continuation sequences as defined in Equations 8 and 9.

T <i : the sequence formed by S ( M ) <i concatenated with the LLM's continuation tokens:

The routing decision at each step i is given by:

The continuation sequences after the i -th token are:

where ⊕ denotes sequence concatenation.

E.2 Theorem

The sequence S ( M ) , generated by the path-following routing strategy, is guaranteed to achieve the same quality as its LLM-only counterpart S ( L ) under V .

E.3 Proof

Base Case. At i = 0 , S ( M ) < 0 = S ( L ) < 0 = [ x 0 , . . . , x n -1 ] , so T < 0 is simply the LLM's full sequence. Thus, V ( T < 0 , S ( L ) ) = 1 .

Inductive Hypothesis. Suppose that for some k with 0 ≤ k < n , we have V ( T <k , S ( L ) ) = 1 ; i.e., continuing from S ( M ) <k with the LLM produces a sequence of equal quality to S ( L ) .

Inductive Step. Consider the ( k +1) -th token y ( M ) k determined by Eq. 7. There are three cases:

(a) Identical: y k ( θ s | S ( M ) <k ) = y k ( θ l | S ( M ) <k ) . Then y ( M ) k matches the LLM's output, so the sequence remains identical and V ( T <k +1 , S ( L ) ) = 1 .

̸

(b) Neutral: y k ( θ s | S ( M ) <k ) = y k ( θ l | S ( M ) <k ) but V ( S s , S l ) = 1 . The router selects the SLM token, so T <k +1 = S s . By definition, V ( T <k +1 , S ( L ) ) = 1 .

(c) Divergent: V ( S s , S l ) = 0 . The router selects the LLM token, so the mixed sequence again matches the LLM's output, and thus V ( T <k +1 , S ( L ) ) = 1 .

Conclusion. By mathematical induction, for all i ∈ [0 , n ] , the continuation T <i maintains the same quality as S ( L ) under V . At generation completion ( i = n ), S ( M ) = S ( M ) <n +1 , so V ( S ( M ) , S ( L ) ) = 1 .

■

F Prompts and Examples

F.1 Prompt for Verifier Model

As discussed in Section 4.1, we design a structured prompt for the verifier model (Qwen2.5-72BInstruct) to assess whether divergences between two sentences affect their meaning, reasoning, logic, or conclusions. Please refer to Text 2. for the exact prompt. The prompt highlights the divergence point and provides explicit criteria for labeling. It instructs the model to justify its judgment with a brief explanation and includes illustrative examples to guide the model's understanding of both scenarios.

F.2 Response Example

We use an example question from AIME and responses from R1-1.5B, R1-32B and R2R to provide an intuitive understanding of our method.

Text 1. Question

Find the largest possible real part of

where z is a complex number with | z | = 4 .

Text 3-5 shows the example responses. The R1-1.5B and R1-32B models produce distinct final answers for the maximum real part, reflecting a divergence in their reasoning paths. By contrast, R2R identifies and corrects this divergence, navigating the correct reasoning path to get the final answer matches that of the stronger 32B model. At the same time, R2R tolerates neutral differences-such

as minor phrasing or presentation-between models when these do not affect the core reasoning or conclusions. This selective routing mechanism enables R2R to deliver both high efficiency and accuracy by only invoking the large model for tokens that would otherwise lead to substantive differences in meaning or logic.

Text 2. Prompt For Verifier Model

Task:

Determine if the divergence between Sentence 1 and Sentence 2 affects the meaning, reasoning, logic, or conclusions derived from them.

Instructions:

The marker « » indicates where the sentences diverge. It is not part of the original text.

Assess whether this divergence changes the meaning, reasoning, logic, or conclusions, or if it introduces new information or contradictions.

Output 1 if:

The divergence causes a change in meaning, reasoning, logic, or conclusions.

It introduces new information, shifts focus, or contradicts prior facts.

The sentences follow different reasoning paths or focus on different aspects.

Output 0 if:

The divergence is superficial and does not affect meaning, reasoning, logic, or conclusions.

Both sentences follow the same reasoning path or lead to the same conclusion.

Reasoning: Provide a brief explanation of how the divergence impacts (or does not impact) meaning, reasoning, logic, or conclusions.

-

"The ratio of adults to total people «chang»ed from 5/12 after adding 50 people."

Reasoning: The change from "is" to "changed" does not affect the overall meaning, reasoning, logic, or conclusions.

Reasoning: The change in method (from integration by parts to substitution) alters the reasoning and approach to solving the problem.

Text 3. R1-1.5B Response

Okay, so I have this problem: I need to find the largest possible real part of the expression

, where z is a complex number with | z | = 4 . Hmm, complex numbers can sometimes be tricky, but let me try to break this down step by step.

First, I know that any complex number z can be written in polar form as z = 4 e iθ since its magnitude is 4. That might make things easier because then I can express z in terms of its angle θ . So, let me write that down:

z = 4 e iθ

Then, 1 z would be 1 4 e -iθ because the reciprocal of e iθ is e -iθ .

So, substituting back into the expression, we have:

Let me compute each term separately. First, let's compute (75 + 117 i ) · 4 e iθ .

Multiplying 75 + 117 i by 4 gives:

4 × 75 = 300 and 4 × 117 i = 468 i , so the first term is 300 + 468 i .

· · ·

The final answer is:

Text 4. R1-32B Response

Okay, so I have this problem here: I need to find the largest possible real part of the expression

where z is a complex number with | z | = 4 . Hmm, that sounds a bit tricky, but let me try to break it down step by step.

First, I know that z is a complex number with magnitude 4. So, I can represent z in polar form. That might make things easier because working with polar forms can simplify multiplication and division. So, let me write z = 4 e iθ , where θ is the argument of z . That way, | z | = 4 is automatically satisfied.

Now, let's substitute z = 4 e iθ into the expression. So, the expression becomes:

4 e Simplify each term separately. Let's start with the first term:

Thus, the largest possible real part is:

540

Text 5. R2R Response

Okay, so I have this problem: I need to find the largest possible real part of the expression

, where z is a complex number with | z | = 4 . Hmm, that sounds a bit tricky, but let me try to break it down step by step.

First, I know that z is a complex number with magnitude 4. So, I can represent z in polar form as z = 4 e iθ , where θ is the argument of z . That might make things easier because working with exponentials can sometimes simplify multiplication and division. So, let me rewrite the expression using this polar form. The expression is:

Substituting z = 4 e iθ , we get:

Simplify each term: First term:

· · ·

Which is: 300 cos θ -468 sin θ + i (300 sin θ +468cos θ ) · · ·

Thus, the largest possible real part is:

540

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a token-level routing method to decide whether to use a small language model (SLM) or a large language model (LLM) for each token. The goal is to have the SLM follow the LLM's reasoning path and it seems that only a small fraction of tokens diverge the reasoning paths. Therefore, by identifying those divergent tokens and using the LLM for generating those while using the SLM for other tokens can lead to comparable results with the LLM while having smaller number of activated parameters and having a significant speedup compared to just using the LLM for the entire generation. Their main contributions are (1) proposing an automatic pipeline to label divergent tokens. (2) developing a token-level routing scheme using a lightweight neural router. (3) enabling a more efficient performance scaling at test time using their R2R framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) This work introduces a token-level approach that enables small language models to follow the reasoning paths of a larger model by identifying the divergent tokens and only correcting those by using the LLM. This allows the SLM to achieve comparable performance with the LLM while using the LLM only for generating a small fraction of tokens. This is significant as this can achieve a more efficient performance scaling at test time.\n\n(2) The lightweight routing scheme identifies whether to use the LLM or SLM at each token level, and it can immediately decide whether to accept the SLM generation or not. Therefore, it eliminates the need for rollbacks, which methods like speculative decoding usually face (Speculative decoding methods only periodically verify the tokens and invalidate all tokens after a correction).\n\n(3) The authors report the results of ablation studies on using divergent vs different tokens, as well as using the hidden states, logit, and/or token embeddings as the inputs to the router to show the effectiveness of their choices in the framework. They also report the results on three different datasets and provide comparisons between R2R and the query-level routing methods. Overall, they have a sound experimental setup, in my opinion. \n\nWeaknesses:\n\n(1) I believe the writing can be improved, and it should be revised to correct some sentences. For example, in line 24, I believe \"hundreds of parameters\" is not correct, as you are saying \"LLMs with hundreds of parameters (line 124) and SLMs with \"few billions of parameters (line 26)\"\n\n(2) Some assumptions, such as \"greedy\" and \"sentence-level\" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.\n\nQUESTIONS:\n(1) What are your thoughts on combining R2R with some query-level routing methods so that you can choose which SLM or LLM to use based on the difficulty of the query, and then using R2R to route tokens to the SLM or LLM for the generation? \n\n(2) Can you please clarify what you mean in lines 170-171? I assumed you would rely on the smaller model for generating the majority of the tokens. Why is it said that only 3% of generated tokens are from the smaller model?\n\nLIMITATIONS:\nTheir method is based on greedy decoding and sentence-level assumptions. These assumptions can be limiting. For example, the SLM might want to generate the sentences in a paragraph in a different order but still follow a valid reasoning path and achieve the same conclusion as the LLM in some cases. Also, the greedy decoding may potentially be problematic for tasks where we want to generate more diverse or creative text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes R2R (Roads to Rome), a token-level routing method that combines the efficiency of Small Language Models (SLMs) with the reasoning quality of Large Language Models (LLMs). Rather than relying solely on either model, R2R selectively routes divergent tokens to an LLM, while leaving the rest to the SLM. The authors introduce: (1)A token-level data labeling pipeline to identify divergent tokens; (2)A lightweight neural router trained to predict token divergence; (3) A routing strategy that improves inference efficiency with minimal loss in reasoning quality. By applying R2R between DeepSeek R1-1.5B (SLM) and R1-32B (LLM), they show up to 4.6× accuracy improvement over SLM, with 2.8× speedup over the LLM, surpassing distilled and query-level routing methods in performance-cost tradeoffs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Novel Insight & Intuition: The paper identifies that most token differences between SLM and LLM are either identical or neutral, and that only a small fraction leads to performance degradation—an important insight for efficient inference.\n\n(2) Efficient Labeling Pipeline: The sentence-level LLM continuation and an LLM verifier to distinguish divergent tokens are well-designed, scalable, and avoid the need for expensive human annotations.\n\n(3) Strong Empirical Results: First, it advances the Pareto frontier on reasoning benchmarks (AIME, GPQA, LiveCodeBench). Second, it achieves impressive speed-accuracy tradeoffs (e.g., 2.8× speedup over R1-32B with comparable accuracy).\n\n(4) Robust Engineering: Implementation details are thorough, including training strategy, threshold tuning, GPU memory optimizations, etc.\n \nWeaknesses:\n\n(1) Verifier Dependence: The data labeling pipeline depends on a separate LLM-based verifier, which may introduce a bias or circular dependency (LLM verifying LLM paths).\n\n(2) Domain Bias in Training Data: The training data (math, QA, and code) may not generalize well to open-ended tasks (e.g., embodied, navigation, etc).\n\n(3) Assumes Greedy Decoding: The method is only validated under greedy decoding. It remains unclear how well R2R generalizes to sampling-based strategies used in many real-world applications (e.g., temperature > 0).\n\nQUESTIONS:\n(1) How does R2R perform under temperature sampling or nucleus sampling? Would the divergence pattern be consistent?\n(2) What guarantees does the LLM verifier provide, and could it mislabel semantically divergent tokens as neutral?\n(3) Have you considered batch routing decisions (e.g., look-ahead or segment-level) instead of per-token routing?\n(4) Could R2R apply to generative tasks like summarization or dialogue, where path divergence may not always be objectively verifiable?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a routing scheme that delegates the decoding flow from a small LLM (SLM) to a more powerful LLM (LLM) when there is divergence between the reasoning path. A divergence is defined to emerge when there is a disagreement between the predictions of the SLM and LLM and that disagreement will result in a different reasoning path till the end of the sentence. Here the difference between two reasoning paths is evaluated by an external verifier model (Qwen2.5-72B used in this work).\n\nThe authors further propose a data generation pipeline to collect labels for such divergent tokens, and then train an simple MLP-based token-level router to dynamically decide when to delegate the reasoning to the LLM. The authors apply the proposed method (R2R) to R1-1.5B and R1-32B models from the DeepSeek family and evaluate on a variety of tasks. The results show impressive performance (comparable to R1-32B and better than R1-14B) with significantly reduced inference costs, which are measured by the number of activated parameters and the wall-clock time (latency and token generation speed).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The intuition of the dynamic reasoning makes sense and the definition of divergent tokens is reasonable.\n+ The empirical results are impressive and the decoding acceleration is practically promising. \n+ The authors provide ablation study on the design choice of the MLP router in Section 5.4. While the results are insufficiently discussed (see comments below), the study does support the choice of the input fields to the router and the training objective.\n+ The paper is written clearly with plenty of details in support.\n\nWeaknesses:\nJust to clarify that I think this work is of good quality overall. So many of the weaknesses are not hard criticisms but constructive feedback on which I would invite more discussions with the authors.\n- (Discussion against [3]) I understand that [3] is concurrent. But now that the authors mention [3] in Line 227, I think it would be better to discuss properly in the section of related work.\n- (Insufficient discussion of ablation study) I think the ablation study on the routing objective is not sufficiently discussed. I suppose delegate \"more\" tokens to LLM will not hurt the performance this much. Is it because the objective change causes the router misses more divergent tokens? Moreover, it is not clear what the authors mean by \"it fails to reach the original accuracy within the same amount of LLM usage, ...\" Does it imply that the authors manually control the activation of the LLM in this experiment?\n\nQUESTIONS:\n- I wonder if MoE can be included in the context here. In my opinion, MoE is another way of utilizing a model with larger capacity. But the difference is that MoE implicitly uses the whole capacity with fixed cost at each token. In contrast, R2R uses small capacity model with low costs most of time but escalate to larger model when needed. It would be great if we can find a way to properly compare the two. I understand that this can be too demanding for the review period. So this is not a request but an open question out of curiosity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Roads to Rome (R2R), a token-level routing method that selectively utilizes LLMs only for tokens that cause reasoning path divergence when generated by small language models (SLMs). The core insight is that only ~11% of tokens genuinely diverge reasoning paths between SLMs and LLMs, with most differences being either identical or neutral variations (e.g., abbreviations, alternative expressions). The authors develop an automatic data generation pipeline to identify divergent tokens and train a lightweight router for real-time routing decisions. Experiments on DeepSeek R1 models demonstrate 2.8X speedup over R1-32B while maintaining comparable performance, and superior efficiency compared to query-level routing and distilled models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. **Novel and well-motivated insight**: The distinction between neutral and divergent token differences is conceptually sound and empirically well-supported. The observation that most SLM-LLM differences don't affect reasoning paths is valuable.  \n2. **Comprehensive experimental evaluation**: The paper evaluates across several benchmarks (AIME, GPQA, LiveCodeBench) with appropriate baselines including query-level routing, distilled models, and speculative decoding. They also measured real hardware speedup. \n 3. **Strong empirical results**: Achieving 92% of R1-32B accuracy with only 17% average parameters is impressive. The 4.6× accuracy improvement over R1-1.5B with minimal LLM usage demonstrates practical value.  \n\n**Weaknesses:** \n\n1. **Critical limitation to greedy decoding**: Most production systems use temperature-based sampling or other stochastic methods. The authors acknowledge this but underestimate its significance for deployment. \n2. **Limited generalizability**: All experiments use only DeepSeek R1 models. No evidence that the approach generalizes across model families, architectures, or even different size pairs within the same family. \n\n3. **Shallow failure analysis**: Limited discussion of systematic failure modes or scenarios where neutral/divergent distinction breaks down. When does the approach perform poorly, and why?\n\nQUESTIONS:\n1. **Sampling method extension**: How fundamental is the greedy decoding limitation? Could the neutral/divergent distinction be meaningfully applied to temperature-based sampling? This seems critical for practical deployment - can you provide any analysis or preliminary experiments?  \n2. **Cross-model generalization**: Would a router trained on DeepSeek R1-1.5B/32B work with other model pairs? How much retraining would be required?   \n3. **Verifier robustness analysis**: How sensitive is performance to verifier quality? What happens with weaker verifier models? Could you provide human evaluation of neutral/divergent distinctions to validate the verifier's reliability?  \n\n4. **Systematic failure modes**: Can you identify task types or reasoning patterns where the approach consistently fails? Does performance degrade on highly creative or non-linear reasoning tasks?  \n\n5. **Real-world deployment considerations**: How would this integrate with existing serving infrastructure? What are the latency implications of token-level routing decisions in production systems?\n\nLIMITATIONS:\nThe authors adequately acknowledge the greedy sampling restriction and need for system optimizations. However, they significantly understate the generalizability concerns and don't sufficiently address the verifier dependency issues. The authors should also discuss the computational overhead more transparently in the context of alternative approaches.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}