R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
Authors: Tianyu Fu
Yi Ge
Yichen You
Enshu Liu
Zhihang Yuan
Guohao Dai
Shengen Yan
Huazhong Yang
Yu Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM
efficient reasoning
long context
Primary area: deep_learning
TL;DR: We train a token-level neural router to let SLM following LLM reasoning paths by replacing only divergent tokens

Abstract: Large Language Models (LLMs) achieve impressive reasoning capabilities at the cost of substantial inference overhead, posing substantial deployment challenges. Although distilled Small Language Models (SLMs) significantly enhance efficiency, their performance suffers as they fail to follow LLMs' reasoning paths. Luckily, we reveal that only a small fraction of tokens genuinely diverge reasoning paths between LLMs and SLMs. Most generated tokens are either identical or exhibit neutral differences, such as minor variations in abbreviations or expressions. Leveraging this insight, we introduce **Roads to Rome (R2R)**, a neural token router that selectively utilizes LLMs only for these critical, path-divergent tokens, while leaving the majority of token generation to the SLM. We also develop an automatic data generation pipeline that identifies divergent tokens and generates token-level routing labels to train the lightweight router. We apply R2R to combine R1-1.5B and R1-32B models from the DeepSeek family, and evaluate on challenging math, coding, and QA benchmarks. With an average activated parameter size of 5.6B, R2R surpasses the average accuracy of R1-7B by 1.6×, outperforming even the R1-14B model. Compared to R1-32B, it delivers a 2.8× wall-clock speedup with comparable performance, advancing the Pareto frontier of test-time scaling efficiency.

R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
Tianyu Fu 1,2 Yi Ge Yichen You', Enshu Liu', Zhihang Yuan?,
Guohao D Shengen Yan?, Huazhong Yang', Yu Wang!
Tsinghua University
Infinigencc Al
Shanghai Jiao Tong University
Abstract
Large Language Models (LLMs) achieve impressive reasoning capabilities at the cost substantial inference overhead, posing substantial deployment challenges. Although distilled Small Language Models (SLMs) significantly enhance efficiency, their performance suffers as they fail to follow LLMs' reasoning paths. Luckily, reveal that only small fraction of tokens genuinely diverge reasoning paths between LLMs and SLMs. Most generated tokens are either identical or exhibit neutral differences, such minor variations in abbreviations expressions. Lever- aging this insight, we introduce Roads to Rome (R2R), a neural token routing method that selectively utilizes LLMs only for these critical, path-divergent tokens, while leaving the majority of token generation to the SL.. We also develop an automatic data generation pipeline that identifies divergent tokens and generates token-level routing labels to train the lightweight router. We apply R2R to combine R1-1.5B and RI-32B models from the DeepSeck family, and evaluate on challeng- ing math, coding, and QA benchmarks. With an average activated parameter size of 5.6B, R2R surpasses the average accuracy of R1-7B by 1.6x, outperforming even the R1-14B model. Compared to R1-32B, it delivers 2.8x wall-clock speedup with comparable performance, advancing the Pareto frontier of test-time scaling efficiency. Our code is available at httpps:: /gtphtb..t...t...///g//e.g com/ ttttttuunnnn www.nnics/aa.
Introduction
Large Language Models (LLMs) demonstrate strong capabilities across a wide range of tasks [13].. Building upon the largest and strongest LLMs, test-time scaling has become prominent way to further boost their abilities on challenging tasks [4. is typically done by generating extensive Chain-of-Thought (CoT) reasoning before producing the final answer. However, this approach requires massive LLMs with hundreds of billions of parameters to generate thousands of tokens per query [8) resulting in significant inference overhead [9]
Distilled Small Language Models (SLMs), containing only few billion parameters, offer much higher generation efficiency. Through supervised finetuning on LLM responses, SLMS can mimic LLM reasoning behaviors, making them a popular alternative. However, SLMs may still produce different reasoning paths from their LLM counterparts during inference, causing severe performance degradation. For example, compared to the RI-32B LLM, the RI-1.5B SLM provides different final answers 45% questions in the AIME benchmark [10] suffering 4.8x x reduction in accuracy shown in 2
'Equal contribution.
Corresponding author: Yu Wang (yu-wang@tsinghuu.ddu.oo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) question: Compute 99992 9908x1000.
(b)
5% divergent
LLM: Okey, let's think step by step.y
9999' hard rewrite it...
89% identical
11% different
SLM: Okey, let us think step by step... 9999 is 999801
6% neutral
identical v neutral /
divergent X
Figure I: (a) Examples of R2R routing objective. Given a partial response context, if SLM next-token prediction is not identical with LLM's, it is further categorized as neutral or divergent based on their effects the reasoning path. (b) Distribution of identical, neutral and divergent labels in the R2R training set with 7.6M token labels.
Fortunately, we find that SLMs and LLMs often agree on next-token predictions given the same context. Instead, large performance gaps between them primarily arise from cumulative errors: their reasoning paths increasingly drift apart after some crucial differences in partial responses. To investigate this, we treat each step's LLM partial response as prefix context and assess whether the next-token prediction from SLM is identical to LLM (Figure [a)). Across 2,094 queries totaling 7.6M tokens generated by the 32B L.M,, the 1.5B SLM differs on only 11% of tokens-far less frequent than differences observed in final answers. Moreover, some of these differences are merely neutral variations, such as abbreviations or alternative expressions (e.g., ler's vs. let us), which do not affect reasoning outcomes. The key drifts start from subset of different tokens, which we call divergent tokens. These tokens genuinely alter the meaning, logic, or conclusion of the current sentence, thus diverging the subsequent reasoning path. This observation motivates to selectively use SLM and LLM for different generation steps. It naturally leads to a critical research question:
Can SLMs follow LLM reasoning paths by replacing only divergent tokens?
If addressed, we could unlock substantial efficiency advantage of SLMs for most generation steps, yet preserving the high-quality reasoning typical of LLMs. This can enable better test-time scaling by advancing the efficiency-perforee performance Pareto frontier.
The main challenge of SLM-LLM mix inference involves two interconnected parts: labeling the preferred model under certain objective, and designing the routing scheme enforce it during inference. Previous methods typically route at the query level, selecting either SLM or LLM for entire response to maximize human preference win-rate within a cost budget [T 12 However, these approaches rely on human annotations and complex router designs, whose data labeling and routing scheme are both too expensive for fine graiggraddd, token-level routing. Alternatively, speculative decoding methods aim for identical outputs between SLM and LLM at the token level [13 They draft outputs with SLMs (or draft models) and periodically verify them with LLMs. While accurate, this strict verification leads to low acceptance rates. Additionally, mid-draft differences invalidate all subsequent tokens, severely restricting the accepted lengths as shown Db)
To address these challenges, we propose Roads to Rome (R2R), token-level routing method that selectively utilizes LLMs only for path-divergent tokens during SLM generation. We begin by automating leven model preference labeling under path-following objective. Starting from the I.LM's reasoning paths, we identify different predictions for SLM and I.LM, briefly continue generation from the point of difference, then use another LLM as verifier to determine whether the difference is truly divergent or merely a neutral variation. This labeling approach minimizes the lower bound of L.M usage by allowing neutral SLM-LLM differences. Using the resulting labeled dataset, we train a lightweight neural router to predict and immediately route divergent SLM tokens to the LLM for correction. We further improve routing accuracy by identifying predictive indicators of divergence such as SLM uncertainty and token rarity, available directly during SLM inference. Our contributions are summarized follows.
Data Labeling Pipeline. We develop an automatic pipeline to label divergent tokens. We formalize the global token-routing optimization problem, then propose path-following strategy to generate routing labels with highly parallel, local decisions. We validate that SLM can effectively match LLM reasoning quality by following these routing labels.
Token-Router Design. We introduce a token-level routing scheme using lightweight neural router. We investigate SL. outputs that aid accurate token routing and incorporate them into the router, enabling immediate and more accurate routing of divergent tokens.

---

input:
It's
(a)
It's
(b)
SLM:
I's 99
hard
IT's hard
rewrile
It's
I'ss
999801
999801
LLM:
hard
It's hard
It's 91
999801
999801
output:
hard
re
write
hard
Figure 2: (a) R2R uses neural router to inspect SLM outputs each step, immediately corrects divergent tokens with LLM,, then continues generation from the corrected outputs. (b) Speculative decoding uses LLM periodically verify SLM outputs are identical to LLM predictions, invali- dating all tokens after the first correction within the period.
Performance- Pareto Frontier. R2R enables more efficient performance scaling at test time. Compared to query-level routing and distilled R1-14B, it delivers 1.1-1.5x higher AIME accuracy with 1.5-1.6x lower latency. R2R also provides a 2.8x wall-clock speedup over RI-32B LLM at similar accuracy, while raising RI-1.5B SLM accuracy by 4.6x with only 12.9% LLM usage.
2 Related Work
Test-time scaling improves LLM performance at the higher cost of inference, often through the explicit generation of the CoT reasoning paths [4] For more effective scaling, previous works optimize the length and width of reasoning paths, reduce the generation overhead of each path [17 [8].
Controlling reasoning paths. Some approaches reduce LLM output lengths. They employ
prompting [19]. post-training (20), heuristics [21] generate concise CoT with fewer decoding
steps [9 21). Others explore the width of paths. They let LLMs generate multiple reasoning paths in parallel, then select the best outcome with methods like best-of-N voting (22) or external verifiers 23 Both strategies modify the structure of reasoning paths, which are perpendicular to R2R's 's focus on reducing the overhead of each path.
Model routing. Model routing reduces generation cost by selecting the most suitable model for each query based on difficulty and budget. Current works explore selection criteria of learned human preferences [1I] reward signals (24), query tags 225], and model profiles [12. Despite simplicity,
they enforce the same LLM for each response, yielding suboptimal performance for the common mixed-difficully generations. In contrast, R2R routes at token level to further improve efficiency.
Speculative decoding. Speculative decoding accelerates generation by fixing the low parallelism in LLM decoding [13 It drafts outputs through SLM sequential decoding, then periodically verifies them with high-parallel LLM prefilling. However, speculative decoding pursues identical output token (distribution) between SLM and L.M,, causing low acceptance rate. In addition, it often that not all tokens generated by SLM within one draft-verify cycle can pass LLM verification. The mid-draft rejection invalidates all subsequent drafts and LLM verifications as shown in Figure bb), leading to frequent rollbacks. Expanding the single draft-chain to draft-tree alleviates the problem, but also incurs higher overheads that harm batch serving efficiency [66. Considering the internality of CoT process, R2R accepts neutrally different output tokens, and immediately corrects all divergent tokens to avoid any rollback.
Model Preference Labeling
In Section 3. we formalize the token-level routing problem, aiming to minimize generation cost without sacrificing response quality. In Section 3.2 we introduce the path-following strategy for this problem, which assigns model preference labels to each output token, and empirically validate its effectiveness.
3

---

3.1 Token-level Routing Formulation
For autoregressive language models, reasoning can be represented as sequence of next-token predictions. Throughout this paper, we focus on greedy sampling for simplicity:
y/ arg max P Pm, Zo). In Mo.............................. y1-1) arg max Pm, (y)S<.).
(I)
Here, and Yi denote the input and output tokens, respectively. For notational simplicity, we define the token sequence step as Scs [ro, ................................ Fn yo, ...-,-1)1 where Seo is the input tokens. The next-token probability Pm, is predicted by model my E (0a, 01] at step i, where 0, and 0g denote the SLM and LLM, respectively.
The essence of the routing strategy is to define a routing function R that selects the model for each decoding step:
m, = R(Sev, 8g, Bt)
(2)
Our objective is minimize the total generation cost while ensuring that the output sequence matches the quality of LLM-only outputs. We define the cost C as the sum of activated model parameters per token over the entire generation process. The quality of response is evaluated by sssc-specffic criteria, such as correctness for math problems, pass rates for coding tasks, or LLM-based grading for writing tasks. We define as the verifier function, which returns if and only if two sequences are of equivalent quality.
3.2 Path-following Routing Strategy
Optimally solving the token-level routing problem is computationally prohibitive, especially for large-scale data generation. While better routing sequences- diverging from the LLM's reasoning path-may exist, finding them requires exhaustively searching vast O(2") space and generating thousands of output tokens for each search.
To overcome this practical limitation, propose greedy, sentence-level path-following routing strategy that reduces the search complexity to O(z). Rather than exploring all possible model choices, approach incrementally aligns mixed-model generation with the reasoning path established by the LLM. At each generation step, the strategy prefers the efficient SLM unless this would cause a meaningful divergence from the LLM's intended reasoning path, determined by continuation- and-verification mechanism.
Specifically, at each step, we first compare the next-token predictions from the SLM and LLM. If the predictions are identical, we confidently select the SLM, as this does not affect the output sequence. When predictions differ, we must determine whether the difference is neutral or divergent. To do so, we construct two candidate sequences by appending predictions from SLM and LLM to the previous token sequence, respectively. Both sequences stopping criterion is met (c.g., EOS token generated). These continuations reveal how the initial token difference affects subsequent reasoning, measured under optimal yet achievable conditions (i.e., LLM-only continuation). If the first continuation still matches the quality of the second under the verifier function V, the difference is considered neutral; otherwise, it is divergent and the token is routed to the LLM.
then continued using the LLM until
O.,
= = y (0r or V(S,, =
identical
neutral
m
Oz.
V(S.,S:)
divergent
S, S21 t [y (e.1S + [ @ ()) EOS
(4)
SL.M
LIM continuation
S Sei @ (0: +
(ye (Or) Sec @ (0)()))))))) EOS
(5)
L.M token
I.M. continuation
Equations 3 formalize the routing strategy. Here, y (m,; (Sex) indicates that this output token
generated by model m; given the previous sequence Sei. as a simplified expression of Equation] [ The continuation sequences, respectively generated after SLM and I.I. token, are denoted by S. and S.. The operator @ indicates concatenation of token sequences.

---

Table 1: Statistics of tokens difference and divergence across query types in the training dataset.
Type
#Query
#Token
#Different
Diff. Rate
#Divergent
Div. Rate
Math
587
2.9M
195.1K
6.8%
81.8K
2.8%
Code
698
3.2M
329.0K
10.3%
151.9K
4.7%
QA
735
1.4M
290.8K
20.2%
139.4K
9.7%
Summary
2094
7.6M
814.9K
10.8%
373.1K
4.9%
When continuation produces the full response by stopping only at the regular EOS token, we call this full path-following routing. By using the quality verifier from Section[.1] the mixed generated token sequence is guaranteed achieve the same quality its LLM-only counterpart, it always remains on a path that could achieve LLM-only quality. The formal proof of this quality guarantee is provided in Appendix E While the resulting model choice sequence M = [mo--.. ... - can be used as labels for router training, full continuation is computationally expensive for large-scale data generation. In addition, the effect of current difference the final output quality thousands of tokens away is too hard to learn for the neural router to be trained.
sentence-level path-following routing, where the continuation ends at the current sentence, as shown in F 2). We monitor contence-ending symbols, like period, during continuation and use existing semantical sentence separators [26 27 to conclude generation if the sentence truly ends. To verify this local continuation, a capable LLM serves as a sentence-level verifier V", shown in Figure (step 3). It prompted to compare the continuations and determine whether the initial token difference introduces a meaningful divergence from the L.LM's intended reasoning path, or merely neutral abbreviation. Instead of verifying the entire generation, this approach checks the reasoning path at the sentence level, greatly improving data labeling efficiency.
In practice,
We empirically validate the effectiveness of sentence-level path-following routing using Qwen2.5- 72B (28 as the verifier model, with prompts detailed in Appendix F.I Among 17 AIME-24 questions correctly solved by R1-32B within 8K-token limit, our path-following strategy achieves pa comparable accuracy (16 questions correctly answered) while relying on the larger R1 R1-32B model for only 3% of generated tokens.
By locally evaluating token choices through sentence-level path- nath-following routing, we closely align mixed inference with the L.M''s high-quality reasoning path, eliminating the prohibitive overhead of global evaluations. However, direct use of this strategy for real-time inference is impractical, as relies costly LLM continuation and verification. Instead, the local nature of our strategy simplifies routing decisions, creating an casier learning task for neural router compared to global routing. We therefore design and train a lightweight neural router that efficiently approximates this strategy, relying solely on SLM outputs to determine when to use the LLM during inference.
4 Token-Level Neural Router
This section describes our methodology for constructing the neural router. Specifically, we detail how routing labels are generated for training using the sentence-level fath-following strategy (Section[4.1 identify predictive SLM indicators for the router (Section 4.2), and outline the neural router's architecture along with its routing scheme for inference deployment (Section 4.3)
4.1 Training Data Generation
We use sentence-level path- hollowollooing routing to generate training labels for the neural router, incor- porating several optimizations to control data labeling overhead.
Figure B shows our data generation pipeline. Given queries from existing datasets, we first obtain the complete LLM response, either directly from the dataset or via batched LLM inference, Next, we use highly parallel SLM prefilling to efficiently identify tokens where the SLM prediction is identical to the L.M,, allowing to exclude about 90% of tokens from further processing. For the remaining 10% of differing tokens, we perform batched L.M continuations from each SLM prediction. To
further improve efficiincy, apply prefix caching in current frameworks [29 30 to reuse KV-Cache
5

---

query
Compute 9999* 9998x1000.
stepd:
LLM response
Let
think
step
by
step
99
hard
re
write
stept:
SLM prefir
Lel
think
step
stop
99
99
99 + different
wrlle
+ differeen
stap2:
Let
us
about step by step.
LLM continuation
Let
's
think step
step 99
99
99
9801.
step3: verify
Verify
Let's think step by step and
Let us think about it step step
neutral
Verify
99992 is hard, rewrite
and
9999? is 999801
divergent
output label
SLM
SLM
SLM
SLM
SLM
SLM SLM
SLM
SLM
SLM
SLM
LLM
SLM
SLM
SLM
SLM
Figure 3: R2R data labeling pipeline. Given query question, the LLM first generates response to establish the desired reasoning path. The SLM then prefills this path to identify identical and different next-token predictions. For each different SLM token, the LLM continues generation from that point. Finally, verifier model determines whether each difference leads neutral or divergent outcome, labeling the model preference as SLM or LLM, respectively.
Data Type
Divergence Rate (%)
100
Data Type
Percentage (%)
Divergent
Math
40-
Non-Divergent
60
Code
Average:
Average:
QA
0.45
1.70
40
20
20
2
10-5
10
10-4
10
10
Entropy
Token Frequency (#Occurrence/# Total Taken)
Figure 4: Oracle insights for router design. (a) SLM entropy distribution, clipped at 99th percentile for visualization clarity (b) Divergence rate and frequency of different tokens.
computations for shared context prefixes across multiple continuations (c.g., everything preceding Let in Figure 3) Continuations for the corresponding LLM tokens, S())), are directly extracted from the pre-generated LLM response, eliminating redundant computation. Finally, the verifier model compares both continuations and label routing preference. Further analysis of the robustness of the verifier and the comparison human experts are provided Appendix B.I
Using this pipeline, we efficiently generate 7.6 million routing labels in approximately 2.3 days on A800 GPUs, covering topies of math, coding, and QA with queries from the Bespoke-Stratos (31 dataset. Table summarizes the statistics of the generated training dataset.
4.2 Predictive Indicators of Divergence
We explore predictive indicators that can help identify divergent tokens. To enable immediate routing, focus indicators that can be acquired solely during the SLM's next-token predictions. The following analysis is based on 7.6 million tokens in our training set.
SLM logits. As shown in Figure Fa), divergent tokens exhibit substantially higher entropy in the SLM's output logits, with a 3.8x mean value over that of non-divergen tokens. We observe similar trends with other uncertainty measures (32), and concurrent work [33 also confirms such observation. These empirical results indicate that increased uncertainty in SLM predictions is strongly correlated with token divergence. Motivated by this, our router takes top- 100 SLM logit values as one of its input features.
Token frequency. Figure 4(b) shows that low-frequency tokens in the dataset are more likely to be divergent. This likely arises from the long-tail token distribution the training data, making rare tokens harder for SLMSs to model effectively due to the limited capacity (34). Given this insight, our router explicitly incorporates ookenffreueency biases by using the token embedding of router inputs.

---

4.3 Router Design and Routing Scheme
Model architecture. Guided by insights in Section 4.2 we design the neural router as a lightweight. six-layer feed-forward network (FFN) with 56M parameters. It takes the SLM's output logits and tokenized embedding, along with its last-layer hidden states for additional semantic context. All inputs are linearly projected, concatenated, and fed into the FFN backbone. The router outputs binary classification probability, indicating whether the current token diverges from the LLM's reasoning path. Full network architecture detail descriptions are in Appendix A.1
Training scheme. We train the router with cross-entropy loss using the labeled data described in Section 4.1 To address class imbalance caused by the low divergence rate, we re-weight the loss inversely to class frequency. After training, we use the validation set select the routing threshold that meets the user-defined LLM usage rate. The full training details are provided in Appendices A.3 and 2
Routing scheme. Unlike speculative decoding methods that periodically verify SLM outputs, our routing scheme aims to immediately decide whether to accept each SLM token, eliminating the need for rollbacks. As shown in Figure this approach reduces unnecessary draft and verification compu- tations, which is especially beneficial in computation -intensivve batch-serving scenarios. Specifically, the neural router estimates divergence probability at each generation step using SI.M outputs. When this probability exceeds predefined threshold Pt. the LLM is invoked to correct the current output token. Following speculative decoding methods [3 15], we utilize highly parallel prefilling for efficient LLM KV-Cache updates, whose overhead can be further reduced by overlapping them with SLM decoding 35
5 Experiment
5.1 Setup
models as baselines, denoted by RI-MB, where M indicates the model size in billions. We designate R1-1.5B and R1-32B as SLM and LLM, respectively, while intermediate sizes (7B, 14B) capture distillation scaling behavior. We compare various query-level routing (QR) methods from the RouteLLM framework [11.. including similarity- weighted ranking (QR-SW), matrix factorization (QR-MF), BERT-based classifier (QR- (QR-BER BERT), and Llama3- SB-based classifier (QR-LLM). For speculative decoding, adopt EAGLE2 [14] and HASS [5] with R1-32B LLM. We use the official HASS draft model, and train the EAGLE2 draft model using its official script, as no pre-trained EAGLE2 draft is provided for R1-32B.
Baselines. We use SeepSeek-RI-Distll---
R2R setup. R2R routes between R1-1.BB and R1-32B using lightweight 360--arameter FFN router, trained 7.6M token-level routing labels described in Section 4.1 More details on router architecture, training data, hyperparameters, and implementation are presented in Appendix A Note that the router weights average parameter usage on the validation set. cefformance-efficiengg trade-offs are controlled solely by adjusting Pu. without retraining the router.
fixed for all evaluations. The routing threshold Pth is selected for 6B
Benchmark. We evaluate methods across challenging reasoning benchmarks, including mathe- matics (AIME 2024-2025 [10] denoted as AIME), graduate-level question-ansseringg (GPQA- Diamond 36 denoted as GPQA), and coding tasks (LiveCodeBennch 2024-08-2025-00 denoted as LiveCodeBench 37 All experiments use a maximum output length of 32K tokens and zero generation temperature to ensure reproducibility.
Efficiency metric. Following previous works 2 38). we use the average activated parameters per token hardware-agnostic efficiency metric, referred to the average parameter (M) for brevity. For query-level routing, M is computed as the weighted average of SLM and LLM parameters based on their activation ratios across all outputs. For R2R, M includes the SLM and router parameters. along with the LLM parameters weighted by the LLM activation ratio. We also report the total Cost (C), defined as average activated parameters multiplied by the average output tokens per query. The average parameter and total cost reflect the average decoding speed and total latency, respectively. In addition, we report hardware-specific decoding speed on NVIDIA A800-80GB GPUs using SGLang (29] framework.
7

---

AIME
LiveCodeBench
GPOA
60
50
40
40
Our
Accuracy
Diail
GU.M
30
GRA1N
QR-BERT
20
20
GR-SW
SL.
10
2.5
7.0
14.0
32.0
32.0
1.5
14.0
Parametars
Activated Farameters
Activated Parameterz
Figure 5: Scaling of accuracy versus average activated parameters per token, evaluated across AIME, GPQA, and LiveCodeBench. R2R advances the Pareto frontier beyond distillation and query-level routing methods.
Table 2:
Performance and efficiency comparison across benchmarks and methods. Param. denotes the average activated parameters per token in billions; Cost is the average output tokens (thousands) query multiplied by average parameters (billions).
AIME
LiveCodeBench
GPQA
Average
Type Method
Acc.
Param.
Cost
Acc.
Param.
Cost
Acc.
Param.
Cost
Acc.
Param.
Cost
SLM R1-1.5B
12%
1,5
42
9%
1.5
43
8%
1.5
42
10%
1.5
42
LLM
RI-32B
57%
32.0
487
45%
32.0
606
46%
32.0
519
50%
32
537
R1-7B
32%
7.0
148
24%
7.0
168
29%
7.0
147
28%
7.0
154
QR-SW
27%
7.2
168
16%
7.1
188
16%
7.1
179
20%
7.1
178
7B
QR-LLM
27%
7.1
170
13%
7.1
195
7.0
172
20%
7.1
179
QR-BERT
28%
7.1
160
15%
7.0
189
21%
7.0
169
21%
7.0
173
QR-MF
27%
7.5
168
16%
7.1
190
16%
7.1
181
20%
7.2
180
R1-14B
48%
14.0
239
38%
14.0
267
44%
14.0
197
43%
14.0
234
QR-SW
37%
14.5
295
27%
14.0
333
25%
14.1
318
30%
14.2
315
14B
QR-LLM
45%
14.8
277
21%
14.1
356
28%
14.1
299
31%
14.3
311
QR-BERT
37%
14.0
25%
14.0
342
14.1
297
30%
QR-MF
42%
14.7
284
25%
14.0
336
20%
14.2
338
29%
14.3
319
R2R
Ours
55%
5.5
101
39%
106
44%
6.3
101
46%
5.6
5.2 Performance
Scaling behavior. Figure shows accuracy scaling with average activated parameters per token. Query-level routing (QR) methods exhibit near-linear accuracy scaling from 1.5B to 32B parameters. Distilled models (R1-7B, RI-14B) achieve superlinear gains with extensive training, reaching 88% of R1-32B's accuracy with just 50% of the parameter size at 14B. By routing only divergent tokens to the LLM, R2R achieves 92% average accuracy with only 17% average parameters, delivers even Pareto frontier. Moreover, due reduced output lengths, R2R offers
better scaling better trade-off in terms of accuracy versus total test-time cost C (see Appendix B.7). The routing threshold in R2R also enables flexible, post-training control of this trade-off.
Numerical comparison. Table shows numerical details model performance around average parameter sizes of 7B and 14B. With an average parameter size of 5.6B, R2R outperforms the best query-level routing methods (in both 7B and 14B) by 1.4-2.4x and 1. 1.2-1.4x, respectivvilll. Compared to distilled models, R2R improves accuracy by 1.4-1.7x over R1-7B and even surpasses R1-14B in average accuracy by 1.1x Relative to the extremes, R2R achieves 4.6x higher accuracy than R1-1.5B and retains 92% of R1-32B's accuracy, while using the LLM for only 11-15% of tokens.
Generalizability. Beyond the R1 family, we further train and evaluate R2R on the Qwen3 series (both dense and MoE variants) spanning 0.6B, 1.7B, 8B, 30B-A3B, and 32B parameter scales. Across all configurations, R2R consistently surpasses both the base Qwen3 models and query-level routing baselines, demonstrating strong adaptability and cross-architecture generalization. We also evaluate on Arena-Hard (dialogue) and MML.U-Redux Sew.-U-RRx--PPhossooo Philosophyy benchmarks, which are beyond the selected mathematical reasoning, QA, and code generation tasks. Remarkably, R2R continues to outperform

---

Method
#Token(K)
Latency(s) (tkes(
(9]
50-
Thinking Process
Roply
Average
R1-1.5B
28.2 +10.
199 =81
141.6
25th 75th Percenti
30-
RI-14B
17.1 #123
328 +272
52.1
R1-32B
15.2 122.
498 +456
30.5
10-
QR-SW
20.3 113.5
336 +:79
55.5
0.4
Normalzzed Tolkan Iistion
QR-LLM
18.7 1335
332 +334
56.1
(6)
Thought
QR-BERT
19.9 1132
350 +367
57.0
LLM Usage Natoo (6
QR-MF
19.3 +13.3
347 +359
55.6
40-
Ragle2
17.4 113.1
244 1994
71.4
20-
HASS
18.8 +129
256 +197
73.3
Ours
18.44133.
218 +161
84.3
0.0
0.2
0.4
0.5
0.8
1.0
Normalized Token Posttion
Table 3:
Comparison of latency, output token length, and average speed across methods. Sub- scripts note the standard deviations across AIME.
Figure 6: LLM usage rate at different posi- tions, normalized by (a) thinking and reply process, (b) each thought.
R1-14B while maintaining average activated parameter size of only 6.1-6.7B, confirming its robust generalization across domains and model families (see Appendix B.2.2).
Sampling method extension. Beyond greedy decoding, we extend R2R to nucleus (top-p= 0.95) with temperature (temperature 0.6). R2R still greatly advances Pareto frontier, with 1.4-1.5x higher AIME score over query-level routing, reaching R R1-14B score with only 8.6B average parameters. Implementation details and theoretical analysis are provided in Appendix C.I
5.3 Efficiency
Wall-clock speedup. Table 3 reports the wall-clock latency and speed for all methods on the AIME benchmark. All baselines use the official, highly efficient SGLang (29) framework and are evaluated with tensor parallelism on two NVIDIA A800-80GB GPUs. R2R uses the same thresholds as Table 2 query-level routing methods use the 14B version for comparable performance. R2R achieves 1.62x and 2.76x generation speed over R1-14B and R1-32B, respectively. Compared to query-level routing, R2R delivers 1.48-1.52x speedup. also outperforms highly optimized speculative decoding methods with tree-like drafts, which speedup mostly the current single-batch setup [16]. Further system-leve level optimization can be done to yield even greater gains for R2R. Note that, in theory. speculative decoding should exactly match the R1-32B LLM's output. However, we occasionally observe inconsistencies for very long outputs, likely due to cumulative numerical errors. We faithfully report this observation without questioning the equivalence guarantee of speculative decoding.
Computation and memory access. Compared with the LLM baseline, R2R achieves a 5.4x reduction in per-token memory access. The total computation increases only marginally due to the additional SLM and router computation. Because decoding is largely memory- -boudd,, the reduced memory traffic yields higher throughput. This substantial reduction memory traffic translates into higher throughput that outweighs the minor compute overhead. Compared with speculative decoding approaches such as Eagle2 and HASS, R2R requires about 17.0x less total computation, primarily because its immediate correction prevents the LLM from repeatedly prefilling unused tokens during periodic verification (see Figure 2). At the same time, it reduces memory access by 2.4-2.5x relative to these speculative methods, yielding more balanced compute-memory memory trade-off. More details provided in Appendix[3) B.3.22
5.4 Ablation Study
Starting from R2R in the first row of Table 4 we evaluate the effectiveness of our design by retraining the router with alternative objectives or reduced inputs. We adjust the routing thresholds pm of ablation variants to match slightly exceed the LLM rate of our original router. All experiments are conducted on the AIME benchmark with all other settings held constant. Further discussions on the trade-off between the LLM usage rate and recall provided in Appendix B.5.3

---

Table Ablation study on routing objectives and router inputs. HS denotes last-layer hidden states; Token denotes token embedding. Recall denotes the recall rate of divergent token on the validation dataset. Italicized words indicate ablation focuses.
Objective Router Input
Acc.
Recall LLM Rate #Token(K)
(aram.(B)
Cost(KB))
(atency(s)
Divergent HS+Token+Logits
55%
95%
12.4%
18.4
5.5
101
218
Different HS+ HStToken+Logits
40%
88%
13.1%
21.0
5.7
119
228
HS+ + Token
47%
85%
13.3%
18.8
5.8
109
253
Divergent
HS
42%
83%
14.1%
18.4
6.0
110
245
Routing objective. As discussed in Section 3 we categorize different next-token predictions as either neutral divergent. R2R improves efficiency by tolerating neutral differences and only routing truly divergent tokens to the L.M.. When the router is trained to use the LLM for all different tokens, fails to reach the original accuracy within the same amount of LLM usage, facing 1.4x accuracy degradation, as shown the second row of Table 4 This confirms that restricting LLM usage to only divergent tokens crucial for reducing cost while maintaining high accuracy.
Router input. As discussed in Section 1 both SLM logits and token embeddings are strong indicators of divergence to be used as router inputs. When gradually remove these features, accuracy drops by up to 1.3x, underscoring their importance. Note that while SLM logits can be computed from last-layer hidden states within the router in principle. However, doing requires the capacity of the -34M-parreeter embedding layer, which exceeds the capacity of the 56M 16M-parameter neural router.
SLM-LLM combination. We extend R2R to the Qwen3 family by fixing the LLM and varying the SLM to form three combinations:
Doven3-1.7++25 and Qwen3-4B+8B, As Figure [0 Appendix (..endiix.53335eee shows, with the LLM fixed, smaller SLM delivers a better Pareto frontier. indicates that, when selecting SLM-LLM combinations, favoring a smaller SLM is often more efficient for the same accuracy target due to lower cost. We further analyze R2R alongside orthogonal
wwen3-0.6B+88h
techniques such as MoE and query-level routing Appendix B.4
5.5 Routing Result Observation
We analyze the routing behavior of R2R on the AIME benchmark, considering finished responses within the 32K token limit. figureB[a) shows the LLM usage rate across response positions. Each response is divided into the thinking process and the subsequent reply, with positions normalized to [0, 1J. The subplot widths reflect their respective average lengths. The results show that R2R routes noticeably fewer tokens to the LLM during the reply phase. It reflects the intuition that after internal thinking, the reply itself straightforward and less demanding.
Following prior work (2I). we further segment the thinking process into sequential thoughts based on tokens such as Wait and Alternatively. Figure b) examines the LLM usage ratio within each thought. It shows that R2R relies more on the LLM at the beginning and end of each thought. This aligns with the expectation that the initial tokens set the direction for the thought, while the concluding tokens determine whether to end the thought, branch into alternatives, or continue deeper reasoning. Notably, these routing patterns are not hand-crafted but naturally emerge from router training. It helps R2R to effectively allocate LLMs for more better test-time scaling.
6 Conclusion
We introduce R2R, a token-level router that lets an SLM track an LLM's 's reasoning by correcting only path-divergent tokens. We propose fath-following labeling strategy and identify predictive signals that train a neural router for accurate token selection. On challenging benchmarks, R2R outperforms R1-14B with less than 7B average parameters, boosts SLM performance by 4.6x with under 15% LLM usage, and achieves a 2.8x wall-clock speedup over the L.M at comparable accuracy.
Limitations. Our routing strategy is chiefly tuned and tested for greedy sampling. While we have validated R2R on a limited set of alternative sampling policies, broader exploration could improve versatility, Further sestem-level optimizations are also needed to realize its full cost benefits.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation China (No. 62325405, 62104128 U19B2019, U21B2031, 61832007, 622041644 92T04444422222200 92364201), Tsinghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). thank Zinan Lin, Xuefei Ning, and Donglin Yang for their valuable discussions and suggestions. We thank Chao Xiong for his support with the SGLang interface. We also thank all the support from Infinigence-AL.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[2] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. /9437, 2024.
[3] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[4] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint
2024.
[5] OpenAL. Openai o3 and 04-mini system card, April 2025, URL httthtttt///ccle openai com/ pdf 22T1:222105000022-2777-000000--0 pdf
[6] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arX1v:2501.1229944 2025.
[7] Qwen Team. Qwen3, April 2025. URL httppps/://///ennn -it.ub.thhbb.... i0/ /llo//llg/// blog/qwen3///////o qwen3/
[8] Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https://github.. com htt.a///ooooouugggggaa/////////a
[9] DeepSeek AI. Day 6: One more - inference system overview. https: //
more htt.://...aggaeaa.eee.a.aaaaaaaa Accessed: 2025-05-03.
2025.
[10] Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL http::::sst.tw/attteess com/
Hatassts/hemsshverrabbol
aime pwwplem-ooooe-po--... prw.lemsse--eeeeeeeeeeee------ee --w--193----0 2024
[11] Isaac Ong. Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms from preference data. In The Thirteenth International Conference on Learning Representations, 2024.
[12] Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-basee based router for Ilm selections. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. holt:tttt:///penrreee
Poster presentation.
[13] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Fagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint wrXiv:2401.1507, 2024.
[14] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint 268.188.0..6888. 2024.
11

---

[15] Lefan Zhang, Xiaodan Wang. Yanhua Huang, and Ruiwen Xu. Learning harmonized represen- tations for speculative sampling. arXiv preprint arXiv:2408. 15766, 2024.
[16] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprini arXiv:250300184) 2025.
[17] Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21661 2025.
[18] Yang Sui, Yu-Neng Chuang. Guanchu Wang, Jiamu Zhang, Tianyi Zhang. Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503.16419, 2025.
[19] Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502.18600 2025.
[20] Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware IIm reasoning. arXiv preprint arXiv:2412.18544 2024.
[21] Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of ol-like 1lms, arXiv preprint arXiv:2501. 18585, 2025.
[22] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[23] Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-provee Advancing theorem proving in lims through large-scale synthetic data. arXiv preprint arXiv:2405. 14333, 2024.
[24] Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311. 086692, 2023.
[25] Xinyuan Wang. Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. Mixllm: Dynamic routing in mixed large language models. arXiv preprint arXiv:2502. 18482, 2025.
[26] Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028. 2002.
[27] Fengxiang Sun. jieba: Chinese text segmentation.
2013. Accessed: 2025-04-01.
[28] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[29] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2023.
[30] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[31] Bespoke Labs. lation. Bespoke-stratos:
The unreasonable effectiveness of reasoning distil-
reasoning- distilitn,,tillationn, 2025. Accessed: 2025-01-22.
12

---

[32] Huan Ma, Jingdong Chen, Guangyu Wang, and Changqing Zhang, Estimating Ilm uncertainty with logits. XrXiv preprint arXiv:2502.00290, 2025.
[33] Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song. Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025.
[34] Rodolfo Zevallos, Mireia Farrús, and Núria Bel. Frequency balanced datasets lead to better language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7859-7872, 2023.
[35] Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F AbouElhamayed, Yueying Li, and Mohamed S Abdelfattah. Splitreason: Learning to offload reasoning. arXiv preprint arXiv:2504.16379. 2025.
[36] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
[37] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024.
[38] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint S0.iv::2001.000850 2024.
[39] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0 0/068, 2022.
[40] Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du. Wenjun Gao, Qihao Zhu, Dejian Yang. Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. Deepseek-proverr-V/ Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search, 2024. URLhttpe //arxiv..//ii////aa /bs/ 2408 08152]
[41] Dongwon Jung, Wenxuan Zhou, and Muhao Chen. Code execution as grounded supervision for IIm reasoning, 2025. //tps/////// orgo/o/////o /ttpaa//.....//////////////eee 10343
[42] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024. URL /attps://arxils owtg/rgg////eeeeoooooooooeeeeeee abs/aab//2 11939
[43] Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with mmlu?, 2025. httpss::: http:/:://xxvvo/o////////
04127
[44] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training computc-optimal large language models. arXiv preprint arXiv:2203.
2022.
[45] Fan Yang, Xinhao Yang, Hongyi Wang, Zehao Wang, Zhenhua Zhu, Shulin Zeng, and Yu Wang. Glitches: Gpu-fpga Imm inference through a collaborative heterogeneous system. In 2024 IEEE High Performance Extreme Computing Conference (HPEC), pages 1-7. IEFE, 2024.
[46] Wenheng Ma, Xinhao Yang, Shulin Zeng, Tengxuan Liu, Libo Shen, Hongyi Wang, Shiyao Li, Ke Hong, Zhenhua Zhu, Xuefei Ning. Tsung- Yi Ho, Guohao Dai, and Yu Wang. Cd-llm: A heterogeneous multi-fpga system for batched decoding of 70b+ lims using a compute-dedicated architecture, ACM Trans. Reconfigurrblle Technol. Syst, October 2025. ISSN 1936-7406. doi: 10.1145/3771288, URL URL https: htopps://do.. /ww.///iiioiaa org. 10. 1145/3771288
Just Accepted.
13

---

[47] Zixuan Zhou, Xuefei Ning. Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey on efficient inference for large language models. ArXiv, abs/2404. 14294, 2024.
[48] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprini arXiv:2309. 17453, 2023.
[49] Tianyu Fu, Haofeng Huang, Xuefei Ning. Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 11200611900 14909, 2024.
[50] Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, and Jianfei Chen. Spargeattn: Accurate sparse attention accelerating any model inference. In International Conference on Machine Learning (ICML), 2025.
[51] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware-aligner and natively trainable sparse attention. arXiv preprint arXiv:2502. 1089, 2025.
[52) Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv: /3/89, 2025.
[53] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[54] Shiyao Li, Xuefei Ning, Luning Wang. Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guo- hao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 18480-28522 2024.
[55] Jintao Zhang, Jia Wei, Pengle Zhang, Jun Zhu, and Jianfei Chen. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025.
[56] Jintao Zhang, Haofeng Huang, Pengle Zhang, Jia Wei, Jun Zhu, and Jianfei Chen. Sageatten- tion2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML), 2025.
[57] Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training. arXiv preprint arXiv: 1594, 2025.
[58] Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision ky cache quantization for long-cot llms. arXiv preprint arXiv:2505. 1.0.:.0005...00, 2025.
[59] Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: En- hancing small-monl model reasoning with large model guidance inference time. arXiv preprint arXiv:2504.1 12329, 2025.
[60] Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specrea- son: Fast and accurate inference-ting compute via speculative reasoning. arXiv preprint arXiv:2504.0 07891, 2025.
[61] Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P Xing, Hongyi Wang, and Huaxiu Yao. Citer: Collaborative inference for efficient large language model decoding with token-level routing. arXiv preprint arXiv:2502.01976 2025.
[62] Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guiderd guided speculative decoding for efficient Ilm reasoning. arXiv preprint arXiv:2501.1933 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims of R2R to let the SLM follows the reasoning path of LLM with the routing of divergent tokens.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The theoretical result and proof are provided in Sections 3.1 3.2 and Ap-
pendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We specify all the training and test details in Section A and our code will be open source soon.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is contained implementation materials and will be open source soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppclccccuuddeeeeoo Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Section A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We set the temperature to zero to allow the reproducibility of the experiment and report the actual latencies and speeds in Section 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We use NVIDIA A800-80GB GPUs, with eight for data generation and two for inference, detailed in Section [.1]an and Appendix B.3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research does not contain potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: R2R is general- purpose method aimed at improving the efficiency of rea- soning LLMs. It is not tailored to any specific application domains that might pose clear positive or negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used are properly cited and elarified in Section 5.1 and Appendix A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is contained in implementation materials and has been open source. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We use the LLM verifier when generating the training data. The way of LLM usage is detailed in Sections and
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Additional Experiment Setups
A.1 Router Architecture
Inputs projection. At each decoding step, we use the hidden states of the last layer, the top 100 logits with the highest values and the embeddings of the predicted token from the SLM to generate the routing result. We first apply linear projections align their dimensions with the hidden states and then concatenate the features from the logits, hidden states, and token embeddings. Finally, we use another linear layer project the concatenated features to match the input feature dimension of the model backbone.
Neural network backbone. For the router architecture, adopt six-layer feed-forward network (FFN) with residual connections between blocks as the backbone, using a hidden size of 1024. The architecture of each FFN follows the common design used in previous LLMs [39 Each block begins with LayerNorm for input normalization, followed by a pair of linear projections forming an expand- then-chnnrace structure with expansion factor of 4. Dropout is applied to each linear layer, with a GELU activation function between them. These blocks are connected using residual connections. At the end of the last block, we apply an additional layer normalization and linear layer to convert the output to a single value, followed by a sigmoid function to produce the normalized prediction of the router. A predefined threshold Pt between 0 and 1 is used for generating binary results from the router output. Predictions above the pm are considered that current tokens diverge from the LLM's reasoning path.
A.2 Routing Data
Training dataset. Our training data for the router are sourced from tasks across three distinet scenarios: mathematics, code, and question answering (QA). The mathematics problems are drawn from the American Invitational Mathematics Examination (AIME) [10), covering the years 1983 to 2022. Code and QA problems are sampled from Bespoke-Stratos- 7k dataset [31]. We use only the formatted questions from these datasets as prompts and generate responses using DeepSeek-RL Distill-Qwen-322 with the temperature set to 0 and maximum generation length of 32,766 tokens. Only responses that contain an end-of-sequence (EOS) token within the specified length are retained as effective samples, which will be used for subsequent stages of our data generation pipeline, as discussed in Section
Validation dataset. Our validation dataset are constructed in the exact same way as the training data, but with different queries. The validation dataset comprises all 30 problems from AIME 2023, 69 coding problems from the Bespoke-Stratoo Sirikts--tttts--11 - 17k dataset that are excluded from the training set, and 60 QA problems selected from the GPQA-Extended [36] dataset.
A.3 Training Scheme
Loss function. Due to the significant class imbalance in the training data, adopt the weighed BCEWithLogitsLoss as our loss function. The weight of each class is calculated inversely proportional to its frequency in the class, which encourages the model to pay more attention to underrepresented classes.
Training hyperparameters During training, we employ the Adam optimizer with hyperparame- ters 31 0.9 and B2 0.999. The learning rate is set to 5x 5x100-, with a dropout rate of 0. and a weight decay of 5 x 10-1 We train the neural network with float32 precision. The router is trained for up to 50 epochs using a batch size of 1024, with early stopping applied based on a patience of 10 epochs. Validation is performed at every epoch. We adopt the checkpoint corresponding to the best-performing epoch on the validation set as the final router used.
Threshold selection. After training, we use the validation dataset to select preferred threshold. We pass the pre-collee pre-collected neural router inputs from the validation dataset through the neural router and record the predicted divergence probabilities. By sweeping Pm from 0 to 1, we analyze how different thresholds affect the LLM usage rate and average parameter size., as shown in Figure g This process is efficient, as all router inputs (SLM logits, token embeddings, and last-layer hidden states) are pre-collected and evaluated in a single pass. During inference, given any user-defined average parameter budget, we set the threshold to meet the target budget accordingly.
22

---

A.4 Routing System Implementation
Model initialization. The routing system consists of three components: SLM (RI-1.5B), a LLM (RI (R1-32B), and a router model. The SLM is loaded onto single GPU (GPU 1) using the SGLang scheduler, with the mem comm__racioon_teee static set to 0.15. The LLM employs tenser-paralle inference distributed across two GPUs (GPU O and GPU I) via SGL.ang schedulers managed by PyTorrch' distributed multiprocessing framework with the NCCL backend, with the mem cre__raction.oo caaction_statii set to 0.80. The router model is directly loaded onto GPU 0 using standard PyTorch, independent of the SGLang interface. Prior to inference, each of SLM and LLM individually warmed up using simple inputs stabilize GPU kernels and caches, ensuring consistent inference latency.
Inference workflow. During each inference step, the workflow begins with the SLM decoding single token, returning the corresponding last-layer hidden states and output logits. The router generates divergence probability based on these outputs of SLM. the probability surpasses the predefined threshold, the L.LM is activated to extend the sequence by one token. Specifically, new request, constructed from the input token IDs, is placed into the L.LM's input queue. Subsequently, new schedule batch is initialized for the LLM, explicitly setting the forward mode EXTEND and allocating appropriate memory for handling input sequences. The system maintains prefix indices to track processed tokens, enabling efficient token management between models. When the LLM extends a token, it is communicated back through an output queue to replace the SLM's predicted token. A token manager actively tracks the sequence states during the generation process, managing active sequences and handling termination conditions effectively. At each token position, the dynamic routing mechanism assesses model outputs, determines the appropriate routing decision, and updates sequence states accordingly. This iterative process continues until a sequence is completed or reaches the predefined maximum token limit.
B Additional Experiment Results
B.1
Verifier Robustness
B.1.1 LLM Verifier Against Human Verifier
Setup. We conducted a human evaluation to validate the verifier's 's reliability. Specifically, we recruited four undergraduate students independently label 1.357 differences as either neutral or divergent. The differences are between R1-1.5B and R1-32B on the first six AIME-2024 questions. Three annotators' labels determined the ground truth for divergence by majority voting (general divergence, positive rate: 24.8%) and unanimous consent (core divergence, positive rate: 11.9%). The fourth annotator's labels, along with several LLM verifiers, were then compared against these ground truths. All annotators and LLM verifiers received identical contexts and instructions.
Results and discussion. Our selected verifier (Owen2.5-72B) closely matches human expert perfor- mance. The detailed results are shown in Table 5
Table 5: Verifier performance against majority vote (General Divergence, Positive Rate: 24.8%) and unanimous consent (Core Divergence, Positive Rate: 11.9%) ground truths.
Majority Voting (General)
Unanimous Consent (Core)
Verifier
Positive Rate
Accuracy
Recall Precision
Accuracy Recall Precision
Human
0.82
0.84
0.59
0.76
0.96
0.32
0.35
Qwen2.5-72B
0.84
0.88
0.62
0.76
0.97
0.33
0.35
Qwen2.5-7B
0.78
0.85
0.54
0.71
0.94
0.28
0.39
Qwen2.5-3B
0.75
0.69
0.50
0.72
0.75
0.26
0.34
B.1.2 Impact of Verifier Quality
Setup. Next, we examined the sensitivity of our method to verifier quality. We applied the sentence- level path-following routing method [Equations[3p to the AIME 2024 benchmark across verifiers (Qwen2.5-3B to Qwen2.5-72B) within 8K token budget.
23

---

Results and Discussion. Our analysis reveals that even moderate-recit verifiers (e.g., Qwen2.5-3B) still significantly improve reasoning path guidance compared the distilled 7B baseline. Note that the current verifier prompt is tuned for the Qwen2.5-720 verifier. Empirically, smaller verifiers should benefit from prompts biased towards classifying ambiguous cases as divergent, thereby enhancing recall.
Table 6: Impact of verifier quality path-following routing performance AIME 2024.
Type
Model Param. (B)
Labeling Verifier
A1ME'24 #Acc. @8K
LLM Baseline
32
17
SLM Baseline
1.5
2
Distilled
8
Path-following
Qwen2.5-72B
16
Path-following
2.5
Qwen2.5-7B
12
Path-following
2.2
Qwen2.5-3B
11
B.2
Generalizability
B.2.1
Data Labeling Method Generalizability
The labeling method of R2R, relying on semantic comparison rather than formal verification, can easily generalize across tasks and achieve high performance even in some out-domain tasks. Unlike methods requiring task-specific external tools (e.g., formal proofs 40), code execution 4I]). R2R uses an LLM verifier to identify general semantic divergence in meaning, reasoning, logic, or conclusions
In practice, we apply the identical labeling strategy and verifier prompt across closed-form math, coding tasks, and open-ended QA tasks. This consistency enables router naturally generalize to unseen tasks during inference, as demonstrated later.
For tasks with subjective divergence criteria that are challenging to identify semantically within a single sentence, the continuation length can be extended. This adjustment gradually transitions from sentence-level routing to full routing, the latter of which only requires overall response quality evaluation, which is a feasible requirement commonly met for LLM tasks.
B.2.2 Router Generalizability
Generalize across benchmarks. Our router leverages outputs from the SLM (e.g., logits) to predict token divergence. Benefiting from the inherent generalizability of SLMs, indicators such as logits entropy robustly identify divergent tokens across different tasks. To validate the router's general- izability, we directly apply our router, trained for math, QA, and code, to additional benchmarks: Hard for Dialog [42]. and the Philosophy split from MMLU-Redux [43), which are never included our training dataset. As table 7 shows, although
cost-accuracy trade-off gains are bit less pronounced on out-of-domain tasks, R2R nonetheless exhibits strong generalization, which continues to outperform the 14B model with less than 7B parameter size.
Table 7: Performance of R2R Arena-Hard for Dialog and MMLUURedux-Philosse
Dialog
Philosophy
Type
Model(s)
Score
Param.
Cost
Acc.
Param.
Cost
LLM
R1-32B
5.0
65.9
79
32
32.2
SLM
R1-1.5B
0.2
1.5
14.3
38 %
1.5
8.7
Distill
R1-7B
0.3
35.0
57 %
7
20.6
Distill R1-14B
2.2
14
56.5
77 %
14
18.5
R2R
Ours
2.8
6.1
40.9
81 %
6.7
8.3
24

---

Generalize to different LLMs. We test whether a router, trained on data from specific SLM- LLM pair (e.g., 0.6B-32B), can be effectively applied different pair (e.g., 0.6B-8B) without retraining. The experiments are conducted with the Qwen3 model family (0.6B, 8B, and 32B), tested on the AIME (2024+2025) benchmark with a target recall of 0.95 for divergent tokens. The results are summarized in Table g Our empirical results indicate acceptable performance when directly substituting the LLM of a trained router without additional retraining. This can be attributed to the common divergence patterns shared by strong LLMs paired with the same weaker SLM.
Table 8: Router generalizability on the AIME benchmark. Param. denotes the average activated parameters per token in billions; Cost is the average output tokens (thousands) per query multiplied by average parameters (billions).
Type Model(s)
Router Variant
Acc.
Param.
Cost
SLM Qwen3-0.6B
12%
0.6
15
LLM Qwen3-8B
67%
8.0
130
LLM
Qwen3-32B
75%
32.0
469
Trained on 0.6B+8B
65%
2.7
49
R2R
0.6B+8B
Generalized from 0.6B+32B
53%
2.6
50
Trained on 0.6B+32B
67%
10.2
154
R2R
0.6B+32B
Generalized from 0.6B+8B
70%
9.9
151
B.3
Efficiency Analysis
B.3.1
Data Generation Overhead
Table 9: Latency and GPU usage across different stages of data labeling.
Stage
Latency (hours)
#GPU
GPU Hour
LLM response
35
8
280
SLM Prefill
0.1
8
0.5
LLM Continuation
7
8
56
Verify
14
8
112
Total
56
8 448
As Table Table shows, our four-stage pipeline completes in 56 (448 GPU hours). The LLM response stage dommates runtime (35 h, 280 GPU hours), but it can be mitigated by directly utilizing the responses of LLMs from open-source SFT datasets, provided they were generated by the same LLM used for routing. The SLM prefill step is highly efficient, requiring only 0.1 h of wall clock time. The subsequent LLM continuation and verification stages take h (56 GPU hours) and 14 (112 GPU hours), respectively. Compared to downstream tasks, the overall data generation pipeline remains relatively lightweight and efficient.
B.3.2 Computation and Memory Access
Following prior work 44 46
we analyze the computational and memory access overhead of R2R against various baselines on the AIME benchmark. Table presents a detailed comparison total computation (TFLOPs), total memory access (TB), and the average memory access per token (GB).
Given that the decoding process large language models is predominantly memory-bound, the volume of memory access is a critical factor influencing overall throughput. As shown in the table, our method, R2R, markedly reduces total memory access to just 216 TB, a 4.5x reduction compared to the R1-32B model. This efficiency is achieved with only modest increase in total computation, which primarily arises from the additional KV-ccche updates required when switching between the LLM and SLM.
In contrast, speculative decoding approaches such as Fagle2 and HASS incur substantially higher computational costs-more more than 22x that of R1-32B. This overhead is direct consequence of their
25

---

Table 10: Computational and memory overhead on the AIME benchmark. R2R demonstrates significant reduction in memory access compared to the large model (R1-32B) and speculative decoding methods, with only a marginal increase computation.
Total Computation
Total Memory Access (TB)
Avg. Memory Access
Model
(TFLOPs)
Per Token (GB)
R1-1.5B
157
104
3.7
R1-14B
630
503
29.4
R1-32B
1136
966
63.6
Eagle2
25490
515
29.6
HASS
26809
544
28.9
R2R
1502
216
11.7
complex tree-structured draft and verification processes, which can involve up to 60 tokens per cycle. Despite this significant computational demand, these methods also achieve considerable throughput improvements by effectively reducing the memory access bottleneck during the decoding phase. R2R, however, provides a more balanced trade-off, achieving significant memory savings without the extreme computational overhead of speculative methods.
B.3.3 Inference Latency Breakdown
We integrated R2R into the production-level SGLang serving infrastructure, which is demonstrated by the high absolute throughput results for method and the baselines. To assess the latency implications of our token-level routing, we performed a detailed runtime breakdown. The analysis was conducted the AIME benchmark, averaged over 60 questions, using A800 GPUs.
The results, shown Table TT indicate that the token-level routing decisions made by our 56M parameter router introduce minimal latency overhead (5.96%). Given the small size of the neural router, we anticipate that further runtime improvements can be achieved through additional system- level optimizations.
Table 11: Runtime breakdown of the R2R system components on the AIME benchmark.
Component
Percentage of Total Time
Router
5.96%
LLM (R1-32B)
64.97%
SLM (R1-1.5B)
26.77%
Others
2.30%
B.3.4 LLM Activation Intervals
The cost efficiency the routing system largely depends on how frequently the LLM is activated during generation. Since the EXTEND operation of the LLM is inherently compute-bounded when consecutive LLM calls are spaced further apart. In other words, given a fixed LLM activation rate, longer intervals between two LLM invocations lead to better GPU utilization and less scheduling overhead per generated token.
To better understand this effect, we analyze the distribution of the interval between consecutive divergent tokens. Figure illustrates these distributions for both the training dataset and actual inference traces on AIME25. As shown, the actual distribution closely aligns with that observed training dataset, suggesting that router generalizes well to real inference scenarios. More importantly, both distributions exhibit long-tailed shape, indicating that divergence events occur sparsely and irregularly, which allows R2R to sustain high computational efficiency without overusing the LLM.
26

---

20
8.24
Mean:
Mediar: 4.00
Mediam: 2.00
15
Peouuncy
25
1.
20
Between Consecutive Divergent
Interval Between Conserutive Refenence
Figure 7: (a) Divergent token interval distribution for training dataset; (b) Reference token interval distribution on AIME25
B.4
Comparison and Compatibility with Orthogonal Techniques
B.4.1 R2R and Mixture-of- (MoE)
We compare our method, R2R, with the Mixture-of -ixture-of-Experts (MoE) architecture, another prominent approach for efficient model scaling. This section begins with conceptual comparison, followed by empirical results, and concludes by discussing how the two methods can be integrated to create more favorable performance-efficien efficiinfcffcffceeeeeeeeeeeeeeee trade-off.
Conceptual comparison. As noted, both MoE and R2R leverage partial activation, but they differ in key design aspects, as summarized in Table T2 This conceptual distinction motivates their potential integration, which explore next.
Table 12: Conceptual comparison between MoE and R2R.
Design Aspect
MoE
R2R
Partial activation
Yes
Yes
Routing granularity
Fine-grained (Experts)
Coarse-grained (Models) Different parameters per model Router training only
Subject model sizes Training overhead Supervision
Equal parameters per expert Full training from scratch
Next-token predictions
Explicit routing decisions
Empirical comparison. We evaluate the MoE model (0wen333BBA3) and an R2R model (routing between Qwen3-0.6B
Qwen3-8B) on the AIME benchmark. The 60 questions are split into Easy (28) and Hard (32) subsets based whether the question ID is less than or equal to 7. The results in Table 13 highlight the complementary strengths of each approach. Thanks to extensive pretraining, MoË models can match dense model performance on challenging tasks. However, their fixed per-token overhead limits efficiency on simpler inputs. In contrast, R2R dynamically adjusts computation based token-level divergence, achieving lower cost easier examples, though its lighter training cannot match MoE performance on harder questions given the same activated parameters. Given their complementary advantages, explore the integration of both methods.
Table 13: Empirical comparison of R2R and MoE on the AIME benchmark, split by difficulty.
Easy
Hard Acc.
Type
Model(s)
Acc.
Param.
Param.
R2R
Qwen3-0.6B + Qwen3-8B
93%
2.9
41%
2.6
MoE
Qwen3-30BA3B
93%
3.3
59%
3.3
MoE for R2R. R2R is directly compatible with efficiency-optimizzed models like MoEs, We validate R2R's performance using the Qwen3-30BA3R MoE. as the LLM, shown in Table Combining R2R with an MoE model significantly improves the Pareto frontier, achieving robust accuracy at extremely low average activated parameters.
27

---

Table 14: Performance of R2R when using an MoE model as the LLM.
Type
Model
Acc.
Param.
Cost
Dense
Qwen3-0.6B
12%
0.6
15
Dense
Qwen3-1.7B
37%
1.7
33
Dense
Qwen3-32B 0.6B + 32B
75%
32.0
469
R2R
67 %
10.2
154
MoE
Qwen3-30BAAB 0.6B + 30BA3B
75%
3.0
51
R2R
68%
1.3
22
R2R for MoE. Conversely, R2R's 's principles can enhance future MoE designs. Current MoE methods generally uniform-sized experts; introducing mixed-sized experts could enable query-adaptive activation. Furthermore, supplementing an MoE's training objective with explicit divergence or difficulty-based routing supervision from R2R may encourage more adaptive usage of larger experts for only the most critical decoding steps. Although retraining MoE models is beyond this paper's scope, this presents an exciting direction for future research.
B.4.2 R2R and Query-level Routing (QR)
Query-level routing (QR) is methodologically orthogonal to R2R. Consequently, the composition of the two is natural and well-motivated design choice. We investigated combining R2R with query-level routing methods using the Qwen3 model series with varied sizes, evaluated on the AIME benchmark. Specifically, treated R2R-S (0.6B+8B) and R2R-L (0.6B+32B) as the SLM and LLM, respectively, for query-level routing.
Table Performance of R2R when combining R2R and query-level routing (QR).
Type
Model
Ace.
Param. (B)
Cost (KB)
SLM
Qwen3-0.6B
12%
0.6
15
LLM
Qwen3-8B
67%
8
130
LLM
Qwen3-32B
75%
32
469
R2R-S
Qwen3-0.6B + 8B
59%
2.1
40
R2R-L
Qwen3-0.6B + 32B
67%
10.2
154
R2R-M
Qwen3-4B + 8B
65%
5.4
95
QR-BERT
Qwen3-0.6B) + 32B
25%
5.4
230
QR-LLM
Qwen3-0.6B + 32B
23%
5.4
238
QR-MF
Qwen3-0.6B + 32B
32%
5.4
216
QR-SW
Qwen3-0.6B + 32B
30%
5.4
220
QR-BERT
R2R-S R2R-L
63%
5.4
179
QR-LLM
R2R-S R2R-I
67%
5.4
175
QR-MF
R2R-S R2R-L
58%
5.4
181
QR-SW
R2R-S R2R-L
62%
5.4
183
From Table [5] we can conclude that (1) R2R models generally serve as superior SLM/LLM candidates for query-level routing compared to original single-mode modell setups, consistently advancing Pareto frontier; (2) The interaction between medium-sized R2R models and query-level routing involving smaller or larger R2R-LLMs is empirically non-trivial. This observation suggests exciting opportunities for future research into sophisticated routing strategies, which combine the search space of query and token-level routing methods.
28

---

threshold Recall
threshold Param.
Param. Recall
100
100
80
80
15
Recall
60
Param.
2.0
40-
40-
6.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.4
0.6
0.8
1.0
10
15
20
threshold
threshold
Param (8)
Figure Relationship between the routing threshold, recall for divergent tokens, and average parameter size. The average parameter size is computed based on the positive prediction rate of the router at each threshold.
B.5
Additional Discussions and Analysis
B.5.1
Influence of Routing Threshold
Figure visualizes how the routing threshold Pn: affects the average parameter size on the validation dataset. In our experiments, we select thresholds based on the user-specified average parameter budget (e.g.. 6B) measured on the validation set. However, the threshold does not strictly guarantee the same average parameter size during inference, particularly when query difficulty varies significantly from the validation set. Empirically, we observe minimal variance between the target and actual parameter sizes, as the difficulty of our validation and evaluation datasets is generally well aligned. For tasks that are much easier or harder than those considered this paper, users can construct tailored validation set using our data labeling pipeline to determine an appropriate threshold.
Beyond analyzing the relationship between threshold and average parameter size, we also examine its effect on divergent token recall. This links the average parameter size to the ability to recall divergent tokens. As shown in Figure recall rises rapidly with increasing average parameter size, demonstrating the strong predictive performance of our router model.
B.5.2 Sentence-level Path-following Routing
The sentence-level path-following assumption, while necessary manage the vast search space, could be limiting in certain scenarios. For instance, an SLM might rearrange sentences in paragraph while still preserving a valid reasoning path, yet our sentence-level verifier might incorrectly flag this as a divergence. While such mislabeling of a neutral continuation as divergent does not degrade final performance, can introduce unnecessary computational overhead by triggering the LLM more often than required.
To empirically evaluate the impact of this assumption, extended routing strategy more general N-sentence path-following approach. In this setup, the continuation-and-verfffee deseribed in Section 3.2 is not stopped by the first sentence-ending token, but continues for N sentences. As N increases, the strategy transitions smoothly from our default sentence-level approach (N=1) towards the full path-following routing defined in Section 3.1 which is guaranteed find the optimal path (see Appendix E
process
We conducted experiments found that increasing N from to 5 only yields a modest reduction the measured divergence rate, from 2.91% % 2.66%. This corresponds to marginal decrease of 0.08B in the average activated parameters required to maintain the same level of accuracy.
subset of the AIME dataset measure the effect of varying N. We
Given the limited performance gain relative to the substantial increase in data labeling complexity and cost, we empirically selected N=1 (sentence-level routing) as the most practical and efficient default setting for our framework.
29

---

B.5.3 Ablation Study Results
To further investigate the impact of different routing strategies, we perform an ablation study that analyzes the relationship between Positive Rate, defined as the fraction of tokens routed the LLM, and Recall, defined as the proportion of correctly identified positive samples in the training dataset.
Routing objective. As illustrated in Figure 9 R2R (default) consistently achieves higher recall across all positive prediction rates compared to the other variants, demonstrating that its routing policy is more accurate identifying informative tokens. The different variant, which substitutes the semantic divergence metric with a simpler criterion that only considers mismatched predictions, exhibits a noticeable drop in recall. This degradation highlights the importance of fine-gras ane-grained semantic alignment in accurately distinguishing divergence tokens and maintaining routing precision.
Routing input. Figure also shows that IIS and ISkeo variants, which rely solely on hidden-state distance or only combine it with current token ids, yield similar but lower recall curves, indicating that low-level representation similarity alone insufficient capture reasoning divergence and prove the significance of the logits during routing. In contrast, R2R achieves both higher efficiency and better coverage, maintaining near-optimal recall with less than 14% of tokens routed to the LLM.
SLM-LLM combination. To analyze the performance of R2R under difference model pairs, we apply it the Qwen3 family with fixed LLM selection (Qwen3-8B) and vary the SLM to form three different SLM-LLM combinations: Qwen3-0.668 -06.BB Qwen3-1.7B+8Bh and Qwen3-4B+8B, All combinations were tested AIME with identical settings. Figure TO shows the relationship between accuracy and the average activated parameters per token: For a fixed LLM, smaller SLMs yield better Pareto frontier, matching accuracy lower cost achieving higher accuracy the same cost. The intuition is that, for identical tokens that can be predicted by multiple SLMs, routing them to the smaller SLM can preserve correctness of the prediction with less computational cost, thereby improving the overall
efficiecyy tradc-off.
Interestingly, found that all seven AIME questions that the SLM answered correctly were also answered correctly by the LLM, indicating that the LLM consistently outperforms the SLM rather than specializing in different subset of math problems.
These results validate that the semantic divergence criterion used in R2R provides a more discrimina- tive signal for routing, effectively balancing accuracy and efficiency in the overall system. Moreover, appropriate selection of the SLM-LLM combination can further improve the cost-accuracy Pareto frontier.
80
0.9
3.7
Recall
Acturacy
0.5
40
0.4
30
Ablation Va
R28 (debsut))
20
0.68-485
0.
Diffeeent
1.70-88
HS+ Token
10
48:89
3.0
20
(96)
Activated Parameters Per To kon (B)
Figure 9: Relationship between the ratio of LLM usage and the recall of divergent tokens the validation dataset.
Figure 10: Scaling of accuracy versus activated parameters per token, evaluated on AIME across different model combinations in Qwen3 family
B.6 Systematic Failure Analysis
To identify systematic failure modes, we analyzed the response status of R2R (routing between RI-1.5B and RI-32B without sampling) on the AIME and GPQA benchmarks. As shown in Table [6] the primary failure mode occurs when R2R cannot complete its reasoning within the 32K token limit. This typically happens due to repetitive reasoning patterns that were not encountered during the router's 's training phase. Additionally, observe that R2R tends to invoke the LLM slightly
30

---

more frequently on the GPQA benchmark. This is particularly noticeable for queries involving rare tokens, such as complex protein or chemical compound names, where the SLM's uncertainty is higher, triggering the router.
Table 16: Systematic failure analysis of R2R on AIME and GPQA benchmarks.
Wrong (Same as)
Wrong (Diff. from)
Benchmark
Correct
Unfinished
LLM
SLM
LLM
SLM
AIME (60)
33
25
2
GPQA (198)
87
85
6
3
20
23
Total (258)
120
110
3
22
25
B.7 Performance-Cost Trade-off
AIME
LiveCodeBench
GPOA
60
50
50
Dus
40
Di2.i
30
30
0R-54
Q84ERT
GR:L1M
20
SLM
10
10
10
100
200
Billion Parameters))
Parameterss
Cost (K Tokons Billio Parametars)
Figure 11: Scaling of accuracy versus total cost, evaluated across AIME, GPQA, and LiveCodeBench. R2R advances the Pareto frontier beyond distillation and query-level routing methods.
Figure 11 illustrates the trade-off between accuracy and the total cost of generation. As defined in
Section 5.1 the cost metric is calculated as the average number output tokens multiplied by the
average parameter size, serving as a hardware-agnostir indicator of latency across methods. R2R consistently outperforms both query-level routing methods and distilled LLMs, establishing a new Pareto frontier in performance-cos. trade-offs.
C Additional Design Discussions
C.1 Extend R2R to More Sampling Methods
C.1.1 Path-Following Routing Under Sampling
R2R naturally extends to sampling-bas -based decoding by adapting divergence labeling to a probabilistic setting. Specifically, the deterministic continuation sequence pair (S,, St) from Equations 3 are replaced by & sampled continuation pairs under a chosen sampling method. Divergence is then evaluated for each sampled pair. The routing decision for each token then depends whether the overall divergence probability exceeds predefined threshold:
((V Si;) Oli e (0, Puresholld
If setting Phrreshold the LLM's self-divergence probability P(V(S, Si) = 0). it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone.
C.1.2 Efficient Data Generation
Due sampling stochasticity, each mismatched token can branch into multiple continuations. However, sampling multiple continuations for divergence labeling is computationally expensive. Since continuations are sentence-level, we empirically observe that the divergence decision primarily depends the first differing token between the SLM and LLM samples, rather than the subsequent
31

---

continuations. To efficiently approximate probabilistic routing, we therefore only sample for next- token generation yi ((,((5 and yı ((: while leaving continuations deterministic.
Given the extensive volume of tokens, we set k = and Phreshold = 0.5, simplifying the setup. This approximation incurs overhead comparable our datt-generation pipeline adapts to the new setup by using sampling-base baaiing--aedddd generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.
greedy data generation pipeline. In implementation,
Table Statistics of tokens difference and divergence across query types the sampling-base training dataset.
Type
#Query
#Token
#Different
Diff. Rate
#Divergent
Div. Rate
Math
862
7.3M
858K
11.8%
438K
6.0%
Code
987
6.8M
1.2M
17.5%
627K
9.3%
QA
805
2.1M
443K
21.3%
231K
11.1%
Summary
2654
16.1M
2.5M
15.5%
1.3M
8.0%
Using this modified pipeline, generate 16.1 million routing labels, covering topies of math, coding, and QA with queries from the Bespoke-Stratte dataset. Table 17 summmaizzs the statistics of the generated training dataset with the sampling method. Compared with the non-sampled dataset in Table I sampling increases the average divergence rate by only 3.1 %6 which remains low overall, indicating divergence patterns consistent with greedy decoding.
C.1.3 Experiment Results
For preliminary experiments, we use DeepSeek-RI RI's recommended sampling settings (temperature 0.6, top-p= 0.95) for both dataset generation and evaluation. The neural router remains unchanged, as it already takes the sampled token's 's embedding as input.
We evaluate the extended R2R the AIME benchmark, reporting its pass 1 accuracy 16 independent samples per problem, and compare it against query-level routing baselines following Section [.]] As shown in Table 18 R2R continues to advance the Pareto frontier, achieving 1.4-1.5x higher AIME scores than query-level routing methods and matching the R1-14B performance with only 8.6B average activated parameters. Extending R2R stochastic decoding preserves its Pareto optimality relative to both distilled models and query-level routing approaches.
Table 18: Performance and efficiency comparison on AIME and methods with sampling. Param. denotes the average activated parameters per token in billions; Cost the average output tokens (thousands) per query multiplied by average parameters (billions).
Method
Acc.
Param. (B)
Cost (KB)
R1-1.5B
27%
1.5
24.3
R1-32B
61 %
32.0
384.8
R1-14B
58%
14.0
170.5
QR-SW
39%
9.2
135.5
QR-LLM
38%
9.2
138.6
QR-BERT
40%
9.4
136.8
QR-MF
41%
9.1
130.5
R2R
58%
8.6
115.3
C.2 Routing Algorithm Details
As illustrated in Algorithm 1 the objective our routing algorithm to identify and correct path- divergent tokens during inference by using predictions from large language model (LLMM.. Both SL.M and L.M perform greedy decoding, and token is considered identical if both models produce the same prediction.
32

---

When the SLM and LLM outputs differ, the algorithm triggers a continuation process for each model: the SLM and LLM, respectively, continue generating tokens, starting from their initial divergent prediction, until a special separator (SEP) token is produced. These continuations yield two complete sequences that differ only the initial divergence point and subsequent tokens.
To assess whether this divergence impacts reasoning, a separate LLM-based verifier is employed. This verifier receives the two generated sequences and outputs a binary decision: if the sequences are semantically neutral, and I if they diverge significantly in meaning or logic.
If the verifier outputs 0 (neutral), the router accepts the SLM's prediction. However, if the verifier outputs (divergent), the algorithm corrects the current token by adopting the LLM's prediction, thus preventing further drift from the intended reasoning path.
This approach ensures that the system maintains high alignment with LLM reasoning, while minimiz- ing unnecessary reliance on the LLM by routing to the more efficient SLM whenever possible.
Algorithm 1 Path- Foll-Follooing Routing
Algorithm 2 Continuation (S, y)
Input: Partial sequence Szi, models {0,, 0r}
Input: Prefix sequence S, initial token y
Output: Selected model my
Output: Completed sequence S
I: Ys + arg maxy Po. (y Sei)
1: x S +
2: ør < arg max, y Po. (y Ses)
2: while é SEP do
3: if y/ = Y1 then
3:
y - arg maxy Po, (o. |S)
4: m; t 8,
b identical
4: S S+y
5: else
5: end while
6:
S. +- CONTINUATION Us)
return S
7:
S₁ + CONTINUATION(S<41
8:
if Je(S,,, Si) = 0 then + 0,
9:
neutral
Algorithm 3 LLM Verifier V(S,, St)
10: else
11:
m, < O₁
> divergent
Input: Sequences S., S) Output: (0: neutral, I: divergent)
12: end
13:
end if
1: <- L.M verifies if S, and S diverges 2: return o
14:
return 171,
D Additional Related Work
D.1 Model-level Compression
Extensive studies have been proposed to accelerate the costly decoding processes of LLM by compressing the models themselves 47). Prominent techniques include sparse attention mecha- nisms 48 52 and model quantization 53 In contrast, our R2R method focuses on optimizing inference above the model level, complementing these model compression techniques. Therefore, it can be effectively combined with them further enhance inference efficiency.
D.2 Concurrent Mix Inference Methods
Given recent rapid advancements in reasoning L.Mss, several concurrent studies also explore mix inference strategies that integrate small and large language models. These methods differ primarily in their routing granularity, objectives, and specific routing schemes.
Step-level Methods:
Speculative Thinking [59], SplitReason [35], and SpecReason 660 operate at the reasoning step granularity. Speculative Thinking observes that LLMs excel at affirmation and reflection compared to SLMs. Thus, it employs heuristic triggers such as affirmations ("yeah"), "), reflections ("wait"), ) or verification signals (" ("check") to invoke the LLM selectively after detecting delimiter tokens (e.g.. "In'n"), enhancing subsequent SLM --gnerrred segments. SplitReason aims to offload difficult reusoning steps to the LLM. It first uses strong LLM to identify challenging reasoning steps, then trains the SLM to generate a special token (i.e., '<bigmodel>") signaling the LLM to take over these difficult steps. SpecReason adapts speculative decoding to reasoning
33

---

step-level. It utilizes the LLM to evaluate steps generated by the SLM, reverting to the LLM only when the score of SLM-generated steps falls below certain threshold.
Token-level Methods: Unlike step-level methods, CITER [6]] and RSD 62 adopt finer-grained token-level routing strategies. CTTER formulates the routing problem long-horizon reinforcement learning task, optimizing for final answer quality and inference efficiency. Because CITER targets general decoding tasks (e.g., short-form QA), repeatedly generating the complete response deter- mine token-level preferences remains computationally manageable. In contrast, RSD leverages an existing reward model dynamically select tokens for LLM generation whenever the SLM- LLM-produced tokens exhibit low reward scores. This approach performs well on tasks with clear and definable reward signals.
Distinctiveness of R2R: R2R distinguishes itself from concurrent works by specifically targeting immediate divergence correction token granularity. Unlike methods focused offloading complex reasoning steps, R2R addresses the subtle scenario where the SLM and LLM may agree on challenging steps, yet diverge unexpectedly on seemingly straightforward tokens (under human or LLM judgment). Such divergences can significantly alter the subsequent reasoning path, thus requiring immediate correction. Moreover, R2R differs from the speculative decoding scheme, it does not rely on periodic LLM verification steps to inform routing decisions. Instead, R2R immediately routes divergent tokens the LLM, effectively preventing divergence without incurring rollback overhead.
Given these distinct objectives and design choices, integrating R2R with these concurrent methods represents promising direction for future research, enabling even effective mix inference frameworks.
E
Proof of Quality Guarantee for Full Path-Following Routing
E.1 Notations
We summarize the notations used throughout this proof:
0r 02: the large and small language models (LL.M, SLM),, respectively.
S Sei= ( M/...... Vi-1 - the prefix sequence up to, but not including, position
i, where Seo contains only the input tokens.
ys (m) the token generated at position by model m E (0,, Ot) given prefix Sei.
V(--) E {0, 1): the quality verifier function, returning 1 iff the first sequence achieves the
same quality as the second.
s(h) sequence up to generated by the L.LM only.
s(M), <i
sequence up to generated by the mixed (routed) strategy.
S,, Sj: continuation sequences as defined in Equations X and
Tzx: the sequence formed by ((M <i concatenated with the LLM's continuation tokens:
T<: = ((A)) + (S Ui+ (0,15 ((((s)( .),S)
(6)
The routing decision at each step given by:
O2: y: (0. = (011Se))
or V(S.. St) S)) =1
identical
newtral
m:
Oi,
V(S,, St)
divergent
The continuation sequences after the i-th token are:
S, S24 è (0.1 @ (m .FOS)
(8)
S, = Sei e (0)))))) @ (0. e
(9)
where @ denotes sequence concatenation.
34

---

E.2
Theorem
The sequence S(M), generated by the path-following routing strategy, is guaranteed achieve the same quality as its LLM-only counterpart ((L) under V.
E.3
Proof
Base Case. At = 0, MMM) 5(L) Thus, V(T<o, S(L)) = 1.
<0 <0 = [2o. .nn11,, so Tea is simply the LLM's full sequence.
Inductive Hypothesis. Suppose that for some k with 0 I k n, have V(T<A, 1; i.e., continuing from s(M) with the LLM produces a sequence equal quality to S(D).
<k
Inductive Step. Consider the (k + 1)-th token y/ (M) determined by Eq. There are three cases:
(a) Identical: yx.(0, <k = yn (0 Then U/. (M4) matches the LLM's output, so the sequence remains identical and , =1
(b) Neutral: (0 * yn
<k (0115 but S(),,S) 1. The router selects the SLM token, so T<k+1 Sg. By definition, ((T<++I, S(D)
(c) Divergent: V(S.,,)) O. The router selects the LLM token, the mixed sequence again matches the LLM's output, and thus V(T<k+1,
Conclusion. By mathematical induction, for all E [0, nl), the continuation T<i maintains the same quality 5(L) under V. At generation completion n)), S(M)
s snn+1 so V(S(M),
F Prompts and Examples
F.1 Prompt for Verifier Model
design structured prompt for the verifier model (Qwen2.5-72B. Instruct) to assess whether divergences between two sentences affect their meaning, reasoning, logic, conclusions. Please refer to Text 2. for the exact prompt. The prompt highlights the divergence point and provides explicit criteria for labeling. It instructs the model to justify its judgment with brief explanation and includes illustrative examples to guide the model's understanding of both scenarios.
As discussed in Section 4.1
F22 Response Example
We use an example question from AIME and responses from R1-1.5B, R1-32B and R2R provide an intuitive understanding of our method.
Text 1. Question
Find the largest possible real part of
96 + 144
(75 + 1171) +
2
where is complex number with |z| = 4.
Text 3-5 shows the example responses. The R1-1.5B and R1-32B models produce distinct final answers for the maximum real part, reflecting divergence in their reasoning paths. By contrast, R2R identifies and corrects this divergence, navigating the correct reasoning path to get the final answer matches that of the stronger 32B model. At the same time, R2R tolerates neutral differences- such
35

---

as minor phrasing or presentation between models when these do not affect the core reasoning or conclusions. This selective routing mechanism enables R2R to deliver both high efficiency and accuracy by only invoking the large model for tokens that would otherwise lead to substantive differences in meaning or logic.
36

---

Text 2. Prompt For Verifier Model
Task:
Determine if the divergence between Sentence and Sentence affects the meaning, reason- ing, logic, conclusions derived from them.
Instructions:
The marker > indicates where the sentences diverge. It is not part of the original text.
Assess whether this divergence changes the meaning, reasoning, logic, or conclu- sions, or if introduces new information or contradictions. Output if::
The divergence causes change in meaning, reasoning. logic, conclusions.
It introduces new information, shifts focus, contradicts prior facts.
The sentences follow different reasoning paths or focus on different aspects.
Output 0 if:
The divergence is superficial and does not affect meaning, reasoning, logic, or conclusions.
Both sentences follow the same reasoning path or lead to the same conclusion.
Reasoning: Provide brief explanation of how the divergence impacts (or does not impact) meaning, reasoning, logic, or conclusions.
Example (Same- 0):
Sentence 1:
"The ratio adults to total people <is> now 11/25." Sentence 2:
"The ratio adults to total people <changsed from 5/12 11/25 after adding 50 people."
Output:
Reasoning: The change from "is" to "changed" does not affect the overall meaning, reasoning, logic, or conclusions.
Example 2 (Different- 1): Sentence 1:
"Let's solve this using <integrations by parts." Sentence 2:
"Let's solve this using <u->substitution.
Output:
Reasoning: The change in method (from integration by parts to substitution) alters the reasoning and approach to solving the problem.
Now complete the task:
Common Context:
[CONTEXT)
Sentence
(SENTENCE1}
Sentence 2:
(SENTENCE2)
Answer (Output: <0 1>)
Reasoning:
37

---

Text 3. R1-1.5B Re- sponse
Text 4. R1-32B Re- sponse
Text 5. R2R Response
Okay, I have this problem: need to find the largest possible real part of the expression
Okay, so I have this problem: need to find the largest possible real part of the expression
Okay, so I have this problem here: need to find the largest possible real part of the expres- sion
96 144i
(75+1171)
96 + 144i
(75+1171)z+
(75+1171)2+ 96 144i
2
where complex number with |ж| = 4. Hmm, that sounds bit tricky, but let me try to break down step by step.
where is com- plex number with |z| 4. Hmm, complex num- bers can sometimes be tricky, but let me try to break this down step by step.
where = a complex number with |=| 4. Hmm, that sounds a bit tricky, but let me try to break it down step by step.
First, know that = complex number with magnitude 4. So, can represent in polar form as 2 = 4et0, where 0 is the argument of z. That might make things easier because working with exponentials can sometimes simplify mul- tiplication and division. So, let rewrite the ex- pression using this polar form. The expression
First, know that any complex number be written polar form as 4e since its magnitude is 4. That might make things eas- ier because then can ex- press in terms of its an- gle 0. So, let me write that down:
First, know that 2 is complex number with magnitude 4. So, I can represent z in po- lar form. That might make things easier be- cause working with po- lar forms can simplify multiplication and divi- sion. So, let me write 4c where 0 is the argument 2. That way, [2| 4 is automatically satisfied.
40
Then, 1 would be te- because the reciprocal of e00 is ii0
96 + 144i
So, substituting back
Now, let's substitute
(75+1176)2+
into the expression, we have:
= 4e into the expres-
2
sion. So, the expression
Substituting
4e
(75 + 1171) 96+114
40
becomes:
we get:
(75 1171) 4e
(75 1171)(4e +
Let me compute each term separately. First,
96-144:
96+144;
Simplify each term sep- arately. Let's start with the first term:
Simplify each term:
let's compute (75 +
First term:
1171) "
Multiplying 75 + 117i by 4 gives:
(75 + 1171) 40
(75 + 1171) " =4(75 11
x 75 300 and 4 X
+4
117i 468i, so the first term 300 + 468í.
Which is:
The final answer is:
4x 75 cos 300 cos f
300 cos - 468 sin + i(300 sin 0 468 cos 0)
4 (-117 sin 0))
300 12V13
- 468 sin
Thus, the largest possi- ble real part is:
Thus, the largest possi- ble real part is:
540
540
38


Reviews:
[{"review_id": "ZqqkjgF5vW", "reviewer": "Reviewer_ZxyV", "summary": "This paper proposes Roads to Rome (R2R), a token-level routing method that selectively utilizes LLMs only for tokens that cause reasoning path divergence when generated by small language models (SLMs). The core insight is that only ~11% of tokens genuinely diverge reasoning paths between SLMs and LLMs, with most differences being either identical or neutral variations (e.g., abbreviations, alternative expressions). The authors develop an automatic data generation pipeline to identify divergent tokens and train a lightweight router for real-time routing decisions. Experiments on DeepSeek R1 models demonstrate 2.8X speedup over R1-32B while maintaining comparable performance, and superior efficiency compared to query-level routing and distilled models.", "questions": "1. **Sampling method extension**: How fundamental is the greedy decoding limitation? Could the neutral/divergent distinction be meaningfully applied to temperature-based sampling? This seems critical for practical deployment - can you provide any analysis or preliminary experiments?  \n2. **Cross-model generalization**: Would a router trained on DeepSeek R1-1.5B/32B work with other model pairs? How much retraining would be required?   \n3. **Verifier robustness analysis**: How sensitive is performance to verifier quality? What happens with weaker verifier models? Could you provide human evaluation of neutral/divergent distinctions to validate the verifier's reliability?  \n\n4. **Systematic failure modes**: Can you identify task types or reasoning patterns where the approach consistently fails? Does performance degrade on highly creative or non-linear reasoning tasks?  \n\n5. **Real-world deployment considerations**: How would this integrate with existing serving infrastructure? What are the latency implications of token-level routing decisions in production systems?", "limitations": "The authors adequately acknowledge the greedy sampling restriction and need for system optimizations. However, they significantly understate the generalizability concerns and don't sufficiently address the verifier dependency issues. The authors should also discuss the computational overhead more transparently in the context of alternative approaches.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. **Novel and well-motivated insight**: The distinction between neutral and divergent token differences is conceptually sound and empirically well-supported. The observation that most SLM-LLM differences don't affect reasoning paths is valuable.  \n2. **Comprehensive experimental evaluation**: The paper evaluates across several benchmarks (AIME, GPQA, LiveCodeBench) with appropriate baselines including query-level routing, distilled models, and speculative decoding. They also measured real hardware speedup. \n 3. **Strong empirical results**: Achieving 92% of R1-32B accuracy with only 17% average parameters is impressive. The 4.6× accuracy improvement over R1-1.5B with minimal LLM usage demonstrates practical value.  \n\n**Weaknesses:** \n\n1. **Critical limitation to greedy decoding**: Most production systems use temperature-based sampling or other stochastic methods. The authors acknowledge this but underestimate its significance for deployment. \n2. **Limited generalizability**: All experiments use only DeepSeek R1 models. No evidence that the approach generalizes across model families, architectures, or even different size pairs within the same family. \n\n3. **Shallow failure analysis**: Limited discussion of systematic failure modes or scenarios where neutral/divergent distinction breaks down. When does the approach perform poorly, and why?", "was_revised": true, "final_justification": "The authors did many new experiments and analyses, and addressed all my concerns (and also addressed most of the concerns of the other reviewers)", "rebuttal": "We sincerely thank Reviewer ZxyV for the extensive feedback. We appreciate your valuable recognition of the **novelty, value and strong experimental results** of our approach.\n\n### [W1, Q1]: Assumes greedy decoding, should extend to other sampling methods\n\n> W1: Critical limitation to greedy decoding: Most production systems use temperature-based sampling or other stochastic methods. \n\n> W2: Sampling method extension: How fundamental is the greedy decoding limitation? Could the neutral/divergent distinction be meaningfully applied to temperature-based sampling? \n\nWe summarize the formulation, data generation pipeline, and experimental results of R2R under sampling-based decoding. \nDue to word limit, please refer to our response to reviewer gjwQ, [W2.1,L1] for additional details.\n\n**Divergent Label and Data Generation Under Sampling**\n\nR2R naturally extends to sampling-based divergence by extending the verifier function (Equation 3) under probabilistic setup. The current token is *divergent* if the divergence probability (under deterministic verification) for SLM and LLM continuation samples exceeds the LLM’s intrinsic self-divergence.\n\nObserving that divergence primarily depends on the first differing token between SLM and LLM samples (rather than subsequent continuations), our data-generation pipeline easily adapts to sampling. Specifically, it uses sampling-based generation for LLM generation (step 0) and SLM prefill (step 1), while keeping other procedures unchanged.\n\n**Experiment Results**\n\nWe adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for experiments.\n\n|Method|Acc. (pass@1)|Param. (B)|Cost (KB)|\n|-|-|-|-|\n|R1-1.5B|27\\%|1.5|24.3|\n|R1-32B|61\\%|32.0|384.8|\n|-|-|-|-|\n|R1-14B|**58\\%**|14.0|170.5|\n|QR-SW|39\\%|9.2|135.5|\n|QR-LLM|38\\%|9.2|138.6|\n|QR-BERT|40\\%|9.4|136.8|\n|QR-MF|41\\%|9.1|130.5|\n|R2R|**58\\%**|**8.6**|**115.3**|\n\nUnder sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.\n\n### [W2]: Generalizability across model families, architectures, and size pairs\n\n> All experiments use only DeepSeek R1 models. No evidence that the approach generalizes across model families, architectures.\n\nWe extend our experiments to evaluate generalizability across:\n\n1. **Model families** (Qwen3 in addition to DeepSeek-R1)\n\n2. **Architectures** (dense vs. MoE)\n\n3. **Size pairs** (including 0.6B, 1.7B, 8B, 30B-A-3B, and 32B)\n\nWe report accuracy on the AIME benchmark using the same setup as Table 2.\n\n|Type|Model(s)|Acc. |Param. (B)|Cost (KB)|\n|-|-|-|-|-|\n|SLM|Qwen3-0.6B|12\\%|0.6|15|\n|SLM|Qwen3-1.7B|37\\%|1.7|33|\n|LLM|Qwen3-8B|67\\%|8.0|130|\n|LLM|Qwen3-30BA3B (MoE)|75\\%|3.0|51|\n|-|-|-|-|-|\n|Distill|Qwen3-4B|62\\%|4.0|71|\n|QR-SW|0.6B+8B|47\\%|4.1|85|\n|QR-LLM|0.6B+8B|52\\%|4.0|81|\n|QR-BERT|0.6B+8B|48\\%|4.0|77|\n|QR-MF|0.6B+8B|45\\%|4.1|79|\n|QR-SW|1.7B+8B|49\\%|4.0|75|\n|QR-LLM|1.7B+8B|50\\%|4.0|72|\n|QR-BERT|1.7B+8B|50\\%|4.1|73|\n|QR-MF|1.7B+8B|55\\%|4.1|73|\n|-|-|-|-|-|\n|R2R|0.6B+8B|65\\%|2.7|49|\n|R2R|1.7B+8B|68\\%|3.7|62|\n|R2R|0.6B+30BA3B (MoE)|70\\%|1.5|25|\n\nThese results demonstrate that R2R generalizes well across model families, architectures, and various size pairs. \nNotably, R2R constantly outperforms distilled models and query-level routing baselines, while activating fewer parameters ($\\leq$ 3.7B on average).\n\nWe will extend Figure 5 in the revised manuscript to include the full cost–accuracy trade-offs across these settings, allowing readers to better visualize how R2R's Pareto front shifts under different model combinations and routing thresholds.\n\n### [W3, Q4]: Systematic failure analysis\n\n> W3: Limited discussion of systematic failure modes or scenarios where neutral/divergent distinction breaks down.\n\n> Q4: Can you identify task types or reasoning patterns where the approach consistently fails? Does performance degrade on highly creative or non-linear reasoning tasks?\n\nTo identify systematic failure modes, we analyzed the response status of R2R on AIME and GPQA benchmarks. As shown below, the primary failure mode occurs when R2R cannot complete reasoning within the 32K token limit. This typically happens due to repetitive reasoning patterns not encountered during router training. \nAdditionally, R2R tends to invoke the LLM slightly more frequently on the GPQA benchmark, particularly for queries involving rare tokens such as complex protein or chemical compound names.\n\n|BenchMark|Correct|Unfinished|Wrong-same answer with LLM/SLM|Wrong-different answer with LLM/SLM|\n|-|-|-|-|-|\n|AIME (60)|33|25|0/0|2/2|\n|GPQA (198)|87|85|6/3|20/23|\n|Total (258)|120|110|6/3|22/25|\n\nWe will include a more detailed analysis of these failure cases in the final manuscript.\n\n### [Q2] Cross-model generalization of the router\n\n> Would a router trained on DeepSeek R1-1.5B/32B work with other model pairs? How much retraining would be required?\n\nWe wish to clarify that the R2R router is designed to be trained per SLM-LLM pair, requiring about 448 GPU hours for initial data generation and 2 GPU hours for router training. \nThis overhead remains significantly lower than the distillation processes commonly employed by SOTA models. The cross-model generalizability under this setup is evaluated in [W2].\n\n| |Models|Router Training Data|AIME Accuracy|Param. (B)|Cost (KB)|\n|-|-|-|-|-|-|\n|SLM|0.6B|-|11.7\\%|0.6|15|\n|LLM1|8B|-|66.7\\%|8|130|\n|LLM2|32B|-|75.0\\%|32|469|\n|R2R|0.6B+8B|0.6B+8B|65.0\\%|2.69|49|\n|R2R|0.6B+8B|0.6B+32B|53.3\\%|2.61|50|\n\nNevertheless, our empirical results indicate acceptable performance when directly substituting the LLM of a trained router without additional retraining. This can be attributed to the common *divergence* patterns shared by strong LLMs paired with the same weaker SLM.\n\n### [Q3] Verifier robustness analysis\n\n> How sensitive is performance to verifier quality? What happens with weaker verifier models? Could you provide human evaluation of neutral/divergent distinctions to validate the verifier's reliability?\n\n**Human Evaluation**\n\nWe conducted a human evaluation to validate the verifier's reliability. Specifically, we recruited four undergraduate students to independently label 1,357 *differences* as either *neutral* or *divergent*. The *differences* are between R1-1.5B and 32B, on the first six AIME-2024 questions. Three annotators' labels determined ground-truth *divergence* by unanimous consent (core *divergence*) and majority voting (general *divergence*). The fourth annotator's labels were compared with ground truth. Annotators and LLM verifiers received identical contexts and instructions. Our selected verifier (Qwen2.5-72B) closely matches human expert performance:\n\n|Verifier|*Divergent* Rate (%)|Core-Recall (%)|Core-Accuracy (%)|General-Recall (%)|General-Accuracy (%)|\n|-|-|-|-|-|-|\n|Human|35|96|76|84|82|\n|Qwen72B|35|97|76|88|84|\n|Qwen7B|39|94|71|85|78|\n|Qwen3B|34|75|72|69|75|\n\n**Verification Quality**\n\nTo examine sensitivity to verifier accuracy, we applied our sentence-level path-following routing method (Equations 3–5) across verifiers. Our analysis reveals that even moderate-recall verifiers (e.g., Qwen2.5-3B with 66% recall) still significantly improve reasoning path guidance compared to the baseline (R1-7B).\n\n|Type|Model Param. (B)|Labeling Verifier|AIME'24 #Acc.@ $\\leq$8K|\n|-|-|-|-|\n|LLM|32|-|17|\n|SLM|1.5|-|2|\n|Distill|7|-|8|\n|Path-following routing|2.3|Qwen2.5-72B|16|\n|Path-following routing|2.5|Qwen2.5-7B|12|\n|Path-following routing|2.2|Qwen2.5-3B|11|\n\nNote that the current verifier prompt is tuned for Qwen2.5-72B verifier. Empirically, smaller verifiers should benefit from prompts biased towards classifying ambiguous cases as *divergent*, enhancing recall. We will include detailed prompt-tuning procedures and extended results in the final manuscript.\n\n### [Q5] Real-world deployment considerations \n\n> How would this integrate with existing serving infrastructure? What are the latency implications of token-level routing decisions?\n\nWe integrated R2R into the production-level SGLang serving infrastructure, as evidenced by the high absolute throughput results of our method and baselines.\n\nTo assess latency implications of R2R, we performed a detailed runtime breakdown of R2R using two A800 GPUs on the AIME benchmark, averaged over 60 questions:\n\n|Component|Percentage of Total Time|\n|-|-|\n|Router|5.96%|\n|LLM(R1-32B)|64.97%|\n|SLM(R1-1.5B)|26.77%|\n|Others|2.30%|\n\nThis indicates that token-level routing decisions introduce minimal latency overhead. Additionally, given the small size of the neural router (56MB), we anticipate further runtime improvements through additional system-level optimizations.\n\n### [L1] Discuss computational overhead\n\n> The authors should also discuss the computational overhead more transparently in the context of alternative approaches.\n\nFollowing prior work [1, 2], we analyze both FLOPs and memory access overhead on the AIME benchmark across different approaches.\n\n|Model|Total Computation (TFLOPs)|Total Memory Access (TB)|Avg. Memory Access Per Token (GB)|\n|-|-|-|-|\n|R1-1.5B|157|104|3.7|\n|R1-14B|630|503|29.4|\n|R1-32B|1136|966|63.6|\n|Eagle2|25490|515|29.6|\n|HASS|26809|544|28.9|\n|R2R|1502|216|11.7|\n\nGiven the memory-bound nature of decoding, memory access significantly impacts throughput. R2R notably reduces memory access compared to R1-32B with only a modest increase in computation, primarily due to additional KV-cache updates from switching between the LLM and SLM.\nIn contrast, speculative decoding approaches (e.g., Eagle2, HASS) incur substantially higher computational costs from their tree-structured draft and verification processes (60 tokens per cycle). Despite this overhead, they achieve significant throughput improvements through reduced memory access during decoding.\n\n[1] Hoffmann, Jordan, et al. \"Training compute-optimal large language models.\" NeurIPS. 2022.\n\n[2] Yang, Fan, et al. \"GLITCHES: GPU-FPGA LLM Inference Through a Collaborative Heterogeneous System.\" HPEC. 2024."}, {"review_id": "BbuUhAODLE", "reviewer": "Reviewer_PS9S", "summary": "This work proposes a routing scheme that delegates the decoding flow from a small LLM (SLM) to a more powerful LLM (LLM) when there is divergence between the reasoning path. A divergence is defined to emerge when there is a disagreement between the predictions of the SLM and LLM and that disagreement will result in a different reasoning path till the end of the sentence. Here the difference between two reasoning paths is evaluated by an external verifier model (Qwen2.5-72B used in this work).\n\nThe authors further propose a data generation pipeline to collect labels for such divergent tokens, and then train an simple MLP-based token-level router to dynamically decide when to delegate the reasoning to the LLM. The authors apply the proposed method (R2R) to R1-1.5B and R1-32B models from the DeepSeek family and evaluate on a variety of tasks. The results show impressive performance (comparable to R1-32B and better than R1-14B) with significantly reduced inference costs, which are measured by the number of activated parameters and the wall-clock time (latency and token generation speed).", "questions": "- I wonder if MoE can be included in the context here. In my opinion, MoE is another way of utilizing a model with larger capacity. But the difference is that MoE implicitly uses the whole capacity with fixed cost at each token. In contrast, R2R uses small capacity model with low costs most of time but escalate to larger model when needed. It would be great if we can find a way to properly compare the two. I understand that this can be too demanding for the review period. So this is not a request but an open question out of curiosity.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The intuition of the dynamic reasoning makes sense and the definition of divergent tokens is reasonable.\n+ The empirical results are impressive and the decoding acceleration is practically promising. \n+ The authors provide ablation study on the design choice of the MLP router in Section 5.4. While the results are insufficiently discussed (see comments below), the study does support the choice of the input fields to the router and the training objective.\n+ The paper is written clearly with plenty of details in support.\n\nWeaknesses:\nJust to clarify that I think this work is of good quality overall. So many of the weaknesses are not hard criticisms but constructive feedback on which I would invite more discussions with the authors.\n- (Discussion against [3]) I understand that [3] is concurrent. But now that the authors mention [3] in Line 227, I think it would be better to discuss properly in the section of related work.\n- (Insufficient discussion of ablation study) I think the ablation study on the routing objective is not sufficiently discussed. I suppose delegate \"more\" tokens to LLM will not hurt the performance this much. Is it because the objective change causes the router misses more divergent tokens? Moreover, it is not clear what the authors mean by \"it fails to reach the original accuracy within the same amount of LLM usage, ...\" Does it imply that the authors manually control the activation of the LLM in this experiment?", "was_revised": true, "final_justification": "The authors responses addressed my questions and concerns. From my point of view, the authors have evaluated the proposed method with latency and token throughput (# token / sec), where the results are in favor. I look forward to more discussion and hopefully numerical comparison with more related works in the final version.", "rebuttal": "We sincerely thank Reviewer PS9S for the extensive feedback. We appreciate your valuable recognition of the **intuition, quality and impressive empirical results** of our approach. \n\n### [W1] Discussion on concurrent work (SplitReason)\n\n> I understand that [3] is concurrent. But now that the authors mention [3] in Line 227, I think it would be better to discuss properly in the section on related work.\n\nThank you for the great suggestion. We will include a proper discussion of the concurrent work SplitReason in our revised manuscript. As the SplitReason repository is under maintenance during the rebuttal period, we plan to include the experimental comparison in the final version.\n\nBoth R2R and SplitReason explore mixed inference strategies combining small and large language models, but differ in routing granularity and objectives.\n\nSplitReason aims to offload **difficult reasoning steps** to the LLM. Specifically, it first uses a strong LLM to identify challenging reasoning steps, then trains the SLM to generate a special token (i.e., `<bigmodel>`) signaling the LLM to take over these difficult steps.\n\nIn contrast, R2R addresses **token-level *divergences*** rather than step-level difficulty. Different from SplitReason, R2R also targets the subtle scenario where the SLM and LLM may agree on challenging steps, yet diverge unexpectedly on seemingly straightforward tokens. Such *divergences* can significantly alter the subsequent reasoning path, thus requiring immediate correction.\n\nWe believe the two complementary perspectives would offer comprehensive contexts for efficient reasoning, inspiring even more effective mixed inference future works.\n\n\n\n### [W2] Insufficient discussion of ablation study\n\n> I think the ablation study on the routing objective is not sufficiently discussed. \n\nThank you for the thoughtful questions. We clarify as follows:\n\n> I suppose delegate \"more\" tokens to LLM will not hurt the performance this much.\n\n1. **More LLM tokens do not commonly hurt AIME performance.**\n\n    Indeed, all 7 AIME questions correctly answered by the SLM are also correctly answered by the LLM, confirming that the LLM consistently outperforms the SLM rather than specializing in different types of math questions.\n\n> Is it because the objective change causes the router misses more divergent tokens?\n\n2. **Yes, the *different* objective leads to more missed *divergent* tokens.**\n\n    To investigate this, we evaluate the *divergent* token recall of different routers. We use ground-truth responses and *divergence* labels generated with our data pipeline on AIME. Routers trained with *different* and *divergent* objectives are marked at Router1 and Router2, respectively.\n\n|LLM Rate|Router1-Recall|Router2-Recall|\n|-|-|-|\n|5%|0.50|0.28|\n|10%|0.75|0.46|\n|11%|0.77|0.51|\n|12%|0.80|0.55|\n|13%|0.83|0.59|\n|14%|0.85|0.63|\n|15%|0.87|0.66|\n|16%|0.90|0.69|\n|20%|0.94|0.79|\n\nAs demonstrated in the table, the router trained with the *different* objective shows lower recall, even at higher LLM usage. This is because it wastes LLM calls on *neutral* differences rather than targeting truly *divergent* tokens.\n\n> Moreover, it is not clear what the authors mean by \"it fails to reach the original accuracy within the same amount of LLM usage, ...\" Does it imply that the authors manually control the activation of the LLM in this experiment?\n\n3. **Yes, we manually adjust the routing threshold to control LLM activation.**\n\n    Since R2R allows for flexible trade-offs between LLM usage and performance, each router defines a trade-off curve. For ablation comparison, we adjust thresholds to match or slightly exceed the LLM rate of our main router, and then compare their accuracies. Despite higher LLM usage, the routers with  objective or reduced inputs fail to reach comparable accuracy.\n\nWe will revise the manuscript to improve the clarity of this ablation discussion.\n\n\n\n### [Q1] Including MoE in the context\n\n> I wonder if MoE can be included in the context here. In my opinion, MoE is another way of utilizing a model with larger capacity. But the difference is that MoE implicitly uses the whole capacity with fixed cost at each token. In contrast, R2R uses small capacity model with low costs most of time but escalate to larger model when needed. It would be great if we can find a way to properly compare the two. I understand that this can be too demanding for the review period. So this is not a request but an open question out of curiosity.\n\nThank you for raising this interesting perspective. Below, we first present a conceptual comparison between MoE and R2R, followed by empirical comparisons, and finally discuss how the two approaches can complement each other to further improve the efficiency-accuracy trade-off.\n\n**Conceptual Comparison**\n\n|Design Aspect|MoE|R2R|\n|-|-|-|\n|Partial activation|Yes|Yes|\n|Routing granularity|Fine-grained (Experts)|Coarse-grained (Models)|\n|Subject model sizes|Equal parameters per expert|Different parameters per model|\n|Training overhead|Full training from scratch|Router training only (SLM+LLM pre-existing)|\n|Supervision|Next-token predictions|Explicit routing decisions|\n\nAs noted by the reviewer, both MoE and R2R leverage partial activation of larger capacity models, but with distinct granularities, subject sizes, training overhead, supervision and complementary insights. This conceptual distinction motivates their integration, which we explore later.\n\n**Empirical Comparison**\n\nWe evaluate the MoE model Qwen3-30BA3B and the R2R model (combining Qwen3-0.6B and Qwen3-8B) on the AIME benchmark. The 60 questions are split into Easy (28) and Hard (32) subsets based on whether the question ID $\\leq$ 7.\n\n|Type|Models|Easy-Accuracy|Easy-Parameters (B)|Hard-Accuracy|Hard-Parameters (B)|\n|-|-|-|-|-|-|\n|R2R|Qwen3-0.6B + Qwen3-8B|93\\%|2.9|41\\%|2.6|\n|MoE|Qwen3-30BA3B|93\\%|3.3|59\\%|3.3|\n\nThese results highlight the strengths of each approach. Thanks to extensive and sophisticated pretraining, state-of-the-art MoE models can match dense model performance (e.g., both Qwen3-30BA3B and Qwen3-32B achieve 75% accuracy on AIME in our evaluation). However, their fixed per-token overhead limits efficiency for simpler tasks.\n\nIn contrast, R2R dynamically adjusts computation based on token-level *divergence*. It achieves lower cost on easier examples by relying more on the SLM, but its small router and much lighter training cannot match MoE performance on harder questions with the same activated parameters. \n\nGiven these complementary advantages, we explore the combination of both methods.\n\n**Combining MoE and R2R**\n\n*MoE for R2R*\n\nR2R is directly compatible with efficiency-optimized models such as MoE models. We validate R2R performance using the Qwen3-30BA3B MoE as the LLM:\n\n| |Acc.|Param. (B)|Cost (KB)|\n|-|-|-|-|\n|Qwen3-0.6B|12\\%|0.6|14.7|\n|Qwen3-1.7B|37\\%|1.7|32.8|\n|-|-|-|-|\n|Qwen3-32B|75\\%|32.0|469.1|\n|R2R (0.6B+32B)|67\\%|10.2|154.0|\n|-|-|-|-|\n|Qwen3-30BA3B (MoE)|75\\%|3.0|50.5|\n|R2R (0.6B+30BA3B MoE)|68\\%|1.3|21.7|\n\nCombining R2R with MoE models significantly improves the Pareto frontier, achieving robust accuracy even at extremely low activated parameters per token.\n\n*R2R for MoE*\n\nR2R insights can also enhance MoE model designs.\nCurrent MoE methods generally use uniform-sized experts. Introducing mixed-sized experts like R2R could enable query-adaptive expert activation. \nMoreover, supplementing MoE's token prediction objective with explicit *divergence* or difficulty-based routing supervision from R2R may encourage adaptive usage of larger experts only for critical decoding steps.\nAlthough this approach would require retraining MoE models and is beyond the scope of this paper, it opens exciting future research directions.\n\nWe will incorporate these discussions into the revised manuscript, providing insights for further exploration."}, {"review_id": "mXURX3jd5N", "reviewer": "Reviewer_uytF", "summary": "The paper proposes R2R (Roads to Rome), a token-level routing method that combines the efficiency of Small Language Models (SLMs) with the reasoning quality of Large Language Models (LLMs). Rather than relying solely on either model, R2R selectively routes divergent tokens to an LLM, while leaving the rest to the SLM. The authors introduce: (1)A token-level data labeling pipeline to identify divergent tokens; (2)A lightweight neural router trained to predict token divergence; (3) A routing strategy that improves inference efficiency with minimal loss in reasoning quality. By applying R2R between DeepSeek R1-1.5B (SLM) and R1-32B (LLM), they show up to 4.6× accuracy improvement over SLM, with 2.8× speedup over the LLM, surpassing distilled and query-level routing methods in performance-cost tradeoffs.", "questions": "(1) How does R2R perform under temperature sampling or nucleus sampling? Would the divergence pattern be consistent?\n(2) What guarantees does the LLM verifier provide, and could it mislabel semantically divergent tokens as neutral?\n(3) Have you considered batch routing decisions (e.g., look-ahead or segment-level) instead of per-token routing?\n(4) Could R2R apply to generative tasks like summarization or dialogue, where path divergence may not always be objectively verifiable?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Novel Insight & Intuition: The paper identifies that most token differences between SLM and LLM are either identical or neutral, and that only a small fraction leads to performance degradation—an important insight for efficient inference.\n\n(2) Efficient Labeling Pipeline: The sentence-level LLM continuation and an LLM verifier to distinguish divergent tokens are well-designed, scalable, and avoid the need for expensive human annotations.\n\n(3) Strong Empirical Results: First, it advances the Pareto frontier on reasoning benchmarks (AIME, GPQA, LiveCodeBench). Second, it achieves impressive speed-accuracy tradeoffs (e.g., 2.8× speedup over R1-32B with comparable accuracy).\n\n(4) Robust Engineering: Implementation details are thorough, including training strategy, threshold tuning, GPU memory optimizations, etc.\n \nWeaknesses:\n\n(1) Verifier Dependence: The data labeling pipeline depends on a separate LLM-based verifier, which may introduce a bias or circular dependency (LLM verifying LLM paths).\n\n(2) Domain Bias in Training Data: The training data (math, QA, and code) may not generalize well to open-ended tasks (e.g., embodied, navigation, etc).\n\n(3) Assumes Greedy Decoding: The method is only validated under greedy decoding. It remains unclear how well R2R generalizes to sampling-based strategies used in many real-world applications (e.g., temperature > 0).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer uytF for the valuable feedback. We appreciate your great recognition of the **significance, novelty, and strong experimental results** of our approach. \n\n### [W1, Q2] Verifier depends on LLM, potentially mislabels divergent and neutral tokens.\n\n> W1: The data labeling pipeline depends on a separate LLM-based verifier, which may introduce a bias or circular dependency.\n\n> Q2: What guarantees does the LLM verifier provide, and could it mislabel semantically divergent tokens as neutral?\n\nTo evaluate reliability in classifying *divergences* and *neutral* differences, we conducted additional experiments comparing the different LLM verifiers with human experts. We also exame model robustness to mislabeling.\n\n**Verifier Capacity and Verification Quality**\n\nWe constructed ground-truth *divergent* labels with **three independent human expert** verifiers [1]. The **general** *divergent* tokens are labeled by majority voting among expert answers, taking up 24.8% of differences. The **core** *divergent* tokens, a subset of highly *divergent* tokens with more strict criteria, are labeled with unanimous consent of experts, which takes up 11.9% of *differences*.\n he table below compares the chosen verifier (Qwen2.5-72B) against a **fourth independent expert**. Our verifier achieves a comparable performance over human expert, with high recall on core *divergence*.\n\n|Verifier|*Divergent* Rate (%)|Core-Recall (%)|Core-Accuracy (%)|General-Recall (%)|General-Accuracy (%)|\n|-|-|-|-|-|-|\n|Human|35|96|76|84|82|\n|Qwen2.5-72B|35|97|76|88|84|\n|Qwen2.5-7B|39|94|71|85|78|\n|Qwen2.5-3B|34|75|72|69|75|\n\nNote that the verifier judges semantic differences between two continuations, which is notably simpler than generating correct responses or evaluating the entire reasoning path. \nConsequently, even a smaller verifier (Qwen2.5-7B) provides satisfactory performance. However, significantly smaller models (e.g., 3B) may yield less ideal results.\n\n**Verification Quality and Response Quality**\n\nWe evaluated how verifier accuracy impacts response quality using sentence-level path-following routing (Equations 3–5) on AIME questions. It evaluates the influence of verifier on final response quality, without incurring the neural router. \n\nAs shown below, even a verifier with moderate recall (e.g., 66% for Qwen2.5-3B) can guide better reasoning paths than the baseline R1-7B model. \n\n|Type|Param. (B)|Labeling Verifier|AIME'24 #Acc.@ $\\leq$8K|\n|-|-|-|-|\n|LLM|32|-|17|\n|SLM|1.5|-|2|\n|Distill|7|-|8|\n|Path-following routing|2.3|Qwen2.5-72B|16|\n|Path-following routing|2.5|Qwen2.5-7B|12|\n|Path-following routing|2.2|Qwen2.5-3B|11|\n\nSentence-level path-following routing also connects the verifier prompt to final response quality, enabling direct performance tuning via prompt engineering. \nThe current verifier prompt is optimized for Qwen2.5-72B. Empirically, smaller verifiers can benefit from prompts biased towards classifying ambiguous cases as *divergent*, enhancing recall.\nWe will include detailed prompt-tuning discussions and extended results in the final manuscript.\n\n**Quality-Guaranteed Verifier**\n\nWe also introduce the quality-guaranteed **full path-following routing** (L151-158, proof in Appendix D). For scenarios with shorter reasoning paths and sufficient computational resources, it explicitly ensures the final response quality with generated labels.\n\n[1] We recruited four undergraduate engineering students as annotators. They receive identical contexts and instructions as the verifier LLM, covering 1357 *different* tokens between R1-1.5B to R1-32B on six AIME-2024 questions.\n\n\n### [W2, Q4] Generalizability across domains and tasks \n\n> W2: The training data (math, QA, and code) may not generalize well to open-ended tasks.\n\n> Q4: Could R2R apply to generative tasks like summarization or dialogue, where path divergence may not always be objectively verifiable?\n\nWe appreciate the reviewer’s comments on R2R’s generalizability.\n\n**Generalizable Data Labeling Method**\n\nOur labeling method, relying on semantic comparison rather than formal verification, easily generalizes across tasks. Unlike methods requiring task-specific external tools (e.g., formal proofs [2], code execution [3]), R2R uses an LLM verifier to identify general semantic divergence in *meaning, reasoning, logic, or conclusions* (Appendix E.1).\n\nIn practice, we apply the identical labeling strategy and verifier prompt across closed-form math, coding tasks, and open-ended QA tasks. This consistency enables our router to naturally generalize to unseen tasks during inference, as demonstrated later.\n\nFor tasks with subjective *divergence* criteria that are challenging to identify semantically within a single sentence, the continuation length can be extended. This adjustment gradually transitions from sentence-level routing to full routing (L151-158), the latter of which only requires overall response quality evaluation—a feasible requirement commonly met for LLM tasks.\n\n**Generalizable Router**\n\nOur router leverages outputs from the SLM (e.g., logits) to predict token *divergence*. Benefiting from the inherent generalizability of SLMs, indicators such as logits entropy robustly identify *divergent* tokens across different tasks.\n\nTo validate the router’s generalizability, we directly apply our router, trained for math, QA, and code, to additional benchmarks: Arena-Hard for **Dialog** [4], and MMLU-Redux-**Philosophy**. \n\n|Type|Method|Dialog-Score|Dialog-Param.|Dialog-Cost|Philosophy-Acc.(%)|Philosophy-Param.|Philosophy-Cost|\n|-|-|-|-|-|-|-|-|\n|LLM|R1-32B|5.0|32|65.9|38|1.5|8.7|\n|SLM|R1-1.5B|0.2|1.5|14.3|79|32|32.2|\n|Distill|R1-7B|0.3|7|35.0|57|7|20.6|\n|Distill|R1-14B|2.2|14|56.5|77|14|18.5|\n|R2R|Ours|2.8|6.1|40.9|81|6.7|8.3|\n\nR2R continues to outperform the 14B model with less than 7B parameter size.\n\n[2] X, Huajian, et al. \"Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search.\" arXiv 2024.\n\n[3] D. Jung, et al. “Code execution as grounded supervision for LLM reasoning.” arXiv 2025\n\n[4] We evaluate the first 100 questions in the benchmark to expedite testing.\n\n### [W3, Q1] Sampling method extension\n\n> W3: The method is only validated under greedy decoding. It remains unclear how well R2R generalizes to sampling-based strategies\n\n> Q1: How does R2R perform under temperature sampling or nucleus sampling? Would the divergence pattern be consistent?\n\nWe provide additional analysis and experiments on sampling-based decoding.\n\n**Path-Following Routing Under Sampling** \n\nR2R naturally extends to sampling-based decoding by adapting *divergence* labeling to a probabilistic setting. \nSpecifically, the deterministic continuation sequence pair $(S_s, S_l)$ from Equations 3–5 are replaced by $k$ **sampled continuation pairs** under a chosen sampling method. *Divergence* is then evaluated for each sampled pair. The routing decision for each token then depends on whether the overall **divergence probability** exceeds a predefined threshold:\n$$\nP(V(S_{s_i}, S_{l_i}) = 0|i \\in [0,k)) \\geq P_{\\text{threshold}}\n$$\n\nIf setting $P_{\\text{threshold}}$ to the LLM's self-*divergence* probability $P(V(S_l, S_l)=0)$, it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone. We will provide detailed descriptions and theoretical analyses of this method in the final manuscript.\n\n**Efficient Data Generation**\n\nSampling multiple continuations for *divergence* labeling is computationally expensive. However, since continuations are sentence-level, we empirically observe that the *divergence* decision primarily depends on the first differing token between the SLM and LLM samples, rather than the subsequent continuations. \nTo efficiently approximate probabilistic routing, we therefore only sample for next-token generation $y_i(\\theta_s|S_{<i})$ and $y_i(\\theta_l|S_{<i})$, while leaving continuations deterministic.\n\nGiven limited time for rebuttal and the extensive volume of tokens, we set $k=1$ and $P_{\\text{threshold}}=0.5$, simplifying the setup. It incurs overhead comparable to our greedy data generation pipeline. \n\nIn implementation, our data-generation pipeline adapts to the new setup by using sampling-based generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged. \n\n**Experiment Results**\n\nWe adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for preliminary experiments. We retain the neural router architecture, as the router already accepts embedding of the sampled token.\n\n|Method|Acc. (%)(pass@1)|Param. (B)|Cost (KB)|\n|-|-|-|-|\n|R1-1.5B|27|1.5|24.3|\n|R1-32B|61|32.0|384.8|\n|-|-|-|-|\n|R1-14B|**58**|14.0|170.5|\n|QR-SW|39|9.2|135.5|\n|QR-LLM|38|9.2|138.6|\n|QR-BERT|40|9.4|136.8|\n|QR-MF|41|9.1|130.5|\n|R2R|**58**|**8.6**|**115.3**|\n\nUnder sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.\n\n***Divergence* Distribution Analysis**\n\n|Type|#Query|#Token|#Different|Diff. Rate|#*Divergent*|Div. Rate|\n|-|-|-|-|-|-|-|\n|Math|862|7.3M|858K|11.8\\%|438K|6.0\\%|\n|Code|987|6.8M|1.2M|17.5\\%|627K|9.3\\%|\n|QA|805|2.1M|443K|21.3\\%|231K|11.1\\%|\n|Summary|2654|16.1M|2.5M|15.5%|1.3M|8.0\\%|\n\nThe average *divergence* rate under sampling slightly increases (+3.1%) but remains low overall, demonstrating consistent *divergence* patterns with greedy decoding.\n\n\n### [Q3]: Batch routing decisions\n\n> Have you considered batch routing decisions (e.g., look-ahead or segment-level) instead of per-token routing?\n\nAlthough R2R currently routes at the token-level, it can extend to coarser granularities such as segments: we simply aggregate token‑level signals within a segment and issue a single routing decision. This opens up new trade-offs between routing granularity, promptness, and accuracy. We consider this a promising direction for future work."}, {"review_id": "QL0zYeON6g", "reviewer": "Reviewer_gjwQ", "summary": "This paper proposes a token-level routing method to decide whether to use a small language model (SLM) or a large language model (LLM) for each token. The goal is to have the SLM follow the LLM's reasoning path and it seems that only a small fraction of tokens diverge the reasoning paths. Therefore, by identifying those divergent tokens and using the LLM for generating those while using the SLM for other tokens can lead to comparable results with the LLM while having smaller number of activated parameters and having a significant speedup compared to just using the LLM for the entire generation. Their main contributions are (1) proposing an automatic pipeline to label divergent tokens. (2) developing a token-level routing scheme using a lightweight neural router. (3) enabling a more efficient performance scaling at test time using their R2R framework.", "questions": "(1) What are your thoughts on combining R2R with some query-level routing methods so that you can choose which SLM or LLM to use based on the difficulty of the query, and then using R2R to route tokens to the SLM or LLM for the generation? \n\n(2) Can you please clarify what you mean in lines 170-171? I assumed you would rely on the smaller model for generating the majority of the tokens. Why is it said that only 3% of generated tokens are from the smaller model?", "limitations": "Their method is based on greedy decoding and sentence-level assumptions. These assumptions can be limiting. For example, the SLM might want to generate the sentences in a paragraph in a different order but still follow a valid reasoning path and achieve the same conclusion as the LLM in some cases. Also, the greedy decoding may potentially be problematic for tasks where we want to generate more diverse or creative text.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) This work introduces a token-level approach that enables small language models to follow the reasoning paths of a larger model by identifying the divergent tokens and only correcting those by using the LLM. This allows the SLM to achieve comparable performance with the LLM while using the LLM only for generating a small fraction of tokens. This is significant as this can achieve a more efficient performance scaling at test time.\n\n(2) The lightweight routing scheme identifies whether to use the LLM or SLM at each token level, and it can immediately decide whether to accept the SLM generation or not. Therefore, it eliminates the need for rollbacks, which methods like speculative decoding usually face (Speculative decoding methods only periodically verify the tokens and invalidate all tokens after a correction).\n\n(3) The authors report the results of ablation studies on using divergent vs different tokens, as well as using the hidden states, logit, and/or token embeddings as the inputs to the router to show the effectiveness of their choices in the framework. They also report the results on three different datasets and provide comparisons between R2R and the query-level routing methods. Overall, they have a sound experimental setup, in my opinion. \n\nWeaknesses:\n\n(1) I believe the writing can be improved, and it should be revised to correct some sentences. For example, in line 24, I believe \"hundreds of parameters\" is not correct, as you are saying \"LLMs with hundreds of parameters (line 124) and SLMs with \"few billions of parameters (line 26)\"\n\n(2) Some assumptions, such as \"greedy\" and \"sentence-level\" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.", "was_revised": true, "final_justification": "The authors have provided clear and thorough explanations in their rebuttal, and I will maintain my positive assessment of this paper.", "rebuttal": "We sincerely thank Reviewer gjwQ for the constructive feedback. We appreciate your valuable recognition of the **great significance and thorough experiments** of our approach.\n\n### [W1, Q2] Typo corrections\n\n> W1: In line 24, I believe \"hundreds of parameters\" is not correct, as you are saying \"LLMs with hundreds of parameters (line 124) and SLMs with \"few billions of parameters (line 26)\"\n\nWe apologize for the typos. To clarify, in line124, we wish to say \"LLMs with hundred of **billions of** parameters\". \n\n> Q2: Can you please clarify what you mean in lines 170-171? I assumed you would rely on the smaller model for generating the majority of the tokens. Why is it said that only 3% of generated tokens are from the smaller model?\n\nIn line 171, the correct statement should read: \"...while relying on the **larger Rl-32B** model for only 3% of the generated tokens.\"\n\nWe genuinely thank the reviewer for the careful review. We will carefully proofread the paper to eliminate typos in our writing.\n\n### [W2.1, L1] Assumptions on greedy decoding \n\n> W2.1: Some assumptions, such as \"greedy\" and \"sentence-level\" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.\n\n> L1: Also, the greedy decoding may potentially be problematic for tasks where we want to generate more diverse or creative text.\n\nWe thank the reviewer for this insightful question. Below, we provide additional analysis and preliminary experiments on how R2R generalizes to sampling-based decoding strategies.\n\n**Path-Following Routing Under Sampling** \n\nR2R naturally extends to sampling-based decoding by adapting *divergence* labeling to a probabilistic setting. \nSpecifically, the deterministic continuation sequence pair $(S_s, S_l)$ from Equations 3–5 are replaced by $k$ **sampled continuation pairs** under a chosen sampling method. *Divergence* is then evaluated for each sampled pair. The routing decision for each token then depends on whether the overall **divergence probability** exceeds a predefined threshold:\n$$\nP(V(S_{s_i}, S_{l_i}) = 0|i \\in [0,k)) \\geq P_{\\text{threshold}}\n$$\n\nIf setting $P_{\\text{threshold}}$ to the LLM's self-*divergence* probability $P(V(S_l, S_l)=0)$, it ensures the mixed model, under full path-following routing, maintains the same quality expectation as the LLM alone. We will provide detailed descriptions and theoretical analyses of this method in the final manuscript.\n\n**Efficient Data Generation Pipeline**\n\nSampling multiple continuations for *divergence* labeling is computationally expensive. However, since continuations are sentence-level, we empirically observe that the *divergence* decision primarily depends on the first differing token between the SLM and LLM samples, rather than the subsequent continuations. \nTo efficiently approximate probabilistic routing, we therefore only sample for next-token generation $y_i(\\theta_s|S_{<i})$ and $y_i(\\theta_l|S_{<i})$, while leaving continuations deterministic.\n\nGiven limited time for rebuttal and the extensive volume of tokens, we set $k=1$ and $P_{\\text{threshold}}=0.5$, simplifying the setup. This approximation incurs overhead comparable to our greedy data generation pipeline. \n\nIn implementation, our data-generation pipeline adapts to the new setup by using sampling-based generation for both LLM responses (step 0) and SLM prefill (step 1), while keeping other procedures unchanged. \n\n**Experiment Results**\n\nWe adopt DeepSeek-R1’s recommended sampling settings (temperature=0.6, top-p=0.95) for preliminary experiments. We retain the neural router architecture, as the router already accepts embedding of the sampled token.\n\n|Method|Acc. (pass@1)|Param. (B)|Cost (KB)|\n|-|-|-|-|\n|R1-1.5B|27\\%|1.5|24.3|\n|R1-32B|61\\%|32.0|384.8|\n|-|-|-|-|\n|R1-14B|**58\\%**|14.0|170.5|\n|QR-SW|39\\%|9.2|135.5|\n|QR-LLM|38\\%|9.2|138.6|\n|QR-BERT|40\\%|9.4|136.8|\n|QR-MF|41\\%|9.1|130.5|\n|R2R|**58\\%**|**8.6**|**115.3**|\n\nUnder sampling-based decoding, R2R remains Pareto optimal compared to distilled models and query-level routing methods.\n\n***Divergence* distribution analysis**\n\n|Type|#Query|#Token|#Different|Diff. Rate|#*Divergent*|Div. Rate|\n|-|-|-|-|-|-|-|\n|Math|862|7.3M|858K|11.8\\%|438K|6.0\\%|\n|Code|987|6.8M|1.2M|17.5\\%|627K|9.3\\%|\n|QA|805|2.1M|443K|21.3\\%|231K|11.1\\%|\n|Summary|2654|16.1M|2.5M|15.5%|1.3M|8.0\\%|\n\nThe average *divergence* rate under sampling slightly increases (+3.1%) but remains low overall, demonstrating consistent *divergence* patterns with greedy decoding.\n\n### [W2.2, L2] Assumptions on sentence-level path-following routing\n\n> W2.2: Some assumptions, such as \"greedy\" and \"sentence-level\" path-following, might be limiting in some cases and for some tasks, but I understand that it may have been necessary to have such assumptions due to the huge search space.\n\n> L2: For example, the SLM might want to generate the sentences in a paragraph in a different order but still follow a valid reasoning path and achieve the same conclusion as the LLM in some cases\n\nWe appreciate the reviewer's thoughtful comment. While mislabeling *neutral* continuations as *divergent* typically does not degrade performance, it may occasionally introduce unnecessary computational overhead due to additional LLM activations. \n\nTo evaluate mislabeling caused by insufficient continuation, we extend our routing strategy to N-sentence path-following. As N gradually increases from 1 to the total remaining sentences in the LLM-generated response, it transitions smoothly from sentence-level to fully path-following routing. \n\nIn experiments on the subset of AIME dataset, we observe that increasing N from 1 to 5 only reduces the divergence rate modestly from 2.91% to 2.66%, corresponding to an average activated parameter reduction of 0.08B. Given this limited improvement, we empirically select N=1 (sentence-level) as our default setting.\n\n### [Q1] Combining R2R and query-level routing\n> What are your thoughts on combining R2R with some query-level routing methods so that you can choose which SLM or LLM to use based on the difficulty of the query, and then using R2R to route tokens to the SLM or LLM for the generation?\n\nThank you for the insightful suggestion. We investigated combining R2R with query-level routing methods using the Qwen3 model series with varied sizes, evaluated on the AIME benchmark. Specifically, we treated R2R-S (0.6B+8B) and R2R-L (0.6B+32B) as the SLM and LLM, respectively, for query-level routing.\n\n|Type|Model|Acc.|Param. (B)|Cost (KB)|\n|-|-|-|-|-|\n|SLM|Qwen3-0.6B|12\\%|0.6|15|\n|LLM|Qwen3-8B|67\\%|8|130|\n|LLM|Qwen3-32B|75\\%|32|469|\n|-|-|-|-|-|\n|R2R-S|Qwen3-0.6B + 8B|59\\%|2.1|40|\n|R2R-L|Qwen3-0.6B + 32B|67\\%|10.2|154|\n|-|-|-|-|-|\n|R2R-M|Qwen3-4B + 8B|65\\%|5.4|**95**|\n|-|-|-|-|-|\n|QR-BERT|Qwen3-0.6B + 32B|25\\%|5.4|230|\n|QR-LLM|Qwen3-0.6B + 32B|23\\%|5.4|238|\n|QR-MF|Qwen3-0.6B + 32B|32\\%|5.4|216|\n|QR-SW|Qwen3-0.6B + 32B|30\\%|5.4|220|\n|-|-|-|-|-|\n|QR-BERT|R2R-S + R2R-L|63\\%|5.4|179|\n|QR-LLM|R2R-S + R2R-L|**67\\%**|5.4|175|\n|QR-MF|R2R-S + R2R-L|58\\%|5.4|181|\n|QR-SW|R2R-S + R2R-L|62\\%|5.4|183|\n\nFrom these experiments, we conclude:\n\n1. R2R models generally serve as superior SLM/LLM candidates for query-level routing compared to original single-model setups, consistently advancing the Pareto frontier.\n\n2. The interaction between medium-sized R2R models and query-level routing involving smaller or larger R2R-LLMs is empirically non-trivial. This observation suggests exciting opportunities for future research into sophisticated routing strategies, which combine the search space of query and token-level routing methods.\n\nWe will incorporate these discussions into the revised manuscript, highlighting insights and promising directions for future exploration."}]

Author rebuttal:

Meta-review: This paper generates tokens rapidly from a small language model (SLM), and uses the state of the SLM to predict at each token whether a large language model (LLM) might have done something different, in which case it consults the LLM.  This is somewhat similar to speculative decoding, except that in speculative decoding the LLM checks every predicted token, whereas here it checks only the ones that are predicted to be divergent.  This may introduce errors but it also allows tolerance of unimportant divergences between the two models.

**Strengths:** A plausible and interesting method with careful evaluation and demonstrated improvements to the Pareto frontier.

**Weaknesses:** The reviewers (and the AC) identified some minor issues, but the only serious weakness raised was the restriction to greedy decoding, which the authors relaxed in the rebuttal.  

The other main weakness is that this is not simply a drop-in decoding method.  The predictor that decides whether it is worth calling the LLM requires training data that is expensively generated with the help of a domain-dependent verifier.  Such a predictor may not generalize to new domains.  (Even in-domain, the training method for the predictor [seems suboptimal](https://openreview.net/forum?id=DpeJYRFRQY&noteId=x5Nqh3v9So).)

**Rationale:** Good consensus among reviewers.

**Rebuttal period:** The authors were very responsive and provided many new experiments, which were appreciated by the reviewers.

-------
# Questions from AC during AC-reviewer discussion period that may be useful to authors

## connections to related work

Are they evaluating cost appropriately?  If I understand correctly, they generate an SLM token at *every* timestep.  Then they probe the SLM's state to decide whether an LLM would have been better.  If so, they append all previously generated SLM tokens to the LLM's KV cache so that the LLM can now make a prediction, and they use that prediction instead.

This means that a sequence like SSSSSLLLLLSS will be efficient.  (The initial SSSSS can be encoded in parallel when they are appended to the LLM context to generate the first L; the LLLLL are appended automatically as they are decoded; and the final SS don't need to be appended at all.)

But a sequence like SLSLSLSLSLSS has many switches to L, and hence less parallelism.  Until the final SS, each S must be separately appended to the LLM context in order to generate the subsequent L.  

To distinguish between these sequences, shouldn't they evaluate something like latency, not just activated parameter count (or FLOPS)?   And shouldn't their classifier be optimized to minimize that metric, meaning that it should prefer to reduce the number of switches to L (other things equal)?

## cost evaluation

Are they evaluating cost appropriately?  If I understand correctly, they generate an SLM token at *every* timestep.  Then they probe the SLM's state to decide whether an LLM would have been better.  If so, they append all previously generated SLM tokens to the LLM's KV cache so that the LLM can now make a prediction, and they use that prediction instead.

This means that a sequence like SSSSSLLLLLSS will be efficient.  (The initial SSSSS can be encoded in parallel when they are appended to the LLM context to generate the first L; the LLLLL are appended automatically as they are decoded; and the final SS don't need to be appended at all.)

But a sequence like SLSLSLSLSLSS has many switches to L, and hence less parallelism.  Until the final SS, each S must be separately appended to the LLM context in order to generate the subsequent L.  

To distinguish between these sequences, shouldn't they evaluate something like latency, not just activated parameter count (or FLOPS)?   And shouldn't their classifier be optimized to minimize that metric, meaning that it should prefer to reduce the number of switches to L (other things equal)?
Decision: Accept (poster)