Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectories?
Authors: Aochong Oliver Li
Tanya Goyal
Venue: ICLR 2026 Poster (2026)
Keywords: large langage models; language model reasoning; multi-model collaboration; off-trajectory reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose twin test framework to study LLM off-trajectory reasoning

Abstract: Reasoning LLMs are trained to verbalize their thinking process, yielding strong gains on reasoning tasks. This transparency also opens a promising direction: multiple reasoners should directly collaborate on each other's thinking on a shared trajectory, yielding better inference efficiency and exploration. A key prerequisite, however, is their abilities to assess usefulness of and build on other models' partial thinking traces -- we call this *off-trajectory reasoning*. Our paper investigates a critical question: can standard *solo-reasoning* training pipelines yield desired *off-trajectory* behaviors? To this end, we propose twin tests that capture the two extremes of the spectrum: **Recoverability**, which tests whether LLMs can backtrack from "distractions" induced by misleading reasoning traces, and **Guidability**, which tests their ability to build upon correct reasoning from stronger collaborators. Our study evaluates 15 open-weight LLMs (1.5B–32B) and reveals a counterintuitive finding -- "stronger" LLMs on benchmarks are often more fragile under distraction. Moreover, all models tested fail to effectively leverage guiding steps from collaborators on problems beyond their inherent capabilities, with solve rates remaining under 9.2% for math. Finally, we conduct control studies to isolate the effects of three factors in post-training on these behaviors: the choice of distillation teacher, the use of RL, and data selection strategy. Our results provide actionable insights for training natively strong reasoning collaborators; e.g., we find that sub-optimal recoverability behaviors of teacher models are transferred to distilled students even if the distilled data trajectories are correct. Taken together, this work introduces the framework for evaluating multi-model collaborations under shared reasoning, while revealing limitations of off-the-shelf reasoning LLMs.

Published as a conference paper at ICLR 2026
OFF-TRAJECTOR REASONING: CAN LLMs COLLAB- ORATE ON REASONING TRAJECTORIES
Aochong Oliver Li, Tanya Goyal
Department of Computer Science, Cornell University aochong1 i cscs. cornell e.ddu, tanyagoya @copagge e.edu
ABSTRACT
Reasoning LLMs are trained to verbalize their thinking process, yielding strong gains on reasoning tasks. This transparency also opens a promising direction: multiple reasoners should directly collaborate each other's thinking on a shared trajectory, yielding better inference efficiency and exploration. key prerequisite, however, is their abilities to assess usefulness of and build on other models' par- tial thinking traces - we call this off-trajectory reasoning. Our paper investigates standard solo-reasoning training pipelines yield desired off- off-trajectory behaviors? To this end, we propose twin tests that capture the two extremes of the spectrum: Recoverability, which tests whether LLMs back- track from "distractions" induced by misleading reasoning traces, and Guidabil- ity, which tests their ability build upon correct reasoning from stronger col- laborators. Our study evaluates 15 open-weight LLMs 1.5B-32B) and reveals a counterintuitive finding "stronger" LLMs on benchmarks are often more fragile under distraction. Moreover, all models tested fail to effectively leverage guid- ing steps from collaborators on problems beyond their inherent capabilities, with solve rates remaining under 9.2% for math. Finally, we conduct control studies to isolate the effects of three factors in post-training on these behaviors: the choice distillation teacher, the use of RL, and data selection strategy. Our results provide actionable insights for training natively strong reasoning collaborators; e.g., we find that sub-optimal recoverability behaviors of teacher models are transferred distilled students even the distilled data trajectories are correct. Taken together, this work introduces the framework for evaluating multi-model collaborations der shared reasoning, while revealing limitations of off-the-shelf reasoning L.MMs.
critical question:
INTRODUCTION
LLMs with thinking abilities, such as OpenAl's o-series (Jacch al. 2024). DeepSeck-R1 (Guo al. 2025), and Qwen3 Thinking (Yang et al.) (025a)). have recently emerged frontier models for complex reasoning tasks such as mathematics and coding. These models, trained with reinforcement learning with verifiable reward (RLVR) (Saao et al. 2024) or distillation (Hinton et al. 2015)) learn verbalize their thinking process and exhibit self-reflective behaviors (Gandhi et al. 2025). However, as these models are deployed in settings like agentic systems, their thinking traces are increasingly interleaved with tokens they did not produce tool outputs, code execution results, or retrieved documents.
Solo
Collaborative
Question
Question
O 0
O
3B
reasoning
reasoning
buman safety overseer
reasoning
reasoning
32B
reasoning
reasoning
answer
anSMeR
models can already interleave their thinking with exter- nal content, can multiple reasoners directly collaborate on a shared reasoning trajectory? In this paradigm (Figure [T). stronger model could correct weaker one's misstep mid- derivation, or a human overseer could steer reasoning away from unsafe path. More broadly, this has implications
Figure 1: Comparison of solo col- laborative reasoning setting. On the right half. LLMs of different sizes and functionalities collaborate on shared trajectory.

---

Published as a conference paper at ICLR 2026
Recoverability Test
Question
Original Ressoning
Original Reasoming
safety sptill-23-
Solo- O
siinl Alnsha so bave
lagn(t)
... New, aquic both "yy' -200 liteg
ansewer
Reasoning
Reasoner
Distracting Steer
Contime Reczoning
arower X
Off- Cofjeeaeeytory..omm Reasoning
Wai: Lat me tiis. nead to figert out the macimun age sanple that can be (aned wing aurbon-14 16 [..]
L... Waat. Let tse cleck calotiatirn Ir half ofc carbors in 730 yeurs [...]
350 years
Resispnerr
Guidability Test
Question
Original Reasoning
antwer
Wilocope has
kage ground baaed
Sale- Reasoning
sthink Hure problem about acrmentical objpets. [-.]
struggle to reason by
28
foraper two corcepaaiiia
Reasoner
iuages
Guiding Steer
Conzonue Rezzaning
Off Off-Trajectory Reasoning
OOL
for prapartion will the Boei plone of telesogge if the otjects are saproned [..]
danii
206265, but ence la sccadd, x E- 2220228..
answor
Stronger Ressoner
Ressonec
for cely i7B an
Figure 2: Illustration of the twin tests: we perturb a model's reasoning trajectories with off-trajectory steers to evaluate its recoverability (under a distracting steer) guidability (under guiding steer). The distracting steer is sampled from the same reasoner but for a different question.
for: (1) Efficiency: to balance performance and inference speed, large LLMs can focus on chal-
lenging derivations while offloading routine sub-steps (e.g., arithmetic checking) to smaller models (Akhauri et al. 2025 Yang et al. 2025c Chen et al. 2023). (2) Exploration: models/humans with complementary expertise can broaden the reasoning search by spawning diverse branches (Chen) ctt al. 2025 Qi et al. 2025 Pan et al. 2025) and composing their skills solve cross-domain tasks. (3) Safety: an overseer model or human can intervene to steer ongoing reasoning in a safer direction,
rather than abruptly terminating it (Wu et al. 2025 Zhang al. 2025 Korbak et al. 2025).
Most LLMs today are trained to reason entirely on their own what we term solo-reasoning. But multi-model collaboration require models reason over trajectories interleaved with off-distriuution tokens from other reasoners. We call this off-trujectory reasoning and ask: can off-the-shelf solo- reasoning LLMs handle off-distribution thinking tokens in their reasoning trajectories?
We approach this question by decomposing off-trajectory reasoning into two complementary tests, Recoverability and Guidability, and evaluating both in simulated collaboration scenarios (see Fig- ure [) The recoverability test evaluates whether LLMs can backtrack from misleading reasoning injected into their trajectory to continue their original correct reasoning. At the other end of the spec- trum, the guidability test evaluates whether LLMs can build upon correct yet incomplete reasoning from guiding models tackle problems they are unable solve via solo-reasoning
We systematically evaluate 15 gpeght LLMs across math and coding benchmarks MAA
2024 2025| (Hendrycks al. [021 Lewkowyez et al. 2022 He al. 2024 Gu et al., 2024 Chen Austin et al. Liu et al. 2023). Counterintutively. benchmark performance does not pre- dict off-trajectory robustness: AM Thinking 32B, the top math model (82.66 682.6 average), recovers only 33.4% of the time, whereas Qwen3-1. 7B (59.9%) recovers 98.4%. Overall, recoverability falls to 74.9% for math and 59.1% for coding on problems originally solved correctly. The guid- ability test reveals similar ceiling
2021
model exceeds 9.2% math, and while the same models show higher apparent guidability (up to 47.3%), most of the guiding reasoning already contains the answers F..3 These results show that benchmark optimization does not account for off-trajectory reasoning capabilities.
We further investigate how post-training decisions shape off-trajectory behaviors through controlled experiments on three factors: distillation teacher choice, RL training, and data selection strategy. Most strikingly, we find that the recoverability weaknesses of teacher models transfer to distilled students even when training uses only the teacher's correct trajectories. We also show that RL
We systematically test for correctness of reasoning in this paper. However, our framework can be extended for other aspects of alignment, e.g., can solo LLMs robustly reject unsafe collaborator trajectories?
2

---

Published as a conference paper at ICLR 2026
substantially improve off trajectory robustness where supervised fine-tuning saturates, and that aggressive data filtering can introduce high variance recoverability across training checkpoints.
In summary, work makes the following contributions:
1. We introduce the Recoverability and Guidability tests as a systematic framework for evaluating off- trajectory reasoning. Our setup complements existing standard solo-reasoning benchmarks by offering an orthogonal view into model reasoning capabilities. ($2)
2. Equipped with this framework, we evaluate 15 open-weight LLMs for off- off-trajectory reasoning. Our analysis reveals key limitations of "strong" solo reasoners and shows that almost all models fail at exploiting correct guidance to improve beyond their inherent capability limits. ($3)
3. We conduct the first controlled study isolating the direct effects of post-training decisions oifetrajectory behaviors. Our experiments reveal that a teacher's recoverability weaknesses trans- fer to distilled students, that RL yields substantial recoverability gains where SFT plateaus, and that "less-is-more" data filtering introduces high variance in off-trajectory robustness. ($4)
2 TWIN TESTS FOR OFF-TRAJECTORY REASONING
Preliminaries and Notations. Let M be reasoning model and (g, a²) be a training or test data
point. In standard solo-reasoning, M generates reasoning trajectory r = [ri, 22, rs....l. a se- quence of reasoning units, and final answer for an input question q. i.c. (r.a) M(19)). The granularity of reasoning unit can be flexibly determined.
In contrast, in the collaborative setting, multiple models or different instantiations of the same model contribute different parts to the reasoning trajectory r. Recent work has explored some collabora- tion strategies, such as dynamically off-loading reasoning sub-parts bub-parts to weaker/stronger models (Yan) et al. 2025 Zhou et al., 2025 Akhauri et al. 2025; Yang al. (025c),, orchestrating meta-thinking and reasoning agents (Wan et al. (0225), tooling (Jin et al. 2025). injecting control tokens to extend
reasoning (Muennighoff et al. 2025). aggregating parallel samples (Zhao et al. 2025 Qi ct al.
2025) during both training and inference time.
The success of such collaboration hinges on the main model M's ability process and build upon
trajectory mixing both in- and off-distribution reasoning units [r ...M', ..M ..... In this paper, we instantiate a simplified setup of two-model collaboration to probe off-trajectory reasoning capabilities in frontier open-weight L.LMs.
Two-model Setup. We simulate a collaboration between two reasoning systems, where the main model M and the collaborator Mstoer jointly contribute to an off-trajectory reasoning [ In practice, we construct y by sampling from the main model M and stopping generation at m tokens, i.e. | = m. Similarly, paterr is sampled from the collaborator with | limited to n tokens. To measure off trajeereecory reasoning performance, we concatenate these two incomplete trajectories construct a shared off aff-distribution trajectory. Finally, we sample reasoning completion and final answer from M conditioned on the original question and this trajectory.
ysteer]]
(roff ofll)
M( q, [r steer))
For domains with verifiable rewards, we measure success as the accuracy of the final answer off against the ground truth a"
Considerations for designing the steer. This simplified setup enables to flexibly and scalably test how M behaves when pleer falls at two extremes. We design twin tests: (i) Recoverability, which tests whether LLMs resist incorrect and distracting steer, and backtrack to previous reasoning, and (ii) Guidability, which tests models' abilities to successfully leverage a guiding steer surpass its solo-reasoning ability.
These twin tests differ mainly in two aspects: (1) the selection of test questions, and (2) the construction of steered trajectories [r**, ytteerr. Given original test set and test model M, our protocol automatically constructs M-specific off-trajectory datasets for both tests, each consisting of (q. fr , steeer], a²) a²). The overall process for this is shown in Figure and described below.
3

---

Published as a conference paper at ICLR 2026
2.1 RECOVERABILITY TEST
Selecting test datapoints {(g, a*) a²)). For a given test model M, we select the subset of test ques- tions that M correctly answer in solo-reasoning, i.e. a a", where (r, M.lg). This disentangles the effects of distracting steers from M's inherent capabilities.
Constructing steered trajectories. The trajectory consists of two parts: ro8 and ytter We truncate the reasoning trajectory from solo-reasoning, to the first m tokens to obtain 708, always at the end of the nearest sentence to preserve coherence.
We require y yteer to be a strong distractor for the test model M. However, it is difficult to a priori determine which model Msteer and steer ysterr will achieve this reliably, Therefore, we simulate the distraction psteer by sampling from M itself, but conditioned on different question q' So. if M is distracted to blindly complete ysteer, reasoning is then guaranteed be incorrect. In practice, we control the length of yteer by truncating it to the first n tokens of r', where (r', "²) ~ MMI''") In our experiments, we control the strength the distractor by varying n, i.c. |psteer] and the insertion point by varying m, i.e. |/ We describe procedure details and design rationale in Appendix E.1
2.2 GUIDABILITY TEST
Selecting test datapoints { }²)}. We select the subset of test questions for which the solo- reasoning solve-rate either out of samples. These are problems M cannot reliably solve on its own, so any improvement must come from the help of guiding steers.
Constructing steered trajectories. First, unlike the recoverability test, do not include M's own partial reasoning r"8 (i.e., set m = 0): the guide's steer is placed at the very beginning of the thinking, before M has produced any tokens. This is because r might already contain errors that anchor M to wrong direction, thereby confounding the measurement of guidability.
We construct pteer using a stronger reasoner Msteer as the guide, i.e. with a higher benchmark performance than M. Figure 2 illustrates this. To test whether M can build on Mateer's 's correct reasoning, we only provide the first n tokens of the complete trajectory. practice, we vary the "amount" of guidance by varying to different fractions of the complete trajectory from the guide. Moreover, we use multiple guiding models Msteer to construct independent steers for each q. This allows us measure guidability under different steer distributions and amount of guidance.
OFF-THE-SHELF EVALUATION & RESULTS
3.1 EXPERIMENT SETUP
Models, Datasets, and Benchmarks. We run our experiments on 15 open-weight models. To illustrate the relationships between these LLMs, we group them into four families (see Figure B)
DeepSeek-R1 (Guo et al.
2025): : R1 R1-wwen-1, 51/.///3/22222 and 21-Llama- 83 are distilled
from DeepSeeK-R using supervised fine-tuning (SFT).
Qwen3 Yang al. 2025a): 3 32B is trained using RL for reasoning without distillation,
while Owen3-1. 3/ 87/8/33332 33B-A3B are distilled from -wen.3-235P and Qwen3-32B.
QwQ: QwQ-32R Qwen Team 2025) is trained from Qwen2. 5--32BBBase model with RL.
OpenThinker3-1 1000-11111100 7B (Guba et al.) 2025) are based Qwen2. 10-nsttrucc and dis- tilled from QwQ on 1.2M curated math and coding examples.
Community: DeepScalee-111 53 (Luo al. 2025 and DeepMath-1 5B (He al. 2025b)
are trained using RI. on R1- 101--wenn11 5B using DeepScaleR and DeepMath datasets respectively. LTTM--32B (Ye et al. 2025) SFT from Qwen2. 5 32 Inst ruct on the LIMO dataset of 817 examples. Finally, AM- AM-Thinking. 32B (Ji al. (025) is a Swen2.5-32B 5-322-B3BBa Base model first distilled on 2.84M examples and RL on 54K math and coding questions.
We evaluate on 1, 507 math questions from five benchmarks: AIME-2024 (MAA 2024), AIME- 2025 (MAA 2025), MATH-500 (Hendrycks et al. (021). Minerva Math (Lewkowyez et al. 2022
and Bench (He et al. 2024). For code, evaluate on 1, 762 questions from four bench-
marks: CruxEval (Gu et al. 2024). HumanEval (Chen 202I). MBPP (Austin et al., 202 l), and
EvalPlus (Liu et al.
2023).
4

---

Published as a conference paper at ICLR 2026
DeepSeek-R1
Community
Qwen3 235B
QwQ
SFT
SFT
QwQ 32B (RL)
RI-Qwen-1.5B
RL
DeepScaleR-I.5B
@
(wen3-32B(RL)
SFT
RI-Qwen-7B
Deep
8
@wwn3311.7B
Open Thinker3 1.5B
RI-Llama-8B
@
@LITO-32B(FFT)
Qwen3-8B
Open Thinkes 7B
R1-Qwen-32B
Qwen3-30B-A3B
Figure 3: 15 open-weight LLMs grouped into four families. The branches indicate the source from which LLMs are derived, and the colors indicate SFT/RL training methods.
Hyperparameter Settings. All LLMs are evaluated under the same hyperparameter settings: maximum of 32K tokens, temperature 0.6, top-p 0.95, and no system prompt. For each question, we sample 8 completions and report the average Pass
Recoverability and guidability Setup. Following the protocols in 2.1 we sample 200 original trajectories yog and trajectories as distracting steers (rsteer) ) for each L.M. By default, we set n. i.e., |rsteer], to be 0.2 times the original length of the distracting trajectory; this leaves sufficient tokens for off-trajectory completion. We set m, i.c., |r*], to be O, 0.2, 0.4, 0.6, and times the length of the original reasoning from the main model. We report recoverability on two subsets: (1) shared subset that includes questions that subset that samples questions independently for each LLM following the criterion defined in $2.1
LLMs can fully solve (8 out 8), and (2) individual
We instantiate the guidability tests using DeepSeek Preeseee--LL. RRR Qwen3-2333 and BWQ-32B as Msteer to sample guiding steers , pttter Since the best LLMs almost saturate the benchmarks, only evaluate on the remaining 10 LLMs that have enough questions with solve rate S 1/8 (Table [3). We set n, i.e., | to be 0.2, 0.4, 0.6, and 0.8 times the total tokens in the guide's reasoning. Similar to the recoverability test, we report guidability scores on two subsets: shared (intersection across models) and individual (per model).
3.2 RESULTS
Our main results are shown in T We group models into low, medium, and high tiers based on their solo-reasoning performance (reported in the Avg. Benchmark column) and report recoverability and guidability results on both shared and individual subsets.
Finding 1: Stronger solo-reasoners J stronger collaborators. Surprisingly, we find that recov- crability largely orthogonal to LLMs solo-reasoning performance. Particularly, we highlight models in the low benchmark tier for math, such OpenThinker3 5B and Qwen3-11 -1. 7B, exhibit substantially better recoverability than medium and high tier models like Dw0-323 and Twen3-32B Noticeably, the best performing solo-reasoning math model MM-Thinking reports the second worst recoverability performance. Similarly, LIMO ATMO-32B claimed to surpass prior SFT approaches using only 1% of training data recovers less than 30% of the time the math benchmarks. Across models, the average recoverability on the shared math subset is 74.9%, 25.1 percentage-point drop from the original 100% solve rate, when their trajectories are perturbed with tangential distractions.
We observe similar trends for code in Table T The three best performing models in the high tier (R1-Qwen-32B. AM-Thinkings 32B and QwQ-32B) report low recoverability rates (in the range 40 48% range), while three out of four models the low tier report greater than 65% re- coverability rate on the shared test subset. Combined, these findings on both math and code provide evidence that models optimized heavily for popular benchmarks may have hidden vulnerabili- ties, particularly in off-trajectory reasoning.
Finding 2: The invisible guidability ceiling. Our results show that all LLMs report exceptionally low guidability scores for math; none of the models report > 10% on the shared subset for math (and at most 35.0% on the individual subset). This indicates that all math models, regardless their solo-reasoning capabilities, struggle to effectively build upon guiding trajectories. Crucially, the performance does not improve even when models are paired with their own distillation teacher, i.c., the model whose samples they were trained on (see Table 16 for full set of results). For example,
5

---

Published as a conference paper at ICLR 2026
Model
Family
Benchmark Avg.
Recoverability Sh.
Guidability Ind.
Ind.
Sh.
Math
Low Benchmark Scores
R1-Qwen-1.55
DS-RI
47.5
60.6
38.67v2
3,0
28.4
R1-Llama-SB
DS-R1
54.1
81.4
49,6,-1
8.7
35.0
DeepMath-1 1,5B
Comm.
88.0r+10
61.8
3.4 1-1
27.1 ++2
DeepScaleR-I.5tt
Comm.
55.3
82.4p+5
52.9
4.1
29.8
SpenThinker3-1158
QwQ
59.2
95.2p49
71.8
5.7
32.7
Qwen3-1.7B
Qwen3
59.9
98.4 449
74.6 4.9
6.1 -40
29.9
Medium Benchmark Scores
R1-Qwen-7B
DS-R1
64.6
73.5
45.8,
6.0 L-2
19.7
LIMO-32B
Comm.
67.3
29.31-2
18.5
8.8 -40
21.5 ..
OpenThinneer377B
QwQ
72.1
85.6
74.5
9.1 -40
20.6 .-7
RI-Qwen-32B
DS-RI
72.3
69.8
45.64-6
-40
22.5
High Benchmark Scores
Qwen3-8B
Qwen3
79.1
85,9
68,8
N/A
N/A
QwQ-32B
QwQ
80.5
79.7
62.6,41
N/A
N/A
Qwen3-324 32B
Qwen3
81.0
71.848
56.94s
N/A
N/A
Qwen3-30B--A3B
Qwen3
81.1
87.8
60.0gs
N/A
N/A
AM-Thinking-32B
Comm.
82.6
33.4
25.31-13
N/A
N/A
(b) Coding
Low Benchmark Scores
21-Qwen-1.55
DS-RI
40.6; 40
28.7 4.0
24.1, 42
40.1 +2
OpenThinker. 1,5B
QwQ
78.2- +12
53.2-
35.8 +2
49.4. +2
DeepScaleR-1.5t
Comm
58.8
65. 1 +6
46.2-
23.24
37.3,
DeepMath-1 1.5B
Comm.
59.8
74.6; +7
56.3 +7
20.94 -3
36.24 -3
Medium Benchmark Scores
Qwen3-1.7B
Qwen3
74.7
77.4, +8
61.4
47.3
56.4 +2
R1 R1-Llama-8B
DS-R1
76.0
56. 1 +1
46.2- +2
47.2 +0
54,4- +0
R1 R1-Qwen-7B
DS-RI
77.7
54.5 -1
37.4,
45.7 -2
53.1 -2
Op
QwQ
80.1
74.9 +4
68.2- +6
N/A
N/A
High Benchmark Scores
Qwen3-8B
Qwen3
87.4
67.6 +1
60.9 +3
N/A
N/A
Qwen3-32B
Qwen3
87.8
46.8 -6
30.8 -8
N/A
N/A
Qwen3-30B
Qwen3
89.0
59.61 -3
47.7.
N/A
N/A
RI RI-Qwen-328 32B
DS-RI
89.7
40.7_
34.8.
N/A
N/A
AM-Thinking 32B
-h2B
Comm.
89.7
43.0 -9
40.7.
N/A
N/A
QwQ-32B
QwQ
90.5
47.9 -9
44.2
N/A
N/A
Table I: Recoverability and guidability results math (a) and coding (b). Columns report benchmark averages and recoverabilitygguidaae subsets. Models are grouped into low/medium/high tiers by Benchmark Avg. Subscripts indicate rank changes relative the benchmark ranking (+k rise. -k drop); green (t) denotes improvement, red (4) decline. "DS-RI" = DeepSeck-RI family, "Comm." evaluated. Across both domains, benchmark performance is largely orthogonal recoverability- the best benchmark model in each panel exhibits among the worst recoverability.
scores for shared (Sh.) and individual (Ind.)
Community models. N/A = not
Owen3-1.. 7B shows no guidability gains when guided by Owen 235B compared to other mod- els. We observe better guidability results the coding benchmarks, with most models reporting 20 50% improvement with external guidance, roughly corresponding to their solo performance.
To understand the failure modes in the guidability setting for math, We find that even these low guidability scores on math are artificially inflated. Since we truncate the guiding steer different lengths, it is possible that some partial Tsteer already contain the correct
further investigations.

---

Published as a conference paper at ICLR 2026
Model
Teach. (%)
Ans.? (%)
A
Desplatt
R29
R1-Qwen-1.55
25.6
U100-228
Dward 373-430
Oward:2
DeepScaleR-1.55
29.8
23.3
6,5
Ownek
R2-Ulam
RI-Llama-8B
DeepMath-L.5B
27.1
35.0
22.9
21.8
13.2
4.2
Recoverability (shared)
100
OpenThinner3-L15B
32.7
26.9
5.8
80
Qwen3-1.78
29.9
18.0
11.9
RI-Qwen-7B
19.7
12.1
7.6
LIMO-32B
21.5
10.2
11.3
OpenThinker3-7B
20.6
13.8
6.8
40
RI-Qwen-32B
22.5
11.2
11.3
Avg.
26.7
18.6
8.1
20
Analysis of guidability results. Teach. = guidability score (individual); Ans.? fraction of steers already containing the correct answer; A Teach. Ans. (pp).
Table 2:
0%
60% Inserted Position (% in Trajectory)
20%
40%
Figure 4: Recoverability (shared) across posi-
tions (%) of the original trajectory.
answer derivation. In such cases, we expect the guidability test to be trivially easy. In Table 2 we report the percentage of guiding steers that already contain the correct answer (Ans.? column). We find that this is true for 18.6% of steers on average (see Table [14 for breakdown by steer length). However, we find that LLMs can often fail to recognize such correct reasoning, reject the given answer and pivot to an incorrect path, resulting the low guidability scores. This suggests that conditioning models correct but leverage them and surpass their inherent capability limits.
traces does not enable them to successfully
Finding 3: The beginning of model reasoning is critical for recovery. To better understand the recoverability trends in Table T we visualize the recovery rates separately for different percentages (%) of the original thinking trajectory where the distracting steer is inserted. Figure 4 shows these results ? Interestingly, we observe a consistent pattern across models distraction at the very start (0%) of the trajectory leads the largest degradation. In particular, the degradation 0% is sub- stantially higher than at 20% position. This is surprising as models tend to restate the question at the opening stage and rarely include substantive problem solving. Given these results, we hypothesize that re-stating the question at the start is critical for models to anchor later reasoning.
To test our hypothesis, we conduct an ablation that re-instantiates the recoverability tests but pre- serves the first paragraph of the original trajectory. We find that most LLMs exhibit noticeable im- provements across positions after this change, especially at the 0% position (complete set of results are included in Table 7 in the Appendix). In fact, the average recoverability score exceeds 83.5% for all models with this small tweak, with the exception of 1LMC-323 and AM-Thinking-- 322B, which improve but remain below 60% consistent with their broader recoverability weaknesses identified in Finding 1. This clearly shows that while re-statement of the problem does not add new information, it is critical for LLM off off-trajectory reasoning.
CONTROL STUDY ON POST-TRAINING DECISIONS
Section 3 reveals that models with similar benchmark scores can have dramatically different off- trajectory robustness. However, the L.MSs used our experiments in Section differ in base mod- els, training data, and post-training recipes. Therefore, the previous results alone cannot pinpoint what drives these differences.
To disentangle these factors, we conduct controlled experiments that isolate the effects of (1) teacher models used for distillation $4.1] (2) conducting RL based post-training after SFT in $4.2, and (3) quality heuristics for data filtering in $4.3 We conduct all experiments in this section on the math benchmarks, as most of the small open-source models are specialized for math.
4.1 HOw Do TEACHERS' BEHAVIORS AFFECT DISTILLED MODELS?
Hypothesis. observe in Table that LLMs distilled from DeepSeek-RY generally have lower recoverability scores compared to those from QwQ and Qwen3. This is despite the fact that most of
Thhe full set of results for both shared and individual metrics reported in Tables and 8 the Appendix.

---

Published as a conference paper at ICLR 2026
AM-Distill 1.58
QwQ-Distill 1.58
Owen3-Distill 1.58 0.5
0.5
0.5
0.4
Recoverability
0..4
(p <005))
Guidability
0.4
MATH500
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Training Step
Training Step
Training Step
AM-Distill
38
QwO-Distil 36
Owen3-Distil
Recoverability
(p 005
MATH500
0.5
0.5
Guidablitty
0.5
0.4
0.4
0.4
0.3
0.3
0.2
0.1
0.1
0.1
200
300
500
100
300
400
500
100
200
300
500
Training Step
Training Step
Training Step
Figure 5: Qwen2.5 models (1.5B and 3B) distilled from AM (-Thinking) 322 show consistently lower recoverability than those distilled from QwQ-32B or 3-2n3-33B,, while having similar per- formance on benchmark and guidability; the gap is significant after step 300 (p S .005). Stars mark each model's peak over training steps.
them are trained from similar base models using distillation. Therefore, we ask do distilled models inherit the vulnerabilities of their teachers' off-trajecton behaviors through distillation?
Setup We conduct our experiment with three LLMs as the distillation teacher models: (WThinkking-332 32B), Q (-32B), Owen3 (2BB). We choose the AM model since it has a similarly high benchmark performance as OwQ and Owen models in Table [ but significantly lower recoverability. We supervised fine-tune two Qwen2. models (1.5B and 3B) on correct tra- jectories from each teacher separately (more details in Appendix G
1. 5B/BB) on MATH500 for benchmark performance and twin tests. We report results for different checkpoints during training Figure 5) we report the overall benchmark (leftmost column), recoverability (middle column) and guidability (rightmost column) performances. Additionally, we highlight checkpoints with signifi- cant differences (p < 0.005) based on two-sample t-tests.
We evaluate the distilled models (AM- http://ww///www--Disttt
Results: Students mirror their teacher's recoverability performance. Our results in Figure 5 show that AM-Di Distill models, i.e. models trained on trajectories from AM-Thinking--33 322, show significantly lower recoverability than QwQ- and Qwen counterparts after step 300, despite similar benchmark and guidability scores. This recoverability gap persists across all model sizes that tested and also remains consistent different positions of the reasoning tra- jectories (Appendix G). We highlight that these surprising recoverability trends mirror the trends observed for the corresponding teacher models in Table
Crucially, all teachers are distilled using only their correct trajectories yet the recoverability gap persists. This means that hidden vulnerabilities of teachers propagate through distillation even when the training data contains no errors, implying that vulnerability is encoded in the reason- ing style, not in the correctness of individual solutions. Therefore, our twin tests can serve as an additional criterion for selecting distillation teachers beyond correctness alone.
4.2 CAN RL FURTHER IMPROVE -FF-TRRJCTT REASONING AFTER SET SATURATES?
Hypothesis. In Table I we do not observe a consistent advantage of RL over SFT distillation on twin tests. However, training recipes of these models are different, making impossible to draw concrete conclusions about RL's impact. Here, we ask can RL further improve both recoverability and guidability even after SFT has safurated?
8

---

Published as a conference paper at ICLR 2026
GRPO 1.58
Baseline 1.58 1SeT@Step 500)
GRPO 38
Baseline 38 (SFT@Step 500)
0.70
0.50
0.65
0.45
MATH500
0.60
Guidabilis
0.40
0.35
0.55
0.30
0.50
0.25
0.45
0.20
0.40
0.15
100
Training Step
Training Step
Training Step
Figure 6: GRPO 1.5B and 3B (from SFT@ Step 400) show noticeable gains on benchmark, recov- erability, and guidability compared the initial checkpoint and basclines (SFT @ Step 500). This improvement is consistent over RL training. Stars mark the peak values over training steps.
Setup. We use distillation checkpoints from Section 4.1 AM-Distill 1.5B and 3B models at step 400 the initial policy for RL training. This choice is motivated by: (1) we observe SFT saturates on benchmarks and twin tests after step 400; and (2) AM-Distill is shown to perform poorly in recoverability, making it more suitable to test the effects of RL. We train both models on MATH8K dataset with Grouped Relative Policy Optimization (GRPO) (Shao et al. 2024).
Results: RL training yields substantial improvements in recoverability. Figure shows the impact of RL training on benchmark scores, recoverability and guidability. While all scores improve with RL, see noticeably high recoverability improvement (e.g. 15.3% (8.9%% accompanying slight increase on benchmark scores (5.4%-7.66 7.6%% and guidability (8.3%-8.7) 8.7%). Notably, RL train- ing completely bridges the gap in recoverability that observed in Figure between AM-Distill and QwQ-/Qwen3-DD poses the model to: SFT trains exclusively on successful demonstrations, teaching what correct reasoning looks like; RL, by contrast, exposes models to failed trajectories and explicitly re- wards recovery, teaching what do when reasoning goes wrong. We leave more thorough investigation of the mechanisms behind observed improvement to future work.
models. We hypothesize that the key difference is what each method ex-
4.3 DOES LESS DATA ALWAYS LEAD TO POORER RECOVERABILITY?
Hypothesis. Recent works have shown that data quality is critical for strong reasoning ca- pabilities (Dang & Ngo 2025) Albalak et al.
FULL-8.8K 3B
LIMO-600 3B
FULL-8K 3B
LIMO-800
0.5
2025 Guha et al. (025). The " "Less-Is-More"
(LIMO) hypothesis (Ye et al. 2025) pushes for an extreme version of this claim minimal
Recoverability
0.4
amount of "high--uality" data is sufficient to elicit reasoning. (Ye et al. 2025) curates the
0.3
0.2
LIMO dataset of 817 examples filtered based heuristics and support their claim with the per- formance of LIMO-32 32B model on popular rea- soning benchmarks. However, we observed contrary result in There the L.MM--325 model reported the worst recoverability despite decent solo-reasoning performance. To under- stand this, we ask is less-is-more paradigm inherently limited for off off-trajectory reasoning?
0.40
0.45 0.50 0.55 0.60 0.65 0.70 MATH500
Figure 7: LIMO-600/-800 3B models exhibit greater variance in recoverability than FULL- 8K/8.8K 3B. Colors: FULL, LIMO. Markers: square = contains data from LIMO-800, star otherwise. We observe that model checkpoints trained on high-quality but limited quantity of data show high variance in recoverability scores across similar benchmark score values.
Setup. We train Qwen2. 5-33 ----Base
Base mod- els on two larger datasets of mixed "quality" and two smaller ones of only "high-quality" data: (1) FULL-8K: MATH8K dataset distilled from QwQ 32B in $4.1, (2) FULL-8.8K: a mix
of FULL-8K and the LIMO dataset. (3) LIMO-800: the LIMO dataset, and (4) LIMO-600: 600

---

Published as a conference paper at ICLR 2026
"challengingg" examples extracted from following the "LIMO" principle, i.e. classified as Level-5 difficulty and with long reasoning trajectories. Figure plots recoverability scores against bench- mark scores at different checkpoints during training.
Results: Models trained on less data are not necessarily worse on recoverability but exhibit extremely high variance between checkpoints. For example, we observe that LIMO-60 600 and LIMO-800 3B models show markedly different levels of recoverability against similar benchmark scores, On the other hand, FULL-SK and FULL-8.8K models trained on larger datasets have mini- mal variance across checkpoints with the same benchmark scores.
Our results emphasize that "over-optimizing" benchmarks through data filtering could introduce unwanted biases in off-trajectory. any off-distribution, reasoning while not being captured by standard solo-reasoning evaluations. Our tests can complement existing criteria for selecting check- points with higher robustness out-o--distribution scenarios.
RELATED WORK
Reasoning LLMs. Recent post-training advances have led to substantial improvements on math and coding benchmarks (Huang & Yang 2025 Guo et al., 2025). with both closed- and open-source
reasoning LLMs emerging since OpenA o o-I (Jach et al. [2024), e.g. [uoo et al. (2025); ): Yang et al.)
; Ye et al. (025); Ji et al. (2025). These models are trained to produce extended reasoning traces using RL al aggorttmms such as PPO (Schulman et al.) 2017), GRPO (Shao ict al. 2024), or their variants with verifiable rewards. At smaller scales, models are primarily trained with distillation (Hinton et al.) 2015). We analyze fifteen representative open- open-weight reasoning LLMs spanning diverse families and training paradigms.
(2025a); Guha et al. 2025
LLM Reasoning Intervention and Collaboration. Recent studies intervene LLM reasoning to understand and control their behaviors, including perturbing intermediate steps to examine their faithfulness (Areuschin al. 20255 Baker al. (2225), improving instruction following and align-
ment behaviors ( [Wu et al.) 2025 Muennighoff et al. (0055), and interpreting (Lee et al. 2025b Marjanović al.) 2025) and stress-testing cognitive behaviors (Gandhi et al. (025). Wen et al. (2025) examine the impact of thinking patterns on outcome correctness, while He et al. (225a)) Lec al. (2025b) systematically categorize different types of reasoning strategies and errors. In closely related work, He et al. (2025a) investigate LLMs' ability to recover from unhelpful thoughts. Our work differs three ways: (i) our distractors are sampled from models rather than hand-designed, making them naturally model-specific; (ii) we introduce guidability as complementary test of lever- aging correct off-trajectory guidance; and (iii) we trace fragility to specific training choices through controlled experiments.
Our work is also closely related to hybrid parallel and serialized scaling approaches (Pan et al, 2025), including offloading challenging reasoning parts larger models Akhauri et al. (2025); Yang et al. (2025c) and orchestrating different models for high-level planning and downstream reasoning Lee al. (2025a): Wan et al. (2025). In contrast these systems, we evaluate how solo-reasoning LLMs fail when asked to continue from a shared reasoning trajectory.
6 CONCLUSION
In this work, we investigate off-trajectory reasoning LLMs their ability think on trajectories steered by other reasoners. Through our twin tests Recoverability and Guidability we evaluate 15 open-weight reasoning L.MS and find that standard benchmark performance does not predict off- off-traaectory robustness: models that excel at solo reasoning can struggle recover from erroncous steers or leverage correct guidance from collaborators.
Our controlled experiments reveal that these weaknesses not incidental but directly shaped by training decisions teacher choice propagates hidden vulnerabilities through distillation, RL im- proves robustness where SFT saturates, and data filtering trades stability for variance. Together, these findings show that off-trajectory robustness must be explicitly accounted for during training; it does not emerge byproduct of benchmark optimization.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This project was partially supported by NSF grant IIS-2433072, and a gift from Google. We grate- fully acknowledge of the research computing resources of the Empire Al Consortium, Inc, with support from Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation.
REFERENCES
Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F AbouElhamayed, Yueying Li, and Mohamed S Abdelfattah. Splitreason: Learning to offload reasoning. arXiv preprint arXiv:2504. 16379, 2025.
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. Chain-of-thought reasoning in the wild not always faithful. arXiv preprint 2rXiv:2503.08679, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry. Quoc Le, al. Program synthesis with large language models. arXiv preprint urXiv:2108.077
2021.
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojeiech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint 2rXiv:2.033. 2025.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptise Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/6 2023.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models. arXiv preprint arXiv:2505. 2025.
Quy-Anh Dang and Chris Ngo. Reinforcement learning for reasoning in small Nmss: What works
and what doesn't. arXiv preprint arXiv:2503. 262222533..99,,... 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cogni- tive behaviors that enable self-improvin reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01 01 2025.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. Cruxeval: benchmark for code reasoning, understanding and execution. In Proceedings of the 41st International Conference on Machine Learning, pp. 16568-16621, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint (iXi::25066041778, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. BennhihBBench: A challenging benchmark for promoting AGI with olympiad-leve bilingual multimodal scien- tific problems. In Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the
11

---

Published as a conference paper at ICLR 2026
62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). pp. 3828-3850, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 1.11666//1/2022.cllloogggg
URL https://acllatthooooy.oooo org/2024
acl- sel-long.2111 211/
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang. Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprini arXiv:2502. 19361, 2025a.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang. Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint 1rXiv:2504.11466 2025b.
Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https: http///peeeevieeeeeeeeeeeeee / net ottp:/eveee..neeerroooommmdaaaa
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 103.0033, 2015.
Yichen Huang and Lin F Yang. Gemini 2.5 pro capable of winning gold at imo 2025, arXiv preprint arXiv:2507.15855, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rXiv:2412.167220 2024.
Yunjie Ji, Xiaoyu Tian, Sitong Zhao, Haotian Wang, Shuaiting Chen, Yiping Peng, Han Zhao, and Xiangang Li. an-thinkking-ll Advancing the frontier of reasoning 32b scale. arXiv preprint arXiv:2505.083/1 08 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXir:2503.09516 2025.
Tomek Korbak, Mikita Balesni, Flizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, et al. Chain of thought monitorability: A new and fragile opportunity for ai safety. arXiv preprint arXiv:2507.1 / 2025.
Hynek Kydlíček. Math-Verify: Math Verification Library.
URL
com/
huwgiiggacce/mmtteooo teriattt--errfy
Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, and Jinwoo Shin. Collaborative Ilm inference via planning for efficient reasoning. arXiv preprint arXiv:2506.1157 2025a.
Seongyun Lee, Seungone Kim, Minju Seo, Yongrae Jo, Dongyoung Go, Hyeonbin Hwang, Jinho Park, Xiang Yue, Sean Welleck, Graham Neubig, al. The cot encyclopedia: Analyzing, pre- dicting, and controlling how reasoning model will think. arXiv preprint arXiv:2505.10185 2025b.
Aitor Lewkowyzz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857 2022.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 211-2552-1172, 2023.
Michael Luo. Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol ol-preeeew with 1.5b model by scaling ri, 2025. Notion Blog.
12

---

Published as a conference paper at ICLR 2026
MAA. American Invitational Mathematics Examination 2024 I & II. Sttps://maa..
maa- www.ttaaioonalmompettttt....
2024.
MAA. American Invitational Mathematics Examination 2025 I & II. attps://maa.org
maa lata-inven invitational www.:aatiionall.oompttiiii.ssss. 2025.
Sara Vera Marjanović, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lü, et al. Deepseek-rl thoughtology: Let's think about Iim reasoning. arXiv preprint arXiv:2504.00 07128, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19399 2025.
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprini arXiv:2504. 15466, 2025.
Jianing Qi, Xi Ye, Hao Tang, Zhigang Zhu, and Eunsol Choi. Learning to reason across parallel samples for Ilm reasoning. arXiv preprint arXiv:2506.0900 09014, 2025.
Qwen Team. Qwq- 32b: Embracing the power of reinforcement learning, March 2025. URL hhttps://qwent githbb 1wtp:///w.ooooooog////////2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xiv:1777.0634 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033300 2024.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang, Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, et Rema: Learning to meta-think for iims with multi-agent reinforcement learning. arXiv preprint 150v::250300900... 2025.
Pengcheng Wen, Jiaming Ji, Chi-Min Chan, Juntao Dai, Donghai Hong, Yaodong Yang, Sirui Han, and Yike Guo. Thinkpattems-211 A systematic study on the impact of thinking patterns in Ilms. arXiv preprint arXhv:2503.12918 2025.
Tong Wu, Chong Xiang, Jiachen T Wang, G Edward Suh, and Prateek Mittal. Effectively controlling reasoning models through thinking intervention. arXiv preprint arXiv:2503.24370 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- ooflcolicy guidance. arXiv preprint arXiv:2504. 14945, 2025.
Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
Sohce Yang. Sang-War Woo Lee, Nora Kassner, Daniela Gottesman, Sebastian Riedel, and Mor Geva. How well reasoning models identify and recover from unhelpful thoughts? arXiv preprint arXiv:2506. 10979, 2025b.
Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: Enhanc- ing small-model reasoning with large model guidance inference time. arXiv preprint arXiv:2504.12329, 2025c.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chem, Shijie Xia, and Pengfei Liu. LIMO: Less is more for reasoning. Second Conference on Language Modeling, 2025. URL https://
openreview net/ net/forunt -
Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, and Yinpeng Dong. Realsafe- rl: Safety-aligned deepseek-rl without compromising reasoning capability. arXiv preprint ar0iv:2504. 10081, 2025.
13

---

Published as a conference paper at ICLR 2026
Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Ku- likov. The majority is not always right: RI training for solution aggregation. arXiv preprint 05t1::2509006870, 2025.
Ruiyang Zhou, Shuozhe Li, Amy Zhang, and Liu Leqi. Expo: Unlocking hard reasoning with self-explanation-guuieee reinforcement learning. arXiv preprint arXiv:2507.02821 2025.
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? Advances in Neural Information Processing Systems, 37:123846-123910 2024.
14

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODEL USAGE
In this paper, we use Al with great caution for polishing the language of some texts that are originally written by the authors.
B LLM-AS-A-JUDGE PROMPT
H+t System Prompt
You are an unbiased examiner who evaluates whether student anawer to a giver question is correct
Your task is to determine if the student's final answer matches the standard answer provided, based solely correctness and the question's specific requirements.
Do not perform any additional calculations or reinterpret the question. Simply compare the student's answer to the standard answer to determine if it satisfies the question's requirements,
Focus strictly on:
1. Understanding the exact requirement of the question.
2. Comparing the student's final answer directly and rigorously the provided standard answer.
Your task is not to solve the problem but determine whether the student's answer is correct based on the question's requirementsr Avoid any unnecessary analysis, assumptions, or re-solving the problem.
Note:
For intervals/ranges: The student's s answer must cover the EXACT SAME range as the standard answer, NOT just any single value subset within that range;
If the standard answer contains multiple solutions connected by /'dd'',,', all of them must be listed in the student's s answer; If student's response does not mention any answer, it is considered WRONG;
You be deterministic and rigorous always declare the answer as either CORRECT or WRONG;
Small rounding differences are permitted if all the derivation steps are correct.
Your reaponse
include:
44 Short Analysis
Provide a short and evidence-backer analysis between <analysis> </analysis> tags, in which you should extract the final solution and the student's
value from the standard whether they are the same.
and judge
#4 Correctness
Based on the analysis, you should report label CORRECT WRONG between <judge> </judge> tags (e.g., <judge> judgee or <juude>wRONo<jjudee))
### User Prompt
Problem: (problem]
Standard Answer:
(standardLanswer)
Student Answer: (student.answer)
Table 3: LLM-as-a prompt template for evaluating model responses
15

---

Published as a conference paper at ICLR 2026
To ensure accurate scoring for evaluations in first validate all responses with Math-Verify Kydlfček) and double check with DeepSeek-V3 as a judge. We prompt DeepSeek- V3 for re- sponses that are labeled as wrong by math-verify. Table contains the exact prompt.
C BENCHMARK RESULTS
Here, provide detailed breakdown of all LLM performance five math benchmarks and four coding benchmarks. The Avg. column is the same as the one in Table T
Model
AIME 24
AIME 25
MATH-500
Minerva
Olympiad
Avg.
Low Benchmark Scores
R1 R1-Qwen-1.55 1.5B
30.4
21.7
84.2
47.6
53.7
47.5
R1-Llama-8B
42.9
27.1
88.3
49.0
63.5
54.1
DeepMath- 1.5B
37.5
29.2
90.1
54.8
62.6
54.8
DeepScaleR-1.5B
40.0
30.0
89.9
54.7
61.8
55.3
OpenThinker3-115B
52.1
39.6
92.2
43.7
68.4
59.2
Qwen3-1.7B
44.2
36.7
92.1
59.5
67.3
59,9
Medium Benchmark Scores
R1- R1-Qwen-7B LIMO-32B
55.4
38.3
94.3
64.3
70.8
64.6
55.8
41.7
95.4
70.5
73.0
67.3
OpenThinner337B
63.3
58.3
96.4
64.6
77.8
72.1
RI RI-Qwen-32B
67.9
52.1
95.4
69.9
76.5
72.3
High Benchmark Scores
Qwen3-8B
76.3
70.4
97.3
72.2
79.6
79.1
QwQ-32B
79.6
69.6
97.9
72.6
83.1
80.5
Qwen3-32B
78.3
71.7
97.5
75.0
82.3
81.0
Qwen3-30B-AABB
77.5
73.8
97.6
74.1
82.2
81.1
AM-Thinking-3222
80.4
77.9
98.4
72.8
83.5
82.6
Table 4: Benchmark performance (%) of 15 thinking LLMs. "Olympiad" stands for Olympiad- Bench and "Minerva" is the math subset in Minerva benchmark. "Avg" = unweighted mean of AIME 24, AIME 25, MATH-500, Minerva, and OlympiadBench.
16

---

Published as a conference paper at ICLR 2026
Model
CruxEval
HumanEval
HumanEval+
MBPP
MBPP+
Avg.
Low Benchmark Scores
RI-Qwen-L.a 1.5B
35.4
66.7
60.7
48.5
55.2
53.3
OpenThinker3-1.55
28.1
67.7
62.6
51.9
56.9
53.4
DeepScaleR-1.5B
41.5
74.2
66.3
53.1
59.0
58.8
DeepMath- 1.5B
44.0
73.6
66.2
54.1
60.8
59.8
Medium Benchmark Scores
Qwen3-1.7B
75.2
83.3
75.2
68.5
71.2
74.7
R1-Llama-8B
73.3
85.4
79.7
70.8
70.9
76.0
RI-Qwen-7B
73.8
88.3
82.3
71.7
72.4
77.7
OpenThinker-7B
83.6
84.7
75.9
80.4
75.9
80.1
High Benchmark Scores
Qwen3-8B
91.2
91.5
85.3
88.2
81.0
87.4
Qwen3-32B
94.9
95.3
89.0
84.0
75.9
87.8
Qwen3-30B-A3B
93.8
94.4
86.2
90.4
80.3
89.0
R1-Qwen-32B
92.0
96.3
88.9
90.0
81.4
89.7
AM- Mi-Thinnking-222
94.3
94.4
88.0
91.6
80.4
89.7
QwQ-32B
93.8
96.5
88.0
93.3
80.7
90.5
Table 5: Benchmark performance (%) of 14 thinking LLMs. "MBPP+" and "HumanEval+" "Avg" unweighted mean of CruxEval,
two synthesized splits from EvalPlus (Liu et al. 2023).
MBPP, MBPP-Plus, HumanEval, and HumanEyal-Phus
D RECOVERABILITY TEST
Table 6 reports breakdown of model recoverability performance shared subset across different positions (%) of the original trajectories. Table reports the results of ablation study explained in $3.2 where the first paragraph of model reasoning preserved. The subscripts in Table equal the difference between the corresponding values Table 7 and Table 6, showing the changes in recoverability induced by preserving the original opening.
Model
0%
20 %
40 %
60
80%
Avg.
Benchmark Avg.
RI-Qwen-1.5B
44.0
66.0
64.0
67.0
62.0
60.6
47.5
R1-Llama-8B
65.5
81.5
84.5
82.5
93.0
81.4
54.1
DeepMath-1.5B
71.5
94.0
90.0
94.0
90.5
88.0
54.8
DeepScaleR-115B
61.5
88.0
89.5
85.0
88.0
82.4
55.3
OpenThinker331155
89.0
95.5
96.5
98.0
97.0
95.2
59.2
Qwen3-1.7B
97.0
99.5
99.0
98.5
98.0
98.4
59.9
RI-Qwen-7B
48.5
77.0
79.0
82.5
80.5
73.5
64.6
LIMO-32B
18.0
29.0
36.0
32.5
31.0
29.3
67.3
OpenThinker3-77
81.5
87.0
89.0
84.5
86.0
85.6
72.1
RI-Qwen-32B
21.0
70.5
78.5
90.5
88.5
69.8
72.3
Qwen3-8B
71.0
88.5
89.0
91.5
89.5
85,9
79.1
QwQ-32B
53.0
79.5
86.5
88.5
91.0
79.7
80.5
Qwen3-32B
32.5
74.5
88.5
81.0
82.5
71.8
81.0
Qwen3-30B-A3B
68.0
90.5
93.5
91.5
95.5
87.8
81.1
AM--hinking-3222
16.5
29.0
36.5
41.0
44.0
33.4
82.6
Table Recoverability (shared) results (on 200 questions fully solved by all 15 LLMs eight out of eight). 0%, 20%, 40%, 60%, 80% are the positions of original reasoning where distraction is introduced. "Avg." column averages across all the positions. "Benchmark Avg." " from Table
17

---

Published as a conference paper at ICLR 2026
Model
0%
20% %
40%
60%
80%
Avg.
Benchmark Avg.
RI-Qwen-I.5B
89.0 445.0
94.0 4280
91.0 427.0
89.5 4225
84.0 .22.0
89.5 +28.9
47.5
RI-IJama-8R
95.5 +30.0
96.5 +150
97.0 +12.5
91.5 +9.0
87.0 6.0
93.5 +12.1
54.1
DeepMath-1.5B
99.0 +27.5
98.5 +4.5
98.5 48.5
98.0 +4.0
95.0 +4.5
97.8 +9
54.8
DeepScaleR-I.5B
97.0 435.5
97.5 +9.5
97.5 +8.0
98.0 +13.0
86.0 220
95.2 +12.8
55.3
OpenThinker3 1.5B
96.5 47.5
98.0 +2.5
97.0 +0.5
100.0 +2.0
96.0 -1.
97.5 +2.3
59.2
Qwen3-1.7B
100.0 +3.0
100.0 -0.5
100.0 +1.0
100.0 +1.5
82.0 16.0
96.4 2.0
$9.9
RI-Qwen-7B
91.5 +43.0
95.5 +185
+120
85.0 44.5
+17.0
64.6
LIMO-32B
+40.0
57.5 +285
+185
60.5 +28,0
+22.5
56.8 +27.5
67.3
OpenThinker3-7B
+11.5
+7.
96.0 +7,0
96.5 +12,0
85.0 -10
93.0 +7,4
72.1
R1- RI-Qwen-32B
74.5 453.5
80.5 4100
90.0 411.5
43.0
.3.5
84.7 234.
72.3
Qwen3-8B
95.5
97.0
97.5 48.5
97.0
80.0
93.4 47.5
79.1
QwQ-32B
+1.5
73.0
81.0
90.0
86.5 4.5
79.0 .07
80.5
Qwen3-32B
75.0 +42.5
87.0
95.5
92.5 +11.5
67.5 1500
+115
81.0
Qwen3-30B-A3B
83.5 415.5
88.0
91.0
94.0 +2.5
66.0
.29.5
84.5 .33
81.1
AM-Thinking-322
55.0 438.
53.0 +240
60.0 423.5
75.0 +34.0
42.5.1.5
57.1 423.7
82.6
Table Ablation Study: Recoverability (shared) results with original beginning (on 200 questions
fully solved by all 15 LLMs eight out of eight), 0%, 20%, 40%, 60%, 80% are the positions of origi- nal reasoning where distraction introduced. "Avg." averages across all the positions. "Benchmark Avg." is from Table 4
Table and Table 10 report detailed breakdown of recoverability on individual subset; the for- mer sets the length of distracting steer ytteer be 0.2 times of the reasoning trajectory by default, whereas the latter sets to 0.4 of the reasoning trajectory.
Model
0 %
20 %
40 %
60 %
80 %
Avg.
Benchmark Avg.
RI-Qwen-1.5B
24.0
40.8
40.8
38.8
48.4
38.6
47.5
RI-Llama-8B
32.0
38.4
49.2
57.6
79.8
49.6
54.1
DeepMath-1.555 1.55
54.4
61.6
61.6
64.0
67.6
61.8
54.8
DeepScaleR-1.55 115B
35.2
54.0
56.8
57.6
60.8
52.9
55.3
OpenThinker3-11.5
58.0
69.6
77.6
76.0
78.0
71.8
59.2
Qwen3-1.7B
58.4
70.4
74.4
85.2
84.4
74.6
59.9
R1-Qwen-7B
38.4
48.0
46.4
50.4
45.6
45.8
64.6
LIMO-32B
8.8
21.2
18.8
20.0
23.6
18.5
67.3
OpenThhinker3-1
63.2
72.4
76.4
77.6
82.8
74.5
72.1
RI-Qwen-32B
8.4
37.6
53.6
58.0
70.4
45.6
72.3
Qwen3-8B
51.6
64.4
73.2
76.0
78.8
68.8
79.1
QwQ-32B
50.0
54.5
64.8
68.8
74.8
62.6
80.5
Qwen3-32B
23.6
53.6
67.2
66.4
73.6
56.9
81.0
Qwen3-30B-A3B
36.8
61.6
68.8
67.6
65.2
60.0
81.1
AM-Thinking- 32B
19.6
26.8
29.6
26.4
24.0
25.3
82.6
Table 8: Recoverability (individual) results (on 200 randomly sampled questions for each of 15
LLMs). We sample questions according to the inverse proportions solve rates. 0%, 20%, 40%, 60%, 80% are the positions of original reasoning where distraction introduced. "Avg." averages across all the positions. "Benchmark Avg." is from Table
18

---

Published as a conference paper at ICLR 2026
Model
1/8
2/8
3/8
4/8
5/8
6/8
7/8
8/8
RI-Qwen-I.5B
16.1
27.7
28.6
33.8
36.5
40.0
48.1
58.9
RI-Llama-8B
30.5
23.3
39.4
51.8
56.4
47.9
65.2
70.0
DeepMath-1.555 1.50
26.2
45.6
40.1
56.8
51.3
55.5
81.1
80.7
DeepScaleR-1.5
1.5B
21.9
30.5
32.6
35.3
53,9
61.5
62.7
73.3
OpenThinker-1.3
51.4
51.2
56.8
56.5
70.0
71.9
80.5
93.0
Qwen3-1.7B
40.9
38.0
52.3
65.2
70.6
80.6
84.7
91.3
RI-Qwen-77
28.4
32.0
34.7
46.4
47.7
45.0
44.9
56.9
LIMO-32B
15.3
22.1
21.2
20.0
14.3
13.3
28.7
15.4
OpenThinkers
56.9
61.5
52.0
69.2
56.7
70.8
74.5
83.7
R R1-Qwen-32B
24.7
35.4
40.0
40.0
32.5
41.9
48.0
52.9
Qwen3-8B
45.0
34.3
49.2
55.0
46.7
63.6
71.2
77.9
QwQ-32B
35.0
49.1
25.0
33.3
55.7
58.8
59.2
69.4
Qwen3-32B
37.3
43.3
32.5
42.9
44.6
56.4
45.3
64.7
Qwen3-30B-A3B
42.9
25.0
38.7
54.5
55.0
60.0
60.0
65.9
AM-Thinking-3221
11.1
22.2
27.5
12.5
23.6
16.7
21.4
28.9
Table Recoverability (individual) results (on 200 randomly sampled questions in total for each of 15 LLMs) across different solve rates (N=8). Across all LLMs, we observe that recoverability is positively correlated with solve rates, since models need solve the original question, even after successfully backtracking from distraction.
Model
0 %
20 %
40 %
60%
Avg.
Benchmark Avg.
R1-Qwen-11. 1.5B
11.6
26.0
27.6
24.0
22.3
47.5
R1-LLama-8B
29.2
43.2
54.8
56.4
45.9
54.1
DeepMath-1.55
38.8
54.0
43.6
51.2
46.9
54.8
DeepScaleR-1.5p
24.8
50.0
53.2
50.4
44.6
55.3
OpenThinker3: 1.5B
52.4
70.8
68.8
78.8
67.7
59.2
Qwen3-1.7B
59.2
73.2
76.4
81.2
72.5
59,9
R1-Qwen-7B
25.6
41.2
39.2
36.4
35.6
64.6
LIMO-32B
6.0
10.8
16.8
17.6
12.8
67.3
OpenThinker3-77B
59.6
72.0
70.0
73.2
68.7
72.1
R R1-Qwen-32B
10.8
36.8
49.2
62.0
39.7
72.3
Qwen3-8B
50.4
67.2
71.2
76.0
66.2
79.1
QwQ-32B
44.8
52.0
61.2
68.4
56.6
80.5
Qwen3-32B
23.2
59.6
62.4
65.6
52.7
81.0
Owen3-30B-A3B
31.6
53.2
62.0
59.6
51.6
81.1
AM-Thinking-32BB
22.8
33.6
29.6
26.0
28.0
82.6
Table 10: Recoverability (individual) results with 40% of distracting reasoning. We control
length of distraction to be 40% of distracting reasoning trace (default 20% Table(6). The sampled questions the same in Table (6] 0%, 20%, 40%, 60% are the positions of original reasoning where distraction injected. "Avg." averages across all positions. "Benchmark Avg." from Table
E ADDITIONAL ABLATION STUDIES
E. DISTRACTOR DESIGN RATIONALE AND CROSS-MODEE ABLATION
Stitching procedure. When constructing the steered trajectory 1E steer]] j, truncate n at the end of the nearest sentence to preserve coherence. The distracting steer stter is then appended line, prefixed with short transition phrase ("Wai.. Let think") that mimics the self- correction style common in reasoning traces (see Figure 2). The entire steered trajectory is placed directly inside the model's assistant-side thinking (e.g., within <think> tags).
Why sample from the same model on a different question? Our recoverbbiliiyy protocol constructs distractors by sampling from the evaluated model M itself, conditioned different question o'
19

---

Published as a conference paper at ICLR 2026
This design has three advantages. First, sampling from q' guarantees that blindly com- pleting the distractor leads to an incorrect answer for the original question q. without requiring an external LLM judge to verify incorrectness. Second, the approach model-agnostic: it does not assume knowledge of which collaborator models will be encountered at deployment time. If specific collaborators are known priori, our protocol generalizes straightforwardly to sampling distractors from those models. Third, the strategy is scalable, requiring only one additional generation per test point.
22.
This design is motivated by prior findings that incomplete reasoning traces can mislead LLMs into naively continuing them rather than backtracking. Yang al. (2025b) inject four types of unhelp- ful thoughts (uninformative, irrelevant, misdirecting, incorrect) into reasoning traces and find that models tend to follow the injected reasoning. Similarly, /Zhou et al. (2024) show that irrelevant thoughts in chain-of- offthhught prompting degrade accuracy by 1.4 19.8%, %, confirming that LLMS are susceptible to off-distribution reasoning content.
Cross-model ablation. To verify that same-model sampling does not introduce systematic bias, conduct an ablation where distractors are instead sampled from five randomly selected models. As shown in Table recoverability scores remain consistent across both settings, confirming that our protocol is robust to the choice of distractor source.
Recoverability (Sh.) Different Model
Model
Same Model
RI R1-Qwen-1.5B
60.6
68.8
DeepScaleR-1.5B
82.4
80.7
RI-Llama-8B
81.4
79.9
DeepMath-1
88.0
92.7
95.2
91.7
Qwen3-1.7B
98.4
97.5
R1-Qwen-78 -B
73.5
70.0
LIMO-32B
29.3
25.2
Open Thinkhinker3-7B
85.6
85.5
1--en-322B
69.8
66.1
Qwen3-8B
85.9
85.6
QwQ-32B
79.7
77.8
Qwen3-32B
71.8
73.8
Qwen3-30B-A3B
87.8
86.7
AM-Thinking-3225
33.4
33.9
Table 11: Recoverability under different distractor sources. "Same Model": distractors sampled
from the evaluated model (corresponds to Recoverability (Sh.)" in Table [ "Different Model": distractors sampled from five other randomly selected models.
E.2 GUIDABILITY RESULTS ON ORIGINALLY SOLVABLE PROBLEMS
Section 3.2 shows that LLMs struggle to leverage partial thinking traces from a guidance model on questions beyond their capabilities (solve rate < 1/8). To rule out distribution shift between the guidance model and the evaluated model as a confounding factor, we test guidability on problems that models can fully solve independently (solve rate 8/8). As shown in Table 12 guidability reaches nearly 100% across all models on this subset. confirming that the observed limitations stem from models' inherent capabilities rather than cross-model distribution mismatch.
20

---

Published as a conference paper at ICLR 2026
Model
Guidability (on fully solvable set)
R1-Qwen-1.5BB
98.5
DeepScaleR-1.5B
98.6
RI-Llama-8B
98.4
DeepMath-1.555
98.5
Open TeinThrnne333.5B
99.7
Qwen3-1.7B
98.9
RI-Qwen-7B
99.0
LIMO-32B
98.1
Open Thinkke333
98.9
R1- R1-Qwen-32B
98.3
Qwen3-8B
98.7
QwQ-32B
98.8
Qwen3-32B
98.7
Qwen3-30B-A3B
98.4
AM-Thinking-322B
98.1
Table 12: Guidability originally solvable problems. Results for problems that models solve
independently (solve rate 8/8).
E.3 CODING GUIDABILITY
Table 17 reports the detailed breakdown of coding guidability (individual) results across seven mod- els and four steer proportions. The coding guidability scores are substantially higher than their math counterparts (Table( [4). To assess whether models are genuinely continuing reasoning from the steer or simply benefiting from information already present it, we conduct an answer-forcing test: we append closing </think> tag immediately after the steer, forcing the model to produce swer without any additional reasoning. The resulting pass rates are reported as subscripts. At higher steer proportions, the answer-forcing rate meets exceeds the guidability score for most models, indicating that models cannot genuinely continue reasoning from the steer and in some cases ac- tively hurt performance by overriding correct information already present. For DeepScaleR-1.5B and DeepMath-1 1,5B, answer-forcing matches guidability across all ratios.
F GUIDABILITY TEST
Table 13 reports the number of unique problems and guiding trajectories used per guiding model (sub-column) for each LLM (row). Table 14 reports guidability (individual) results for different length of the guiding steers measured by x% of the trajectories. Similarly, Table I5 reports break- down of guidability on shared subset. Table groups guidability (individual) scores by the guiding models (column) for each LLM (row)
21

---

Published as a conference paper at ICLR 2026
# of Problems
# of Trajectories
DeepSeek-R
Qwen-3
QwQ-32B
DeepSeek-RI
Qwen-3
QwQ-32B
DeepMath-1.5B
152
198
302
231
268
302
DeepScaleR-1.5B
154
196
311
234
269
311
LIMO- Mi0-Qwen-32BB
100
137
185
142
172
185
Open Teenhenker3-1151
151
199
270
236
278
270
Open Thinkernke33
101
146
163
146
186
163
Qwen3-1.7B
130
175
245
192
233
245
RI-Llama-8B
151
196
266
229
269
266
R1- 11-Qwen-1.5B
168
213
363
261
363
R R1- 11-Qwen-7B
107
156
190
151
195
190
RI R1-Qwen-32B
94
145
162
134
182
162
Table 13: Guidability statisties: unique number of problems and trajectories per guiding model
(column) for different student models (row) for Guidability (individual) test.
Model
20% %
40%
60 %
80 %
Avg
Benchmark Avg.
RI-Qwen-1.5B
14.67.7
23.1
33,231.3
43.046.2
28 28.425.6
47.5
RI-Llama-8B
20.85.4
29.615.7
40.027.6
49.7348
35.021.8
54.1
DeepMath-1155B
13.67.2
21.116.2
31.227.5
42.340.6
27.122.9
54.8
DeepScaleRR11.5 1.5B
15.77.5
23.215.7
34.628.1
45.6a18
29.823.3
55.3
OpenThinker3-
18.1
30.6214
36.1
46.042..
32.726.9
59.2
Qwen3-1.7B
18.25.
23.711.8
34.820.6
42.833.
29.918.0
59.9
RI-Qwen-7B
10.83.5
16.2aa
22.013.1
29.925.4
19.712.1
64.6
LIMO-32B
12.62.6
18.84.
24.411.6
30.021.8
21.5
67.3
Open'Thinker3-7B
11.16.5
20.0jo.1
22.615.4
28.723.4
20.613.8
72.1
RI-Qwen-32B
14.23.8
19.76.
24.912.4
31.222.6
22.511.2
72.3
Table 14: Guidability (individual) results (on all questions with solve rate S 1/ for each individual
model). 20%, 40%, 60%, 80% are proportion teacher reasoning revealed to the student model in its thinking window. The subseript value is the percentage of cases where teachers have derived the solution. "Avg" is the average across different proportions. "Benchmark Avg" is the same as in Table 4
Model
20%
%
60%
80%
Avg
Benchmark Avg.
R1-Qwen-1.5B
1.2
0.9
4.1
5.8
3.0
47.5
R1-Llama-8B
5.2
5.8
10.4
13.3
8.7
54.1
DeepMath-1.5BB
0.9
0.9
4.6
7.2
3.4
54.8
DeepScaleR-I.55
1.2
0.9
5.2
9.0
4.1
55.3
OpenThinke33-115B
1.7
5.5
7.0
8.4
5.7
59.2
Qwen3-1.7B
2.3
3.2
7.8
11.0
6.1
59.9
R1-Qwen-7B
2.6
5.2
6.4
9.9
6.0
64.6
LIMO-32B
4.9
10.1
12.8
8.8
67.3
OpenThinker3-77B
4.9
9.0
9.6
12.8
9.1
72.1
R1 R1-Qwen-32B
4.1
7.5
11.0
14.2
9.2
72.3
Table 15: Guidability (shared) results (on questions with solve rate < across all ten models).
20%, 40%, 60%, 80% are proportion of teacher reasoning revealed to the student model in its think- ing window. "Avg" is the average across different proportions. "Benchmark Avg" is the same in Table
22

---

Published as a conference paper at ICLR 2026
Model
DeepSeek-RI
QwQ-32B
Qwen3-235BBA2228
Benchmark Avg.
RI-Qwen-1.5T
28.2
30.4
26.2
47.5
DeepMath- 1.5B
29.0
26.2
26.3
54.8
DeepScalcR-1.55
30.9
31.1
27.3
55,3
RI RI-Llama-8B
37.8
34,4
33.2
54.1
Qwen3-1.7B
33.4
31.1
25.6
59.9
OpenThiiner3-115B
35.7
30.6
32.3
59.2
RI-QQen-7BB
22.0
19.6
18.7
64.6
LIMO-32B
24.5
24.6
15.7
67.3
RI RI-Qwen-32B
23.5
23.0
21.9
72.3
OpenThinker3-7B
22.9
21.4
18.0
77.8
Table 16: Guidability (individual) results (teacher model comparison). Each teacher model aver- ages across Guidability (individual) scores for all proportions, 20%, 40%, 60%, 80%, in Table 14
Model
20%
40%
60 %
80%
Avg
Benchmark Avg.
RI-Qwen-1.5B
19.6 19.618.0
32.829.5
47.045.0
60.866.0
40.1 30.1
53.3
RI-Llama-8B
27.4 27.417.5
46.230.7
63.454.1
80.575.1
54.444.3
76.0
DeepMath-II555 1.5B
17.4
30.227.0
42.041.9
55.360.0
36.235.9
59.8
DeepScaleR- 1.5B
16.916.0
28.827.7
42.742.2
60.760.8
37.336.7
58.8
OpenThinker3-11. 1..5
29.410.4
43.223.0
54.940.6
70.266.6
49.435.1
53.4
Qwen3-1.7B
26.916.8
48.934.0
67.358.4
82.677.
56.446.7
74.7
R1-Qwen-7B
27.018.9
47.1
61.855.1
76.475.a
53.146.2
77.7
Table 17: Coding Guidability (individual) results (on all coding questions with solve rate for each individual model). 20%, 40%, 60%, 80% are proportion of teacher reasoning revealed to the student model. The subscript value is the answer-forcing pass rate: the percentage of cases where the model produces the correct answer when forced to skip reasoning after receiving the steer E.3 "Avg" averages across proportions. "Benchmark Avg" is from Table
G ADDITIONAL CONTROL STUDY
Supervised Fine- Fine-Tuning Hyperparameters. We perform full fine-tuning on Qwen2. 5-1. 5B and Qwen2. 5B base models for epochs. The max tokens is set 16K, batch size 64, learning rate 2e-5, warmup ratio 0.1, max gradient norm 1.0, weight decay 0.01.
Ablation Study. We compare the effects of distillation teachers on Owen2 5-7B models. We observe similar patterns as discussed in $4.1 where AM-Distill models achieve worse recoverabil- ity compared to QwQ--Qwen3-Distill models. The guidability scores are not measured since the benchmark performance are too high to collect sufficient qualified problems.
AM-Distill 7B
QwQ-Distill 7B
Qwen3-Distill 7B
0.8
0.25
0.7
Recoverability
0.20
MATH500
0.15
0.6
0.10
0.05
0.4
0.00
100
200
300
400
500
100
200
300
400
500
Training Step
Training Step
Figure 8: Qwen2.5 7B models distilled from AM (Thinking-vl) 32B also shows lower recoverability than those distilled from QwQ 32B or Qwen 32B, while having similar benchmark performance; the gap is significant for all steps (p < 0.005). Stars mark each model's peak over training steps.
23


Reviews:
[{"review_id": "hY9QTkjhI1", "reviewer": "Reviewer_Y2jS", "summary": "The paper proposes two synthetic tests to assess how well LLMs can collaborate on a shared reasoning trajectory, which the authors refer to as off-trajectory reasoning. The tests are as follows:\n\n- Recoverability: tests whether models can backtrack from incorrect reasoning to arrive at the correct answer. This is done by adding a distractor in the middle of a reasoning trace.\n  - A distractor is constructed by taking a reasoning path from a *different question* and truncating it to 20% of the target reasoning length.\n  - Recoverability tests are performed on questions that the model can already answer correctly.\n- Guidability: tests whether the model can successfully use the guidance of a larger guide model to arrive at the correct answer. This is done by adding a portion of the teacher's reasoning at the beginning of the reasoning trace. The length is selected between 20% - 80% the guide's reasoning length.\n  - Guidability tests are performed on questions where the model's solve rate is less than 2/8.\n\nThe authors perform these tests on a wide variety of models and show that the test metrics are largely orthogonal to the original *solo* reasoning capability of the models. They also perform various ablation on test parameters, and perform control studies on the post-training recipes of the models under evaluation, i.e., SFT distillation and RL.", "questions": "Regarding W1, I'm not sure if we necessarily want or need models to perform well on this test for them to collaborate well with other models. I do not believe the test scores will align with the model's ability when applying realistic inference-time or training-based collaboration methods.\n\nI'm willing to raise my score if the authors can provide a strong argument against this concern.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novel approach: the authors acutely identify recoverability and guidability as two key pre-requisite abilities for LLM collaboration, and target these aspects through novel probing schemes.\n- Simplicity: the two test methodologies are simple and easy to implement\n- Novel findings: The main finding is counter-intuitive and novel.\n- Actionable insights: The authors present clear, actionable insights from the control studies.", "weaknesses": "- W1. Contrived test protocol: the test protocol involves truncating reasoning traces and inserting these partial traces at arbitrary token locations. Also, as I understand, this happens within the internal thinking traces of the models within the, e.g., <think> tags, within the *assistant* message.\n  - This introduces misalignment with the training distribution, which is unrelated to the model abilities which the authors intend to evaluate. Firstly, truncating and inserting at arbitrary token indices can lead to incoherent sentences. Also, in standard SFT pipelines, the training data for the *assistant* message portion of the samples are meant to be outputs from the model itself. Furthermore, in RL, the *thinking* part of the assistant message is only trained on outputs generated by the model itself. The proposed test protocol introduces contexts that are significantly misaligned with this training distribution. I.e., placing the external reasoning in the thinking part of the assistant message indicates to the model, perhaps deceivingly, that it originated from the model itself, rather than some external model. The input distribution shift due to the *protocol* will likely cause significant performance degradation, unrelated to the *semantic* distribution shift of distractor reasoning or guidance from an external collaborator LLMs, on which the author *intend* to evaluate the models.\n  - While some amount of input distribution shift is inevitable in any inference-time probing scheme, I believe the contrived nature of the  specific approach proposed by the authors may cause excessive distribution shift, which would be avoided in potential real-world methods used for LLM collaboration. For example, it may be possible to include external distractor or guidance reasoning as a user message, indicating the that the reasoning does not originate from the current model. The model will then be inclined to identify potential errors or hints and act accordingly.\n- W2. Limited task domain: The study is limited to a single domain, math reasoning. Therefore, the findings may not generalize to other domains such as coding, creative writing, etc., where distractions and guidance may have different characteristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGnB7q4E2S", "reviewer": "Reviewer_BaPD", "summary": "This paper investigates whether reasoning LLMs trained for solo reasoning can effectively collaborate by working on shared reasoning trajectories—a capability the authors term \"off-trajectory reasoning.\" The authors propose twin evaluation tests: (1) **Recoverability**, which measures whether LLMs can backtrack from distracting reasoning traces, and (2) **Guidability**, which tests whether they can build upon correct partial reasoning from stronger models to solve problems beyond their inherent capabilities. The study evaluates 15 open-weight LLMs (1.5B–32B parameters) across five math benchmarks (AIME2024/2025, MATH-500, Minerva, OlympiadBench).", "questions": "- How do results change if steers are from different model families rather than the same model?\n- Can you show cases where models successfully integrate guidance? What makes them different?\n- Does synthetic data generation for off-trajectory scenarios help?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- First systematic study examining whether solo-reasoning training yields collaboration-ready models\n- Comprehensive evaluation across 15 open-weight LLMs from 4 distinct families\n- Clear problem motivation connecting to practical collaboration scenarios (efficiency, exploration, safety)\n- Provides actionable insights for training better collaborative reasoners", "weaknesses": "- All experiments confined to mathematical reasoning benchmarks\n- The assumption that reasoning for question q' inserted into question q constitutes a \"strong distractor\" is not validated\n- No analysis of what specific data characteristics lead to poor/variable recoverability\n- No discussion of computational costs of multi-model collaboration vs. solo reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TMIIvfmrSp", "reviewer": "Reviewer_zPNH", "summary": "This paper systematically investigates the capabilities of large language models in multi-model collaborative reasoning, introducing the novel paradigm of \"off-trajectory reasoning.\" The authors design a dual-test framework of Recoverability and Guidability, revealing that current models trained for solo reasoning exhibit fragility in collaborative scenarios - stronger benchmark performers are more susceptible to interference, and all models struggle to effectively leverage external correct reasoning traces. This work exposes limitations in conventional training paradigms and provides both evaluation tools and improvement directions for building genuinely collaborative reasoning systems.", "questions": "1. When constructing distracting reasoning traces, did the authors consider incorporating statements with logical fallacies as distractors, in addition to sampling from different questions? This would help evaluate the models' ability to discern and reject fundamentally unsound reasoning.\n2. There is a spelling error in the comments below Table 1 on page 5. The word \"bechmark\" in \"the best bechmark model\" should be corrected to \"benchmark\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Innovative methods: The paper's innovative dual-test framework of Recoverability and Guidability provides the first systematic tools for evaluating LLMs in collaborative reasoning. This novel and practical methodology effectively uncovers model deficiencies that remain hidden in traditional solo-reasoning benchmarks.\n2. Rigorous experiments: Through extensive experiments across multiple models and tasks, the paper presents a counterintuitive finding: strong solo-reasoning models do not equate to strong collaborators. This reveals an orthogonal relationship between benchmark performance and collaborative robustness, posing a significant challenge to the prevailing training paradigm that over-optimizes for leaderboard rankings.\n3. Controlled variable experiments: Through controlled variable experiments, the study clearly demonstrates how teacher model selection, RL training, and data filtering strategies directly impact collaborative reasoning capabilities. These findings provide concrete guidance for industrial practitioners to optimize model training.", "weaknesses": "1. Limitations of the experimental scope: The paper's experiments are confined to mathematical reasoning tasks, limiting the generalizability of its findings to other collaborative domains like coding or scientific inquiry. Furthermore, the evaluation relies on simulated two-model interactions rather than testing in real multi-agent environments, leaving its practical efficacy unverified.\n2. The analysis of model defects is relatively shallow: While the study successfully identifies failures in collaborative reasoning, it does not delve into the underlying causes. It remains unclear whether models fail to \"recognize the relevance of guidance\" or \"cannot integrate it into their own reasoning process,\" and the analysis lacks mechanistic investigations (e.g., attention weights or intermediate step tracing) to pinpoint these cognitive bottlenecks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3wT4lHOFIi", "reviewer": "Reviewer_BJqx", "summary": "1. It investigates whether large language models (LLMs) trained for solo reasoning—i.e., generating complete reasoning chains independently—are capable of effective collaborative reasoning, where they must build on or recover from partial reasoning traces produced by other agents (e.g., other models or humans).\n2. Stronger solo-reasoning models are often more fragile under distraction—high-benchmark models like AM-Thinking-32B show poor recoverability.\n3. Reinforcement Learning (RL) after supervised fine-tuning (SFT) significantly improves both recoverability and guidability.", "questions": "1. Since the models used in exp are trained with different training recipe, could you use open-sourced datasets and the numbers of training dataset to study the Recoverability and Guidability at the SFT-stage? \n2. The models listed in exp are almost trained with sft-loss, could you use open-sourced datasets and RL-training to study the impact of RL for Recoverability and Guidability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work is among the first to systematically probe LLMs’ robustness and adaptability in multi-agent reasoning contexts.\n2. The twin-test framework—Recoverability and Guidability—is conceptually original. It cleanly isolates two extremes of collaborative interaction (resisting distraction vs. leveraging guidance) and operationalizes them in a reproducible, controlled manner.\n3. The stronger solo reasoners are more fragile under distraction, and no model effectively uses correct guidance to exceed its solo capability.", "weaknesses": "1. It simulates collaboration via a static, one-shot injection of an off-trajectory steer (either distracting or guiding), followed by continuation from the main model. This simplification overlooks more realistic collaborative dynamics—e.g., iterative back-and-forth exchanges, multi-turn corrections, or mixed-quality steers (partially correct, partially wrong).\n2. All experiments are conducted on mathematical reasoning benchmarks. While math is a strong testbed for structured reasoning, it may not generalize to other domains (e.g., commonsense, legal, or scientific reasoning) where reasoning trajectories are less formalized or more ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)