System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
Authors: Xiaoqiang Wang
Suyuchen Wang
Yun Zhu
Bang Liu
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Reasoning Model
Latent-space Reasoning
Implicit Chain-of-Thought
Dynamic Early Exit
Conditional Computation
Primary area: deep_learning
TL;DR: We propose an adaptive latent reasoning framework that dynamically combines fast and slow reasoning paths in latent space.

Abstract: Chain-of-thought (CoT) reasoning enables large language models (LLMs) to move beyond fast System-1 responses and engage in deliberative System-2 reasoning. However, this comes at the cost of significant inefficiency due to verbose intermediate output. Recent latent-space reasoning methods improve efficiency by operating on hidden states without decoding into language, yet they treat all steps uniformly, failing to distinguish critical deductions from auxiliary steps and resulting in suboptimal use of computational resources. In this paper, we propose System-1.5 Reasoning, an adaptive reasoning framework that dynamically allocates computation across reasoning steps through shortcut paths in latent space.Specifically, System-1.5 Reasoning introduces two types of dynamic shortcuts. The model depth shortcut (DS) adaptively reasons along the vertical depth by early exiting non-critical tokens through lightweight adapter branches, while allowing critical tokens to continue through deeper Transformer layers. The step shortcut (SS) reuses hidden states across the decoding steps to skip trivial steps and reason horizontally in latent space. Training System-1.5 Reasoning involves a two-stage self-distillation process: first distilling natural language CoT into latent-space continuous thought, and then distilling full-path System-2 latent reasoning into adaptive shortcut paths (System-1.5 Reasoning).Experiments on reasoning tasks demonstrate the superior performance of our method.
For example, on GSM8K, System-1.5 Reasoning achieves reasoning performance comparable to traditional CoT fine-tuning methods while accelerating inference by over 20× and reducing token generation by 91.0\% on average.

System-1.5 1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcuts
Xiaoqiang Wang
Yun Zhu Bang Liu Liu 12.3 'DIRO & Institut Courtois, Université de Montréal Mila Quebec AI Institute; 'Canada CIFAR AI Chair (xiaoqiang.. wang, suyuchen. wang, bang. liu @umontreal ca. gabrielzhuyunggmal.. com
Suyuchen Wang 1.2
Abstract
Chain-of-thought (CoT)) reasoning enables large language models (LLMs) to move beyond fast System-1 responses and engage in deliberative System-2 reasoning. However, this comes at the cost of significant inefficiency due to verbose inter- mediate output. Recent latent-space reasoning methods improve efficiency by operating on hidden states without decoding into language, yet they treat all steps uniformly, failing distinguish critical deductionn from auxiliary steps and re- sulting in suboptimal use of computational resources. In this paper, we propose System-1.5 Reasoning, an adaptive reasoning framework that dynamically al- locates computation across reasoning steps through shortcut paths in latent space. Specifically, System-1.5 Reasoning introduces two types of dynamic shortcuts. The model depth shortcut (DS) adaptively reasons along the vertical depth by early exiting non-critical tokens through lightweight adapter branches, while allowing critical tokens continue through deeper Transformer layers. The step shorteut (SS) reuses hidden states across the decoding steps to skip trivial steps and rea- horizontally in latent space. Training System-1.. 1.5 Reasoning involves two-stage self-distill oif-distillation process: first distilling natural language Co'I' into latent- space continuous thought, and then distilling full-path System-2 latent reasoning into adaptive shortcut paths (System-1- 1.5 Reasoning). Experiments on reason- ing tasks demonstrate the superior performance of our method. For example, on GSMSK, System-1.3 115 Reasoning achieves reasoning performance comparable traditional CoT fine-tuning methods while accelerating inference by over 20x and reducing token generation by 91.0% on average.
Introduction
Foundational large language models (LLMs) (Ouyang et al., [0222] Team et al., 2023 Dubey et al.
2024 Hurst et al. [2024 Meta 2025) has revolutionized natural language processing (Liang et al.
2022
Srivastava et al. 2023, Wang et al. 2024) and demonstrated strong potential in diverse agent
applications (Liu et al. 2025), automating real-worle tasks across both digital (OpenAl 2022 Wang
et al. 2023 Hong et al. [2023] Wang & Liu 2024 Qin et al., 2025 Wang et al. 2025) and physical
environments (Brohan et al.| 20033 Driess et al. 2023 Zheng et al. 2024 Team et al. 2024). Recent
advances in test-time scaling paradigms (Snell al.) 2024 Zhang et al. 2025) and the emergence of reasoning-orienter LLMs, also referred to as Large Reasoning Models (LRMs) (Jachh et al. 2024
Guo et al. 2025 Team 2025 Abdin et al. 2025), have leveraged Chain-of- -houghh (CoT) (Wei
et al. 2022 to extend LLM reasoning capabilities (Chen et al. 20025] Wu et al. (025). This approach
facilitates transition from fast and heuristic-driver System- reasoning to slower and deliberate
!Coressooning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Step3
Stept
Step2
Step3
Anower
Quaittion
Question
Stept
Step?
Step3
Question
Step1
(a) System-1.5 Reasoning (Ours)
(b) Chainn-ofTThogghe (CoT)
Early Exit
Answer
Stop1
Step2
Step3
Arswer
Hidden States
Continuous Thought
(Hidden States of Last Layer)
Skipped States
Question
Clusstoon
Step1
Step2
Stepd
(d) Compressed Latent Reasoning
(e) Extra Latent Reasoning
(CoT) reasoning. (c) early-exit (Elbayad et al. 2219)] Elhoushi et al. 2024), (d) compressed latent-space reasoning (e.g., Coconut (Hao et al. 2024) and CCoT (Cheng & Van Durme 2024)), and (e) extra latent reasoning approaches that delay output (e.g., pause token (Goyal et al. (023) and filler token (Pfau et al., (2024). System-1. Reasoning enables flexible latent reasoning along both the horizontal and vertical dimensions through dynamic shorteuts.
Figure Comparison of (a) our proposed System-1.3 1.5 Reasoning, (b)
System-2 reasoning (Li et al. 2025b), and has significantly improved performance in competitive mathematics (Hendrycks et 2021 Li al. 2024) and code generation (Jimenez al. 2023 Jain
et al. 2024).
However, CoT reasoning incurs substantial computational costs during inference due to the need to
generate long reasoning chains before producing the final answer. also suffers from overthinking phenomena (Chen et al. 2024 Team et al. 2025), resulting in redundant and verbose outputs for simple problems, where excessive reasoning leads to unnecessary token generation and escalates inefficiency.
To address it, one line work focuses on language-space efficiency, such as imposing token budgets during prompt input (Xu et al.) 2025a) or decoding interventions (Muennighoff et al., (0255) to constrain the length or complexity (Lee et al. 2025) of generated explanations. Another direction
emphasizes efficient decoding, such as speculative decoding (Leviathan et al., (023) and optimized
sampling (Sun et al. 2024b
Fu et al. 2024 Li al. 2025a).
However, many tokens generated during chain-of-thought (CoT) reasoning primarily serve textual fluency and coherence rather than contributing meaningfully actual reasoning advancement (Song et al. 2025 Luo et al. (023a). Consequently, recent approaches have explored operating reasoning latent space. Methods such as the pause token (Goyal et al., 2023) and filler token (Pfau al.. 2024) insert special tokens and ignore their language-space outputs to allow for additional latent computation, while Coconut (Hao et al. 2024) and CCoT (Cheng Van Durme 2024) compress explicit CoT into hidden states and feed these hidden states, rather than generated tokens, back into the model as subsequent embeddings. By bypassing language constraints, these internal computations enable more compact and flexible reasoning within the latent space.
While these methods improve efficiency, as illustrated in Figure they introduce unidirectional optimization bias, promoting either universally fast reasoning through compressed latent-space reasoning (e.g., Coconut and CCoT) for all reasoning steps or universally slow reasoning that delays outputs at every step (e.g., pause token and filler token). This uniform treatment fails to differentiate between critical deductions and auxiliary steps, leading to inefficient allocation where trivial and complex steps receive nearly equal computational budgets. For example, CoT sequence often includes distinct phases such as problem restatement, exploration, and result verification, each requiring different levels of reasoning effort (Luo et al.) 2025b).

---

This raises a natural question: Can we dynamically tailor computation across different reasoning steps to maximize efficiency while preserving performance? Ideally, models should reason quickly (System-1) 1) on non-critical steps and more carefully (System-2) on critical ones.
Motivated by this, we propose System-1.5 Reasoning, which adaptively combines fast and slow reasoning paths in latent space to handle varying step complexities in the language space. System-1.5 Reasoning introduces model depth shortcut (DS) and decoding step shortcut (SS) to adaptively allocate computation along the vertical and horizontal paths in latent space.
Specifically, the depth shorteut is implemented by augmenting each Transformer layer with router- adapter module, which dynamically determines whether a token continues through deeper standard layers or exits early via lightweight adapter branch. This depth shorteut mechanism enables flexible vertical computation along the model depth, allowing non-critical reasoning steps to be processed with fewer layers while critical steps continue deeper into the model. In addition, the step shortcut enables latent-space skipping across the horizontal dimension of decoding length, where early-exited hidden states at given layer are directly copied the next decoding step, instead of restarting latent computation from the first layer as in standard Transformers.
By supporting adaptive reasoning along both vertical and horizontal paths, System-1.3 15 Reasoning maximizes flexibility in latent-space reasoning and better mirrors human thinking, where difficult reasoning steps are handled with deliberate, reflective System 2 thinking, simple steps are processed quickly through heuristic System-1 thinking, and trivial steps are naturally skipped without thinking.
To enable latent-space shortcut reasoning, we train System-1.5 5 Reasoning via two-stage distil- lation process. The first stage, language-to- distillation, aligns the latent-space behavior of a student model with the language-spane CoT reasoning of a teacher model. This process leverages full teacher-forcing and allows efficient parallel training, avoiding the step-wise scheduling complexities of curriculum learning approaches such as iCoT (Deng et al. 2024) and Coconut (Hao et al. 2024).
The second stage, System-2 System- distillation, further compresses full-depth reasoning trajectories into shortcut execution path by leveraging reasoning criticality estimation in the language space. Specifically, we freeze the original transformer parameters and tune only the router-adapter module using an early-exit loss that encourages non-critical tokens to exit at earlier layers and critical tokens to proceed to deeper layers, while maintaining consistency between the hidden states of the full path and those of the shortcut-exited path.
We validate our approach on challenging reasoning datasets such as GSM8K (Cobbe al., 2021) Experiments demonstrate that System-1.5 5 Reasoning achieves reasoning performance comparable traditional CoT fine-tuning methods while accelerating inference by over x and reducing token generation by 91.0% on average. These results highlight the promise of System-1.5 Reasoning in improving the efficiency and scalability of the LLM reasoning.
2 1y5 Reasoning
We frame System-1.5 Reasoning as adaptive and dynamic latent-space reasoning guided by criticality analysis of language-spar space reasoning. As shown in Figure the core idea is to dynamically allocate computation in latent space through two types of dynamic shortcuts, model depth shortcut (DS) and decoding step shortcut (SS), to handle varying step complexities in the language space. Specifically, by inserting standard router-adapter module into each vanilla Transformer layer, the dynamic depth shortcut enables simple steps to be processed through shallow layers, while complex steps are routed through deeper layers for more extensive computation. In parallel, the dynamic step shortcut allows trivial steps to be skipped by copying hidden states at early exit points and directly reusing them as the hidden states for the next decoding step at the same layer.
To train System-1.. 1.5 Reasoning effectively in latent space, we employ a two-stage distillation process. First, we perform language-to-laten distillation by fine-tuning vanilla Transformer layers, aligning the student model's hidden states, which reason in latent space with those of the CoT-trained teacher model. Then, with the vanilla Transformer parameters frozen, we perform System-2 to System- 1.5 distillation by training the router-adapter modules. Specifically, we leverage atomic thought decomposition (Teng et al. 2025) to estimate reasoning criticality step by step. We decompose the CoT into directed acyclic graph (DAG) of self-contained subquestions. In this graph, independent nodes are labeled as non-critical, while derived nodes that require logical integration are labeled as
3

---

Language-to-Latent Alignment
Shortcut Learning
System-2 Teacher M Fiacher
System-2 Student M Ssuueent
System-1.5 Model M Pystemm-15
LM Head
consistency
LM Head
carly-exdt Ssystem-15
LM Head
student
Large Language Mode reacher
Large Language Mode auddent
Router 11stem
Transformer Layer audeett
Embedding
Embedding
Psyetrrs-ss
Transformer Layer Pander
Hidden States Vanilla Transforme
Question
Ei(Aec) Adapter output produced via the Depth Shortcut (DS)
Reasoning Steps
Copr (Pez) from step t + via the Step Shortcut (55)
Embedding
Answer
Afric traisin Composed hidden state System-1.5
(as defined Eq.
Figure 2: System-1.3 1.reemm1.... 1.5 Reasoning is trained through a two-stage distillation process: (1) Language- to-latent alignment, where the model learns to reason in latent space by minimizing a consistency loss between a language-space reasoning model (System-2 teacher) and a latent-space reasoning model (System-2 student); and (2) Shortcut learning, where dynamic shortcuts are trained by applying an early-exit loss that encourages non-critical steps to exit earlier and critical steps to proceed deeper. The model is simplified to a 2-layer Transformer for illustration purposes.
critical. Based on this, we apply an early-exit loss that encourages tokens of non-critical steps to exit earlier and tokens of critical steps to proceed to deeper layers.
2.1 Dynamic Shortcut Architecture
Formally, given an input text sequence X X= (....... ,t. "...... TT), we denote the vanilla hidden states at the 1-th layer as H₁ = (h4,1. h,,,, h where T is the sequence length and I E {0, 1.. L} with L being the total number of Transformer layers. The initial hidden states are given by Hn = Embed(X), corresponding to the embedding layer. For I 2 1, denoting fi(.) as the operation of a single Transformer layer, the hidden state layer I updated as H, = (H;
Depth shortcut. To adaptively determine how many of the initial Transformer layers are needed for each token, i.e., when to trigger early exit, we employ lightweight router-adapter module, as shown in Figure At each Transformer layer I, the router module R, dynamically decides whether a token should continue through the standard Transformer layer ft for deeper processing, or exit early through an adapter branch gt. Formally, during training, the output at layer I for step is expressed as a weighted combination of the adapter and Transformer outputs, where the weight is determined by the binary router R₁, implemented as a feed-forward network (FFN) layer followed by a sigmoid activation:
h
system 1.5 ds training [,t
1 w + f((h)11.) (1 w)
(1)
(1(h1-111)
During inference, the output of the router Ri serves as a confidence score for early exit, which is compared against a predefined depth exit threshold Adepth to determine whether to halt computation at the current layer for the given decoding step.
h system-l 1.5 S-dd-infereence 1,t
[9-- (hu-1()) (i((h)
if ((i otherwise.
> Adepth.
Step shorteut. The step shortcut is motivated by the observation that, in standard decoding, each step still forces the model to process from the first layer. In other words, while depth shortcuts allow adaptive reasoning along the vertical path in latent space, enabling exits at intermediate layers rather than always reaching the final layer, the model must still sequentially process every decoding step. To address this, we introduce dynamic shortcuts along the decoding steps, allowing the model

---

to udaptively skip steps and reason horizontally in latent space. By supporting adaptive reasoning along both vertical and horizontal paths, this design maximizes flexibility in latent-space reasoning and enables System-1. Reasoning to better mirror human thinking, where trivial steps are often skipped (step shortcut), difficult steps are handled with deliberate and reflective reasoning, and simple steps are processed quickly through heuristic reasoning (depth shortcut). More importantly, since System-1.5 Reasoning operates entirely in latent space, where new decoding steps rely solely on the hidden states from the previous step without introducing new input language tokens, the practice of step shortcut avoids the problem of partial observation of the input context (i.e., skipping token input at the current step) in token compression methods. This allows System-1.5 Reasoning to achieve high computational efficiency while preserving all crucial information and thinking steps.
Formally, similar to the depth shorteut, during training, the output at layer 1 and step t is computed as a weighted combination of the hidden state from step 1-1 at the same layer (note that the step shortcut is not applied when / = 0) and the hidden state at step & from the vanilla Transformer:
system- 15-ss-tminnngg (hb,,,(1 w + f((h)-1))) * (1 w) hi
where the weight w is determined by the router R2 as Eq. 2 By merging Eq. I and Eq.
4
we obtain:
h hi,
system-l S-traaning
91 91
gi (11.--1)) w + fi +(1 w)
depit shoncut
step shartcut
vanilla path
During inference, as determined by Eq. if a decoding step halts computation at intermediate layer, the hidden state at that layer is directly passed to the next decoding step within the same layer to continue reasoning. If computation halts at the final layer (i.e., after traversing all the Transformer layers via dynamic shortcuts, termed as a reasoning cycle), we either (1) feed the final hidden state back for the next reasoning cycle (2) generate the final output. Borrowing the latent reasoning setting from (Hao et al.) 2024), we apply a fixed number of latent reasoning steps, denoted as the decoding step constant Astep- By combining depth exit threshold Adepth in Eq. and decoding step constant Aste to control computation along both the model depth and decoding steps, this approach provides a flexible computational budget for test-time scaling (see Section 3.2
2.2 Two-Stage Distillation: Language-to-Latee Alignment and Shortcut Learning
Training model to reason in latent space with shorteut paths poses two challenges: the latent property and the adaptive property. (1) Vanilla Transformer layers in pre-trained LLMs are optimized for next-token prediction in the language space and are not inherently capable of reasoning in latent space. How can we align latent-space reasoning with language-space CoT? (2) Adaptive reasoning needs the model to dynamically decide varying reasoning paths in the latent space. How we potentially leverage the characteristics of language-space reasoning steps (e.g. reasoning step criticality) to guide the model toward learning adaptive reasoning via shortcut paths?
As shown in Figure 2 for the latent property challenge, we employ language-to-latent alignment through hidden-state distillation. Specifically, we align the reasoning processes between language- space teacher model and a latent-space student model. The teacher model, denoted as Me is trained with standard CoT fine-tuning to generate both intermediate steps and final answers in the language space. The student model, denoted as Mo learns to reason directly in latent space.
For the adaptive property challenge involving dynamic shortcut decisions, we leverage atom-of- thought (Teng et al. (025)) to decompose the original CoT into a directed acyelic graph (DAG) of self-contained subquestions. Independent nodes are identified as non-critical steps (requiring less computation), while derived nodes requiring logical integration are identified as critical steps (requiring deeper reasoning). Based on this, we introduce shorteut learning: We first initialize the System-1.5 1.5 model, denoted as Moqumass by inheriting the parameters from Mo We then freeze the Transformer parameters and insert adapter-router module into each Transformer layer to enable fine-tuning for adaptive reasoning. Lastly, we apply an early-exit loss that encourages non-critical steps to exit at shallower layers while allowing critical steps proceed deeper into the model.
Language-to- alignment. During training, we extract the last- hidden states from the teacher model, apply aoo-gradient,, and use them ground- ttutt features for the student model. These
5

---

features are then fed into the student model together with the question input, enabling teacher-forcing and ensuring training efficiency.
Formally, given an input sequence X = (x),... RT), we further represent it as the concatenation of three segments, the question Q, intermediate reasoning steps R. and the answer A, denoted as X = (Q R: A) (Ul:....TTT:"" "I,... .T C1,...,TT, @T,), where To, TR, and TA correspond to the lengths of each segment.
The last-layer hidden states from the System-2 teacher and student models, denoted by teacher and respectively, are aligned by minimizing the following mean squared error (MSE) loss:
I,,
L,t
To+In+TA
counsistency
Σ
(SE(sg(( uncherr hstudeen
(6)
TA 1
where sg[-] denotes the stop-gradient operation applied to the teacher model.
In parallel, the System-2 teacher model is supervised by the standard negative log-likelihood (NLL)
loss over both intermediate reasoning steps and final answers, while the System-2 student model
is supervised by NLL loss over final answer generation only (i.e., performing latent reasoning for intermediate steps):
To+TR Σ t=To+1
TQ--RRTA Σ
CLM Branter
log Milaator (rt rets Q)
log Moxxta (at
act, R, Q)
(7)
I=To+Tn+1
To To+TR
LLM Bander
Σ
log Monaaa (ar act, Hteacher Q)
(8)
TT++
where Hpeacher
teacher h teacher
+1 h T. + + denotes the extracted last-layer hidden states
teacher
of the reasoning steps, provided input to the student model to enable teacher-foreing. The overall loss for canguage-to-laten alignment is thus:
L1
=
LLM Breache
+
LLM Banden
+
Sonsistency
where Oreaherr and Osudent refer to the original Transformer parameters of the System-2 teacher and System-2 student models, respectively.
Shortcut learning. Given dataset with labeled intermediate reasoning steps R (r1, .Tr. and estimated criticality binary labels for each step loss (Elbayad et al., 2019 Elhoushi et al. 2024) that enforces consistency between the intermediate hidden states of the System- model and the final-layer hidden states of the System-2 student model (which has been trained through language-to -o-taateett--tett alignment):
(TTR) CTR), we define an early-exit
L To+Th
exiil
Σ
Σ
cs.:MSE(sg
h
system-1 1-Straining
(10)
Osysten 1.5
1= ==
where et, is the early-exit weight for the I-th layer at step t, designed to distinguish critical and non-critical steps. Specifically, we define ei, encourage non-critical steps (c: 0) to exit earlier. and critical steps (c: = 1) to exit later, according to:
Eler +a Eラ
( 1- Elr 7
c1 a)
(11)
In parallel, the System-1.5 model is also supervised by NLL over the final answer generation:
To+Ta+TA
LLM 01gaten-15
Σ
log M Moyyonns (as acts Hottddntt Q)
(12)
TTT++Tn+1

---

Table 1: Quantitative results reasoning tasks, reported in terms of accuracy (Acc.), number of decoding steps before generating the final answer (# Steps), average FLOPs reduction rate per step relative to CoT (FLOPs r.), and overall inference speedup relative to CoT measured by wall-clock time. Best and second-best results are highlighted with bold and underline, respectively.
Method
Ace. (ce.(%)
(f))
# Steps
GSMBK
FLOPPE. e.
Speedup
Acc. (%)
GSM-HARD # Steps
FLOPSE.
Speedup
Acc. (%t
&.Sepp
StrategrOA
FLOPs r.
Speedup
CoT
46.94
26
38,32
26
47.62
52
LITE
44.51
28
1.84x
1.6x
57.01
1.56x
1.4%
46.15
42
1.96x
2.36x
LayerSkip
1.8x
iCOT
32.14
1,02x
13,45x
23.17
1000x
6.17x
34,42
2
1822
26.47
Caccout
36.75
1,02%
11.98%
28.25
1.02%
2,07x
18.67
2
1.02%
27.25x
CODI
43.78
2.
1,02x
13.37%
35.91
1002
6.3x
45.12
2
1.02%
25,06x
loken
46.32
38
1,00x
0.8%
38.17
100%
0.89
48.14
61
1.00x
0.82
System-
46.66
20.27
12.45x
2
2.12x
55.65%
where Hu extracts the System-2 student model to enable teacher-forcing. The overall loss for shortcut learning is then given by:
L
LLM 15pttm
+
Beady--exi (hpttmas.
(13)
where Psystem-1.5 15 refers to the parameters of the router-adapter modules. The underlying Transformer parameters are initialized from sudeet and are kept frozen during shorteut learning.
3 Experiments
Datasets, We evaluate the effectiveness and efficiency of System-1.. 1. Reasoning on two reasoning- intensive tasks: mathematical reasoning and common sense reasoning. For mathematical reason- ing, we train on the augmented GSM8K dataset (Deng et al., 2023). which extends the original GSM8K (Cobbe et al.) 2021) with a larger set of grade school- math problems. For common- sense reasoning, we use StrategyQA QA (Geva et al.) (021), which contains multihop questions annotated with supporting facts. Each sample includes a question, decomposed subquestions, and a yes/no answer. We merge the annotated facts and subquestions For both datasets, we label each reasoning step with criticality annotations using atomic thought decomposition, as described in Section 2.2 to distinguish between critical and non-critical steps. We train models on the official training splits and evaluate performance on the respective test sets for in-domain evaluation. Additionally, for mathematical reasoning, we conduct out-of-domain evaluation on GSM-HARD (Gao et al. 2023), a dataset with increased reasoming difficulty designed to test generalization beyond the original GSM8K.
coherent CoT sequence for training.
Baselines. In addition to CoT fine-tuning, we compare System- 1.5 Reasoning against six efficient reasoning methods based on supervised fine-tuning, consisting of: (1) language-space conditional computation methods: LITE /Varshney et al. 2023) and LayerSkip (Elhoushi et al. 2024), which improve efficiency by applying early exit mechanisms; (2) Latent-space compressed reasoning: iCoT (Deng et al. 2024). Coconut (Hao et al. 2024), and CODI (Shen et al. 2025). which compress natural language CoT into compact continuous latent representations; (3) Latent-space extended reasoning methods: pause token (Goyal et al. 2023), which insert extra latent states to delay output and allow for extended internal reasoning. Since the officcill implementations of iCoT and Coconut are based on GPT-2 124M (Radford et al. 2019), while LayerSkip builds on the LLaMA 2 (Touvron) let al., 2023) and LLaMA 3 (Grattafiori et al. 2024) series model, we implement two backbone versions of Systemm1-- 1.5 Reasoning, one using GPT-2 124M and another using LLaMA 3.2 1B, to ensure fair comparisons between different baselines.
3.1 Main Results
System-1.5 Reasoning outperforms previous state-of-the--rt methods in latent-space reason- ing in both accuracy and efficiency. As shown in Table compared to iCoT, Coconut, CODI, and pause token, System-1. 1.. Reasoning achieves higher accuracy and greater overall speedup. In GSM8K, compared to original CoT reasoning, latent-space reasoning reduces intermediate token generation by 92.31 % during inference. This reduction is even more pronounced on Strategy reaching 96.15%. Additionally, compared to early-exit methods, System-1. 5 Reasoning achieves a 1.95x reduction in average FLOPs per decoding step, due to its dynamic step shortcut mechanism,
7

---

which allows hidden states from early-exit layers to be directly copied and reused in the next decoding step. In contrast, early-exit methods still reprocess these states starting from the first layer.
By combining early exits along the model depth and shorteut copying across decoding steps, System-1.5 Reasoning achieves overall inference speedup of 20.27x GSM8K, further improving upon the approximately 10x speedups achieved by previous latent reasoning methods. On StrategyQA, it further accelerates inference by over 55x, significantly enhancing reasoning efficiency.
System-1.5 Reasoning matches CoT fine-tuning on challenging mathematical reasoning tasks and outperforms it on text-rich commonsense reasoning. On GSM8K and GSM-HARD, System-1.5 Reasoning achieves 46.94% and 38.32% accuracy respectively, closely matching CoT fine-tuning results of 46.67% and 38.28%. On StrategyQA, a task requiring reasoning over multiple pieces of textual evidence, System-1.5 Reasoning ing achieves 48.61 % accuracy, outperforming CoT's 47.36%. Similar improvements are observed for latent-space reasoning methods such as pause token, highlighting the promising potential of System-1.. Reasoning in reducing dependency on explicit textual tokens and enhancing the model's inherent reasoning capabilities.
3.2 In-depth Analysis
Direct distillation from language to latent more effective for shortcut learning in System-1.5 Reasoning. We modify the language-to-latent alignment into a curriculum- based learning process, following the approach used in Coconut. This curriculum consists of multiple stages where natural language thoughts are progressively replaced with latent- space thoughts train the latent reasoning model (analogous to using fine-tuned Coconut model as the System-2 student). We then ap- ply the same shortcut learning procedure to train System-1.5 Reasoning. We follow Co- conut's original training schedule, using six epochs in the initial stage and three in each sub- sequent stage. For comparison, we also evalu- ate CODI, distillation- based latent reasoning model that performs competitively in [ as the System-2 student for shorteut learning.
50-
45-
Performance
85
30-
Coconua-Systame System-15
20
co0l-system-15
Ours-System-1
1.5
Joint Leaming para. 5L
Figure 3: Ablation results on optimizing System- 1.5 (shown in solid color) using different System-2 students (shown in light color), namely Coconut- System-1.5 and CODI-System--155 as well as al- ternative learning strategies: joint learning of language-to- to-seennt alignment and shortcut learn- ing, and full- hull-parameter shortcut learning (SL).
Figure 3 shows the results using different System-2 students to train System-1.5 model. We observe significant accuracy drop in the final System-1.5 Reasoning performance when using Coconut as the System-2 student. In addition Coconut's suboptimal perfor- mance observed in Table [ another possible reason for the gap is that its hard curriculum schedule between language and latent-space reasoning limits the flexibility and complete- ness of latent-space modeling. In contrast, System-1.5 Reasoning leverages both "hor- izontal" and "vertical" shortcut across decoding steps and model depth, requiring a more flexible latent reasoning structure that hard curriculum distillation cannot effectively support.
1.0
46.37
47.27
47.77
4830
48.78
40.2 25
49
0.8
46.73
47.25
47.71
4B23
40.75
49.23 26
48
Adapth
0.6
46.06
47.08
47.58
48.1
48.67
40
0.5
46.55
45.50 90
48.05
0.2
46.11
47.55
40.13
0.1
46
16
32
64
Astop
Figure 4: Controllable test-time scaling by tuning the depth exit threshold Atepth and decoding step constant Asep in System-1.5 Reasoning. The red grid-aligned Pareto boundary highlighting the frontier of configurations surpassing the CoT base- line (46.94% accuracy in Table Dp.
Joint learning and full hull-parameter shortcut learning degrade the performance of System-1.5 Reasoning. We further investigate the training strategy of System-1.. .5 Reasoning by exploring two variants: (1) Joint learning, where language- la-lat alignment and shorteut learning are conducted simultaneously, ie., distilling the hidden states of natural language thought directly into the shortcut-exited hidden states; and (2) Full-
8

---

parameter shortcut learning, where instead of freezing the original Transformer parameters, all model parameters are updated during shorteut learning.
As shown in Figure 3 both joint learning and full-parameter shorteut learning degrade the final performance of System-1.5 Reasoning, with joint learning exhibiting more pronounced drop. We attribute this to optimization conflicts between the two groups of parameters: Transformer parameters responsible for latent reasoning and router parameters responsible for shorteut routing. These confficts likely hinder the model's 's ability to simultaneously preserve flexible latent reasoning paths and optimize dynamic shortcut decisions.
System-1.5 Reasoning enables flexible budgel-controllable test-time scaling, Unlike language- space reasoning, where scaling test-time computation often requires exploring complex structural dependencies (e.g., tree-like exploration) or enforcing solution consistency across multiple tra- jectories (Zhang et al. 2025). System-1.5 Reasoning allows for fine-grained control over com- putational budgets through simple threshold tuning during inference. Specifically, System-1. Reasoning inherently introduce two control parameters: the depth exit threshold Adopth. which adjusts the adaptive computation depth for each decoding step (vertically across model layers, where Adeph = forces the computation to proceed the final layer as in a standard Transformer) and the decoding step constant Astep, which determines when to halt intermediate latent thought generation and output a final answer (horizontally across decoding steps).
Figure 4 shows the performance under different configurations of Adept and Assp- We observe that performance is approximately equally sensitive to adjustments along both dimensions, further validating the motivation for adaptive reasoning across both the model depth and decoding steps. Moreover, depth scaling saturates more quickly, and deeper reasoning demands significantly higher training computation, suggesting a synergistic relationship between train-time scaling and flexible test-time scaling in System-1. Reasoning.
4 Related Works
Efficient reasoning models. Efficient reasoning models aim to mitigate the inefficiencies caused by verbose outputs and the overthinking phenomenon (Chen et al. 2024 Team et al. 2025). One line of work focuses on improving language-space efficiency, using length budgeting through prompting (Lee)
al. 2025) and fine-tuning (Liu et al. 2024 Yu et al. 2024 Luo al. 2025a). For example, Chain- Draft (Xu et al. 2025a) applies token budget-aware prompting to guide the model toward more concise reasoning, while S1 (Muennighoff et al., 2025) introduces budget-foreing strategy to terminate the thinking process early. Another line of work focuses on compressing language-space reasoning into latent space, exemplified by iCoT, Coconut (Hao al. 2024), , and CCoT (Cheng & Van Durme 2024), which train models to reason through compact internal representations. Pause token (Goyal et al. (023) and filler token (Přau et al.) (024)) further extend latent-space reasoning by inserting special tokens that delay output and allow for additional latent computation. More recently, Token Assorted (Su et al.) 2025) mixes latent and language thoughts by abstracting early steps into discrete latent codes, SoftCoT (Xu et al. 2025b) mitigates catastrophic forgetting in latent reasoning via prompt tuning, and CODI (Shen et al. 2025) applies hidden-state distillation enhance latent reasoning performance. System-1. 1. Reasoning builds upon latent-space reasoning but further optimizes efficiency by adaptively allocating computation to handle reasoning steps with varying complexity.
Conditional computation. Conditional computation techniques aim to selectively apply heavy computation only to the important parts of an input, thereby reducing unnecessary resource usage. One line of work focuses on system or model switching (Qu al., (025). where inputs are routed between different reasoning systems- ffrrr example, fast System- I and a slower, more deliberate System-2. System-l.x (Saha al. 2024) combines linear reasoning chains and search trees to enable fast and accurate planning in maze navigation tasks, while FaST (Sun et al., 2024a) uses switch adapters dynamically toggle between System-1 and System-2 modes based on task complexity factors such as visual uncertainty or occlusion. Another line of work explores sparse activation via mixture-of-experts models (Jiang et al. 2024; Raposo et al. 2024), where only a subset of the model is activated during inference. Beyond expert routing, dynamic depth models adaptively apply only a portion of the Transformer layers. Early exit (EE) and skip-layer techniques are common approaches: DecBERT (Xin et al. 2020) and CascadeBERT (Li et al. 2021) apply EE in encoder-

---

only architectures, while more recent decoder-only LLMs like LITE @aarsnnyy et al. 2023) and SkipDecode (Del Corro et al., 2023) employ confidence-based exits to speed up inference. Other approaches, such as CoLT5 (Ainslie et al. 2023) and LayerSkip (Elhoushi et al. 2024), perform binary routing to skip redundant attention layers or entire blocks. System-1.5 Reasoning draws inspiration from this line of work and extends it to the latent reasoning space. It not only adapts computation along the model depth via early exits but also introduces dynamic shortcuts across decoding steps, enabling reasoning to proceed efficiently both vertically and horizontally.
5
Conclusion
We introduce System-1.5 Reasoning, a novel reasoning framework that improves inference ef- ficiency in large language models by enabling dynamic shortcut reasoning within latent space. System-1.5 Reasoning adaptively allocates computation based two forms of latentsspace short- cuts: depth shortcuts, which modulate layer-wise computation, and step shortcuts, which streamline decoding steps. To support this adaptive reasoning, System-1.5 Reasoning is trained via a two- stage distillation process: first aligning language-spaa and latent-space reasoning, and then learning shortcut execution paths guided by stepwise reasoning criticality. This training paradigm ensures that the model captures both the expressiveness of System-2 reasoning and the efficiency of bystem---styyl shortcuts. Experimental results on challenging reasoning benchmarks demonstrate that System-1. Reasoning preserves the accuracy of traditional chain-of-though methods while accelerating in- ference by more than 20x. These results highlight the potential of System-1.5 Reasoning as an effective and scalable latent-space reasoning framework for future LLM deployment.
Acknowledgements
This work is supported by the Canada CIFAR AI Chair Program and the Canada NSERC Discovery Grant (RGPIN-2021-031155))
References
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phe---rrn technical report. arXiv preprint arXiv:2504.2 2025.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontanon, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James LeorT. Yi Tay, Yun-Hsuan Sung, and Sumit Sanghai. CoLT5: Faster long-range transformers with conditional computation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5100, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.11866.6 18653/v1/2023. http://2202...2mmaiiiiam 3099. URL
emmlp map..main...m 309/
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et Rt-2: Vision- lariinn-annuag-actttoo models transfer web knowledge to robotic control. arXiv preprint 23Xvv22307.11818 2023.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0955:
2025.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of 2024.
ol-like 1ms. arXiv preprint 2412
Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13177 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 21Xiv22110.14666 2021.
10

---

Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for efficient 11m inference. arXiv preprini arXiv:2307.026628 2023.
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv: 2311.01460, 2023.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405. 2024.
Danny Driess, Fei Xia, Mehdi SM Sajadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pp. 8488, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint arXiv: 10070, 2019.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. LayerSkip: Enabling early exit inference and self-speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12622-12642, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/v1/ acl-long. 601.o/n..........
2024. 21244acl--ong.66 161.
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving IIm reasoning programs with certaindex. arXiv preprint 2rXiv:2412.20993, 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361 2021. doi: 10.1 162/tacl a 00370. URL
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar arXiv:2103.000 2021.

---

Sirui Hong, Xiawu Zheng. Jonathan Chen, Yuheng Cheng, Jinlin Wang. Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint
2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint MrXii:2106609685, 2021.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.0799 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint Ttti:2401.040888 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. bwehbench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Ayeong Lee, Ethan Che, and Tianyi Peng. How well do Ilms compress their own chain-of of-thuuhhtt token complexity approach. arXiv preprint arXiv:2503.0/ 141, 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, pp. 19274- 19286. 2023.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository. 2024.
2024.
Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. CascadeBERT: Ac- celerating inference of pre-trained language models via calibrated complete models cascade. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 475-486, Punta Cana, Domini- can Republic, November 2021. Association for Computational Linguistics. doi: 10.113///v1/022 002/2021. findings-emnin
findings-emnlp.4
Peiji Li, Kai Lv, Yunfan Shao, Yichuan Ma, Linyang Li, Xiaoqing Zheng, Xipeng Qiu, and Qipeng Guo. Fastmets: A simple sampling strategy for data synthesis. arXiv preprint 1121v:2502.11476. 2025a.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. 17419, 2025b.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09111 2022.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01990 019900 2025.
12

---

Tengxiao Liu, Qipeng Guo, Xiangkun Hu, Cheng Jiayang, Yue Zhang, Xipeng Qiu, and Zheng Zhang. Can language models learn skip steps? arXiv preprint arXiv:2411.01855, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: hingtihhammoizzing fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501. 2025a.
Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang. GengRu Chen, Wenbo Su, and Bo Zheng. Deconstructing long chain-of-thought: A structured reasoning optimization framework for long cot distillation. arXiv preprint arXiv:2503. 16385, 2025b.
Meta. The llama herd: The beginning of a new era of natively multimodal ai innovation, 2025.
URL https: //ai meta. www.blog///cccocccccc..cc..wwo.t http///ogg/llaaa--a111
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
OpenAl. Chatgpt: Optimizing language models for dialogue, 2022. URL https::/openai.comm com/ /llog/hhatpt//
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang. Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 2025.
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, et al. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. arXiv preprint arXiv:2503.21614 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap. Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:24044.02258 2024.
Swarnadeep Saha, Archiki Prasad, Justin Chih- Yao Chen, Peter Hase, Elias Stengel- Easgel,Eskiin,.. and Mohit Bansal. System- x: Learning balance fast and slow planning with language models. arXiv preprint arXiv:2407. 1 /4414 2024.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 17456-17472. 2022.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-though into continuous space via self-distillation. arXiv preprint arXiv.2502.21077, 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 0.0.03240...3314,00 2024,
13

---

Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng, Prmbench: A fine- tineggriied and challenging benchmark for process-level reward models. arXiv preprint arXiv:2501.03124, 2025.
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alone Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint MrXiv:2502.03275. 2025.
Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, and Dongfang Liu. Visual agents as fast and slow thinkers. arXiv preprint arXiv:2408.0888
2024a.
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20290. 2024b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599, 2025.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.1222 12211 2024.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL 32b/
Fengwei Teng. Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov Iim test-time scaling. arXiv preprint 1xX 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arxiv:2307.09288 2023.
Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral. Accelerating Ilama infer- ence by enabling intermediate layer decoding via instruction tuning with lite. arXiv preprint arXiv:2310. 10.588,, 2023.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
Xiaoqiang Wang and Bang Liu. Oscar: Operating system control via state-aware reasoning and re-planning. arXiv preprint arXiv:2410. 18963, 2024.
Xiaoqiang Wang, Lingfei Wu, Tengfei Ma, and Bang Liu. FAC Better understanding large language model capabilities by dissociating language and cognition. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13228-1333 13243,, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10118663//1222222mmmmmmmaam //aclanthology . org/ 2224.2024..o2 emnlp-main 734/
URL https
Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, and Bang Liu. Rd mem: Bridging memory retention and retrieval via reversible compression. arXiv preprint arXi:: 2025.
14

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24833 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguisties. doi: 10. 12..6665/1122200mmml-mmmm URL httpps://cclatthhllloe
org/2020.
2020. emnlp-demos 6/
Sifan Wu, Huan Zhang, Yizhan Li, Farshid Effaty, Amirreza Ataci, and Bang Liu. Seeing beyond words: Matvqa for challenging visual-scientifie reasoning materials science. arXiv preprint 0rXiv:2505.18377 2025.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2246-22511 Online, July 2020, Association for Computational Linguistics. doi: 10.18653/v1/ 2020 maidl 204. URL httphtts///cclaattoolooooo org/ 2020 sol-main.200
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502. 18600, 2025a.
Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcor: Soft chain-of-thought for efficient reasoning with llms. arXiv preprint arXiv:2502.12134. 2025b.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 2024.
Qiyuan Zhang. Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Zhihan Guo, Yufei Wang, Irwin King, Xue Liu, and Chen Ma. What, how, where, and how well? a survey on test-time scaling in large language models. arXiv preprint 2:Xi::2503.24235, 2025.
Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. Towards learning generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13624- 1 13634, 2024.
15

---

A Limitations and Societal Impact
Limitations. While System-1.5 Reasoning shows promising improvements in efficiency and performance, it comes with two main limitations. First, reasoning in latent space, despite being compact and efficient, lacks the interpretability of language-space CoT methods. Without explicit intermediate reasoning steps, it becomes difficult to understand, analyze, verify the model'' internal logic. Second, our evaluation is currently limited to medium-scale benchmarks and model sizes. Future work is needed to validate System-1.3 Reasoning across larger-scale settings and broader range of tasks.
Potential Societal Impact. On the positive side, the efficiency of System-1.5 Reasoning could reduce the computational cost of deploying ceasoning-capable LLMs, facilitating their broader adoption in real-world applications. This could contribute to more accessible and sustainable Al technologies, potentially democratizing advanced reasoning capabilities for a wider range users and institutions. On the negative side, the lack of interpretability inherent to latent reasoning poses risks high-stakes or safety-critical settings. Without transparent intermediate steps, it is harder to detect flawed logic or harmful behavior, which may require stronger safeguards than those typically used for explicit language-barr based reasoning systems.
B Experiment Setup
Datasets. We evaluate the effectiveness and efficiency of System--.. 1.5 Reasoning in two two reasoning- intensive tasks: mathematical reasoning and common sense reasoning.
For mathematical reasoning, we utilize the augmented GSM8K dataset (Deng et al. 2023 which extends the original GSM8K dataset (Cobbe et al., (021)) to approximately 400,000 grade school- level math problems. Each problem comprises question, a chain-of- of-thoughtt (CoT) that details the intermediate reasoning steps, and final numerical answer. To assess generalization to more challenging problems, we also perform out-of-domain evaluation using GSM-HARD (Gao et al.. 2023), dataset derived from GSM8K by replacing numerical values with larger, less common numbers, thus increasing the difficulty of the problem.
For commonsense reasoning, we employ the StrategyQA dataset, which consists of 2,780 examples requiring implicit multihop reasoning. Each example includes a concise yes/no question, decom- position into intermediate reasoning steps, and supporting evidence paragraphs. We construct CoT sequences by merging the annotated decompositions and corresponding evidence, enabling the model to learn structured reasoning paths.
To facilitate adaptive reasoning, we annotate each CoT step in both datasets with criticality labels, identifying steps as either critical or non-critical, using atomic thought decomposition, as described in Section 2.2] For both tasks, train System-1.5 Reasoning on the respective training splits and evaluate performance on the corresponding test sets to assess in-domain effectiveness. The inclusion of GSM-HARD allows us further evaluate the model's ability to generalize to more complex mathematical reasoning scenarios.
Baselines. Beyond standard CoT fine-tuning, we compare System-1.3 -.. Reasoning against six ef- ficient supervised fine-tuning- muning-based reasoning methods, categorized into three classes: (1) Language- space conditional computation methods: LITE (Varshney et al. 2023) and LayerSkip (Elhoushi et al., 2024), which improve efficiency by applying early exit mechanisms on decoding natural language thought sequences; (2) Latent-space compressed reasoning methods: iCoT (Deng et al. 2024). Coconut (Hao et al.) 2024), and CODI (Shen et al. 2025), which compress natural language CoT into compact continuous latent representations; (3) Latent-space extended reasoning methods: pause token (Goyal et al., 2023), which insert extra latent states to delay output and allow for extended internal reasoning.
Since the official implementations of iCoT and Coconut are based on GPT-2 124M, while LayerSkip builds the LLaMA 2 and LLaMA 3 series, we implement two backbone versions of System- 1.0mm-1... Reasoning, one using GPT-2 124M and another using LLaMA 3.2 IB, to ensure fair comparisons across different baselines.
Evaluation metrics. Accuracy is evaluated by exact match between the generated final answer and the ground truth: specifically, the numerical value in the GSM8K dataset, and the yes/no label in
16

---

StrategyQA. In addition to accuracy, we assess efficiency across three dimensions: decoding steps, computation cost, and overall inference time.
Decoding steps refer to the number tokens generated before producing the final answer. For latent reasoning models that do not generate explicit tokens, instead count the number of latent reasoning steps applied. We unify the terminology across all methods by reporting the average number of decoding steps required the test set.
The computation cost considers both the number of decoding steps and the computational depth traversed within the Transformer layers. That is, although different methods may require similar number of decoding steps or generated tokens, methods like System-1.5 Reasoning and condi- tional computation models with early exits involve significantly fewer layer computations. Following the setting in previous work (Elbayad et al. 2019 Schuster et al.] 2022), we estimate the computa- tional cost using FLOPs (floating point operations). approximating the dot-product and matrix-vecto operations within Transformer layers, while omitting nonlinearities, normalization, and residual connections. We compute the average FLOPs per decoding step and report the average reduction in FLOPs relative to CoT fine-tuning.
The overall inference time is measured by recording the actual wall-clock inference time on the test set. We report the average inference speedup relative to CoT fine-tuning.
Implementation Details. We developed our method using PyTorch (Paszke et al. 2019). The base models-GPT-2 124M (Radford et al., 2019) and LLaMM 3.1-1B (Grattafiori et al. 2024 are initialized from pretrained checkpoints provided by the Hugging Face Transformers library (Wolf al. 2020). During shortcut learning, we insert a router-adapter module into each Transformer layer. The router module is implemented as a feed-forward network (FFN) followed by sigmoid activation. The adapter module is implemented using LoRA (Hu et al.) 2021), with scaling factor = 32, rank = 8, and dropout rate of 0.1. Unless otherwise noted in test-time scaling analysis (Section 3.2). we set the default depth exit threshold Adepth to 0.6 and the decoding step count Asep to 2. We set the loss coefficient for the language-to coctuagg------tttee to-latent alignment (Eq 9 to 1.0, and similarly, the loss coefficient for shortcut learning (Eq. [3) to 3 = 1.0. Fine-tuming is conducted for 8 epochs using the AdamW optimizer (Loshchilov & Hutter 2018), with a maximum learning rate of 2 x 10-5, 31 = 0.9, 32 = 0.99, and a warmup over 6% of total training steps. We use a hatch size of single NVIDIA RTX A5000 (24 GB) GPU, requiring approximately 26 hours for LLaMA 3.2 IB and 5 hours for GPT-2 (1 (124M) for an 8-epoch run. All experiments are conducted over four independent
2. Training is performed
with different random seeds. We report the average results across these runs to ensure statistical stability.
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The paper claims to adaptively allocate computation based on stepwise rea- soning complexity, which is realized through dynamic shorteuts along model depth and decoding steps. These contributions are clearly stated in the abstract and introduction and consistently supported throughout the method and experimental sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We present the limitations of this paper in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer: [No]
Justification: The paper primarily focuses on running computational experiments to support its motivation and empirical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides all necessary details regarding the experimental setup, implementation, and hyperparameters in Section 3 and Appendix B, ensuring that the main results can be reproduced and the conclusions independently verified.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We provide all necessary details regarding the experimental setup, implemen- tation, and hyperparameters in Section 3 and Appendix B to support reproducibility for reviewer assessment. While the code and data publicly available this stage, plan to release them upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guided SaaiiicguudesS/Cooeeee Stttees//CoeSSuubsisiooooooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ htapss.ccppubllc/uuddeess
wwt.bbbmmssooooooooiy.o for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all necessary details regarding the experimental setup, implemen- tation, and hyperparameters in Section and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:Our method achieves over 20% inference speedup, demonstrating not only statistical but also practical significance. Regarding performance, our claim of comparable accuracy is supported by the consistent results observed across four independent runs with different random seeds (see Appendix B).
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the compute resources used, including GPU type and training time, in Appendix B to support reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://wwwappp.ccccpppb////iiia
Answer: [Yes]
Justification: This paper strictly adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both the positive and negative societal impacts our work in Appendix A.
21

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The model is still under development and has not been released for public use. Additionally, no human evaluations involving external annotators have been conducted or disclosed, so safeguards for responsible release are not applicable this stage.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite the corresponding papers official sources for all datasets and tools used in our work, and have ensured that all licenses and terms of use are fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
22

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide a detailed introduction to our proposed efficient reasoning model in Section 2, Section 3, and Appendix B.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do use LLMs for the core method development in our work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hett:a///earrips.cc///feeeeeeeee for what should or should not be described.
24


Reviews:
[{"review_id": "IgQNwCMbF9", "reviewer": "Reviewer_YTmj", "summary": "This paper introduces System-1.5 Reasoning, an adaptive reasoning framework for large language models that uses dynamic shortcuts in latent space to balance efficiency and performance in chain-of-thought reasoning. The approach leverages two shortcut types—model depth shortcuts and decoding step shortcuts—allowing the system to allocate more computation to important reasoning steps and less to easier ones. Experiments on standard reasoning benchmarks show that System-1.5 Reasoning matches or surpasses existing baselines, while significantly reducing inference cost.", "questions": "- How is the routing for depth and step shortcuts initialized or warmed up? Does the router start with random predictions, and are there training curves available showing its learning progress?\n- Is this method compatible with mainstream inference engines (such as vllm or sglang)? These engines are already highly optimized for fast inference. If System-1.5 is not compatible, then its practical advantage may be limited despite the speed improvements shown in the paper.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The proposed framework is well-motivated, introducing flexible, dynamic computational shortcuts at both the model depth and decoding step levels.\n- The method delivers substantial speed-ups—up to 20× faster inference on GSM8K—while maintaining or even improving accuracy over prior work.\n\n**Weaknesses**\n- The experimental section feels a bit thin. Table 1 only reports final results and lacks detailed ablation studies. For example, how do depth and step shortcuts individually contribute to speed and accuracy? Figure 3 examines student selection and joint/full-parameter learning, but does not directly analyze the role or interaction between depth and step shortcuts. What happens if only one type of shortcut is used?\n- There are no concrete case studies showing how depth and step shortcuts are routed in practice. How does the system decide which shortcut to take for different types of questions? Are there potential issues with load balancing?\n- All experiments are conducted on relatively small models (GPT-2 124M, LLaMA 3.2B). While understandable for research, this limits the relevance for real-world applications, where much larger models (>7B) are common.", "was_revised": true, "final_justification": "The author has solved my problem, and I have increased the score to 4.", "rebuttal": "We thank the reviewer for the constructive feedback and for recognizing the strong motivation behind our framework, as well as its substantial speed-ups while maintaining performance.\n\n> **R1-Q1**: Individual contributions and interaction of depth vs. step shortcuts not analyzed.\n\n**R1-A1**: We would first clarify that the **depth shortcut (DS) and step shortcut (SS) are not fully independent**, making it challenging to isolate their effects directly. \n\nSpecifically, SS depends on the hidden states produced by DS, i.e., it reuses early-exited hidden states from intermediate layers instead of restarting from the first layer at every step as in standard Transformers. Without DS, SS would always operate on final-layer outputs, removing its capacity for effective intermediate reasoning and degenerating into a trivial skip.\n\nTo better address this concern, we introduced architectural and training adjustments to approximate ablations with minimized interdependence. In the DS-only setting, we removed SS and routed early-exited hidden states back to the first layer at the next decoding step, effectively enabling adaptive early exit without horizontal reuse. In the SS-only setting, we disabled early exits but retained routing at the final layer, allowing the model to reuse the last-layer hidden state to skip a decoding step and continue reasoning from the first layer, thus simulating adaptive step skipping without dynamic depth control.\n\nThese modifications inevitably reduce the original flexibility and design intent of System-1.5 and the results are summarized as follows:\n\n| Method     | Acc. (%)  | # Steps | FLOPs r. | Speedup    |\n| ---------- | --------- | ------- | -------- | ---------- |\n| Coconut    | 36.75     | 2       | 1.02×    | 11.98×     |\n| SS-only    | 38.24     | 2       | 0.74×    | 15.26×     |\n| DS-only    | 40.02     | 2       | 0.91×    | 12.53×     |\n| System-1.5 | **46.66** | 2       | 1.95×    | **20.27×** |\n\nBoth ablation settings still achieve comparable performance and improved efficiency over standard latent reasoning baselines like Coconut, consistent with prior studies on early-exit in language space. However, both underperform the full System-1.5 model. Specifically, we observe that DS-only achieves better accuracy than SS-only. A likely explanation is that DS-only is functionally a superset of SS-only: during inference, choosing either the first-layer or last-layer exit effectively simulates both full-layer traversal and skip-step behavior, as seen in DS-only execution. Nonetheless, this does not imply that SS is redundant. In fact, the DS-only variant demonstrates weaker efficiency compared to SS-only, suggesting that SS plays a crucial role in reducing unnecessary reasoning steps.\n\nTaken together, these results underscore the complementary nature of DS and SS. Their integration allows the model, under a fixed number of reasoning steps, to dynamically select the most efficient and effective path by allocating more computation to hard steps and skipping simpler ones—capturing the intended design of System-1.5 Reasoning.\n\n> **R1-Q2**: Unclear how shortcut routing works in practice and how decisions are made; possible load balancing concerns.\n\n**R1-A2**: We would like to clarify that **shortcut decisions are made adaptively at inference time**, based on step-level reasoning criticality. \n\nAs described in Lines 147–155, the depth shortcut (DS), i.e., early exit at intermediate layers, is controlled by a depth exit threshold $\\lambda_\\text{depth}$. This threshold is applied to the confidence score output from the router module, which is trained with an early-exit loss to correlate layer depth with reasoning difficulty: critical steps are routed to deeper layers, while non-critical ones exit early at shallow layers.\n\nThe step shortcut (SS) follows the routing outcome of DS. Specifically, hidden states from early-exited layers are directly passed to the next decoding step within the same layer, instead of restarting from the first layer. This design ensures that the SS mechanism builds on the vertical routing structure provided by DS, forming a coherent and efficient reasoning trajectory.\n\nRegarding potential load balancing concerns, we note that this adaptive shortcut mechanism not only mitigates imbalance but also offers fine-grained control over the reasoning budget during inference. As shown in Section 3.2, by adjusting the depth threshold, users can easily scale computation based on available resources or task complexity. This tunability in latent reasoning space provides a simple yet effective alternative to more complex tree-based exploration strategies or multi-path consistency mechanisms used in language-space test-time scaling.\n\n \n> **R1-Q3**: Small model sizes limit applicability to real-world large-model settings.\n\n**R1-A3**: Thank you for the suggestion. We initially adopted GPT-2 (124M) and LLaMA-3.2-1B as backbones to **ensure fair comparisons with previous latent reasoning baselines** such as Coconut and CODI, which are also built on small models.\n\nTo better address your concern, we have scaled System-1.5 Reasoning to two widely used 7B-scale open-source backbones: LLaMA-3.1-8B and Qwen2.5-7B. The results are shown as follows:\n\n| Model          | Method     | Acc. (%) | Speedup |\n| -------------- | ---------- | -------- | ------- |\n| GPT-2 (124M)   | CoT        | 46.94    | –       |\n|                | System-1.5 | 46.66    | 20.27×  |\n| LLaMA-3.1 (8B) | CoT        | 78.42    | –       |\n|                | System-1.5 | 77.85    | 29.36×  |\n| Qwen2.5 (7B)   | CoT        | 76.90    | –       |\n|                | System-1.5 | 75.88    | 29.02×  |\n\n\nWe find that System-1.5 continues to benefit from model scaling, showing consistent performance improvements compared to its 1B-scale counterpart. Moreover, the relative speedup over CoT baselines increases with model size. For example, while System-1.5 achieves a speedup of 20x at 1B scale, it reaches 29x at 8B. This implies that larger models introduce more computational redundancy, which can be more effectively reduced through dynamic shortcuts, leading to better efficiency while maintaining or even improving reasoning accuracy.\n\n\n> **R1-Q4**: Unclear how the router is initialized or warmed up; training curve of the router.\n\n**R1-A4**: **The router is initialized with standard techniques** and progressively learns to allocate computation, with **shortcut paths gradually activated from zero initialization**. \n\nSpecifically, as defined in Eq. (5), the hidden state at each layer is a weighted combination of the standard Transformer path and the shortcut paths, where the router-predicted weight $w$ (output of a lightweight FFN with sigmoid) controls the balance:\n\n$$\nh_{l,t}^{\\text{system-1.5-training}} = \\left( g_{l-1}(h_{l-1,t}) + g_l(h_{l,t-1}) \\right) \\cdot w + f_l(h_{l-1,t}) \\cdot (1 - w)\n$$\n\nThe router parameters are initialized from a standard normal distribution, and the shortcut adapters $g(\\cdot)$ are initialized to zero as LoRA, ensuring that early in training, the model behaves like a standard Transformer and avoids unstable routing. As training proceeds, the model learns to activate shortcut paths based on supervision.\n\nTo quantify this warm-up behavior, we track the $L_2$ norm ratio between shortcut and main-path outputs during training:\n\n$$\n\\text{Shortcut Ratio} = \\frac{\\Vert g_{l-1}(h_{l-1,t}) + g_l(h_{l,t-1}) \\Vert_2}{\\Vert f_l(h_{l-1,t}) \\Vert_2}\n$$\n\nAs figures can not be supported in rebuttals, we summarize the dynamics numerically by reporting representative values of the shortcut-to-main-path ratio across training epochs:\n\n| Epoch          | 1    | 2    | 4    | 6    | 8    |\n| -------------- | ---- | ---- | ---- | ---- | ---- |\n| Shortcut Ratio | 0.02 | 0.07 | 0.15 | 0.28 | 0.36 |\n\nThis consistent growth confirms that routing behavior is learned stably over time, progressively shifting computation from full-path to shortcut reasoning.\n\n\n> **R1-Q5**: Unclear if System-1.5 is compatible with vLLM/sglang or other optimized inference engines.\n\n**R1-A5**: System-1.5 Reasoning is **fully compatible with** optimized inference engines like vLLM and sglang by adopting standard solutions for early-exit decoding and preserving the vanilla Transformer interface.\n\nThe early-exit behavior introduced by System-1.5, where tokens may terminate at different depths, poses a known challenge in efficient batched decoding, commonly referred to as the “**missing hidden state**” problem. Specifically, if one token exits early at a shallow layer and a subsequent token in the same batch continues deeper, then the early-exited token lacks the key/value states required for deeper attention layers during autoregressive decoding.\n\nThis issue is not unique to our method and has been well studied in early-exit LLMs such as CALM, LITE, and LayerSkip and early-exit inference framework. The widely adopted solution is to copy the shallow-layer hidden states forward into the deeper layers when needed, allowing consistent computation across all positions and maintaining compatibility with batch decoding.\n\nSystem-1.5 adopts the same solution. This ensures that early-exited tokens do not disrupt the standard key/value caching mechanism during decoding. Furthermore, System-1.5 retains the same architectural interface as a vanilla Transformer, meaning that other acceleration strategies, such as optimized attention mechanisms, parallel prefill and decode, and hardware-aware scheduling, remain intrinsically compatible.\n\nIn summary, System-1.5 is fully compatible with high-performance inference engines and can benefit from their existing optimizations while providing additional speedups through adaptive early exit and step skipping."}, {"review_id": "2MB0G7glpu", "reviewer": "Reviewer_ws8G", "summary": "This paper proposes an approach to achieve adaptive reasoning to improve the efficiency of reasoning via language tokens. The approach is evaluated on two categories of tasks (GSM8K and StrategyQA) to demonstrate the it achieves the same accuracy as language CoT but significantly improves inference efficiency. The main approach is distillation from language CoT with a curriculum.", "questions": "1. What's the comparison with different baselines in terms of training throughput (speed)?\n2. The approach is evaluated in both math reasoning and commonsense reasoning. But how does the performance differ with problems of different levels of difficulty? \n3. Regarding evaluating controllable test-time scaling, can you also report pass@k with increasing k and compare it to the trend of baselines?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe paper addresses an important problem of improve reasoning efficiency and provides new empirical results on reasoning in latent space. The design of the method to be adaptive and yet controllable is an appealing property for test-time compute scaling.\n\nWeaknesses:\nThe training is very complex and lacks in-depth ablations to understand the importance of different components, e.g. the importance of curriculum, what are important factors for optimization, e.g. should the router be jointly learnt, how does freezing parameters affect training, etc.", "was_revised": true, "final_justification": "The authors provided further results which have addressed some of the limitations in my original review.", "rebuttal": "> **R2-Q1**: Missing ablations on curriculum and optimization decisions.\n\n**R2-A1**: We would like to clarify that **the original submission includes the ablations the reviewer mentions**, and we appreciate the opportunity to further highlight them here.\n\nFirst, regarding the role of curriculum, we **compare curriculum-based language-to-latent alignment (used in Coconut) with our direct distillation strategy** in the first paragraph of Section 3.2. The results show that curriculum learning leads to suboptimal performance.\n\nSecond, in the second paragraph of Section 3.2, we **analyze optimization strategies for training the router**. Specifically, we compare joint learning, where both Transformer and router parameters are optimized together, with our proposed two-stage approach. The results show a significant performance drop under joint learning, likely due to optimization conflicts between the two parameter groups.\n\nThird, to further analyze component-wise contributions, we **conduct additional ablations isolating the depth shortcut (DS) and step shortcut (SS)**. In the DS-only setting, we disable SS and feed early-exited hidden states back to the first layer in the next decoding step, enabling adaptive vertical reasoning without horizontal reuse. In the SS-only setting, we disable early exits but retain the final-layer routing, allowing last-layer features to be reused in the next decoding step, simulating horizontal step skipping without dynamic depth traversal.\n\n| Method     | Acc. (%)  | # Steps | FLOPs r. | Speedup    |\n| ---------- | --------- | ------- | -------- | ---------- |\n| Coconut    | 36.75     | 2       | 1.02×    | 11.98×     |\n| SS-only    | 38.24     | 2       | 0.74×    | 15.26×     |\n| DS-only    | 40.02     | 2       | 0.91×    | 12.53×     |\n| System-1.5 | **46.66** | 2       | 1.95×    | **20.27×** |\n\nWe find that both ablated variants still outperform standard latent reasoning baselines like Coconut in efficiency and maintain competitive accuracy, consistent with prior work on language-space early-exit. However, both underperform the full System-1.5 model. Notably, DS-only yields higher accuracy than SS-only. A possible explanation is that DS-only, by enabling routing to both first and last layers, can simulate both full-path and skip-step behavior. However, this does not imply that SS is redundant. In fact, DS-only demonstrates weaker efficiency than SS-only, highlighting that SS plays a crucial role in reducing unnecessary reasoning steps. Taken together, these results underscore the complementary roles of DS and SS. Their integration enables System-1.5 to dynamically allocate computation across both depth and step dimensions, faithfully reflecting the core design of our method.\n\n\n\n> **R2-Q2**: Missing comparison of training speed\n\n**R2-A2**: Thank you for the question. We would first like to clarify that although System-1.5 Reasoning adopts a two-stage training procedure, **its overall training efficiency remains competitive**.\n\nIn Stage 1 (language-to-latent alignment), our approach leverages a distillation-based setup with **teacher forcing**. Unlike prior latent reasoning methods such as Coconut and iCoT, which rely on step-wise autoregressive decoding and curriculum scheduling, System-1.5 extracts last-layer hidden states from the CoT-trained teacher model, applies stop-gradient, and uses these states as ground-truth features for the student model. This setup enables parallelized Transformer training without autoregressive rollout, significantly improving throughput.\n\nIn Stage 2 (shortcut learning), only the **lightweight router-adapter** modules are updated, with the base Transformer parameters frozen. These adapter modules require gradients for less than 1% of model parameters, making this phase highly efficient and comparable to standard LoRA-style tuning.\n\nWe report the end-to-end training time and inference efficiency of System-1.5 in comparison to CoT and Coconut baselines using GPT-2 (124M) on the same device.\n\n| Method               | Training Time (hrs) | # Trainable Params | Inference Speedup |\n| -------------------- | ------------------- | ------------------ | ----------------- |\n| CoT Fine-tuning      | 4.2                 | 100%               | 1.00×             |\n| Coconut              | 8.6                 | 100%               | 11.98×            |\n| System-1.5 (Stage 1) | 4.3                 | 100%               | –                 |\n| System-1.5 (Stage 2) | 0.6                 | \\~1%               | –                 |\n| System-1.5 (Overall) | 4.9 (4.3 + 0.6)     | –                  | **20.27×**        |\n\nFrom the results, System-1.5 requires only 4.9 hours to train, slightly more than CoT (4.2 hrs) but significantly less than Coconut (8.6 hrs). Notably, its second-stage shortcut learning phase updates only ~1% of the model parameters, making it lightweight and efficient to fine-tune.\n\n\n> **R2-Q3**: No analysis on how performance varies with problem difficulty.\n\n**R2-A3**: We appreciate the question and note that although GSM8K and StrategyQA lack explicit difficulty annotations, we analyze performance variation by comparing GSM8K to the more challenging GSM-HARD, which reveals **consistent degradation across all methods and reduced efficiency gains**.\n\nSpecifically, GSM-HARD increases difficulty by modifying GSM8K problems and serves as an out-of-domain evaluation, compounding the challenge. As shown in Table 1, performance drops for all methods, CoT (46.94%  $\\rightarrow$ 38.32%), Coconut (36.75% $\\rightarrow$  28.25%), and System-1.5 (46.66% $\\rightarrow$  38.28%), confirming increased difficulty. Despite this, System-1.5 maintains accuracy comparable to CoT and outperforms latent reasoning baselines like Coconut and iCoT in the harder setting. Inference speedups are also lower on GSM-HARD, suggesting that increased reasoning complexity limits opportunities for shortcut execution, aligning with the intuition that harder problems reduce the potential for early exits or step skipping.\n\n\n> **R2-Q4**: Missing pass@k analysis under test-time scaling.\n\n**R2-A4**: Thank you for the question. We would first like to clarify that **applying standard pass@k evaluation is not straightforward and may be inappropriate for latent-space reasoning methods** such as System-1.5 Reasoning, Coconut, and iCoT.\n\nThese methods differ fundamentally from language-space reasoning CoT: they do not decode intermediate steps into natural language. Instead, they operate entirely in latent space by passing internal hidden states, either final-layer representations or early-exit features (in the case of System-1.5), as input embeddings to the next step. Since latent steps do not produce vocabulary distributions via the LM head, top-k sampling or beam search cannot be applied to generate diverse reasoning paths during intermediate steps.\n\nAs a result, pass@k evaluation for latent methods can only be reasonably applied to the final answer decoding step. While one could vary the depth exit threshold to simulate different paths, this would alter the reasoning behavior itself and introduce confounding factors, making it an unfair basis for comparison.\n\nNevertheless, for completeness, we report pass@k on the final answer token (only) using top-k sampling at inference for latent reasoning baselines (System-1.5, Coconut), and apply standard top-k sampling across all tokens for CoT baselines:\n\n| Method                                 | Pass@1 | Pass@5 | Pass@10 | Pass@100 |\n|----------------------------------------|-------:|-------:|--------:|---------:|\n| CoT                                    | 46.94  | **48.71**  | **51.23**   | **51.92**    |\n| Coconut (step = 2)                     | 36.75  | 37.88  | 38.15   | 38.60    |\n| Coconut (step = 4)                     | 37.03  | 38.02  | 38.47   | 39.01    |\n| Coconut (step = 8)                     | 37.22  | 38.31  | 38.69   | 39.12    |\n| Coconut (step = 32)                    | 37.44  | 38.59  | 39.03   | 39.55    |\n| System-1.5 (depth = 0.6, step = 2)     | 46.66  | 48.03  | 48.44   | 48.81    |\n| System-1.5 (depth = 0.6, step = 4)     | 47.08  | 48.22  | 48.66   | 48.93    |\n| System-1.5 (depth = 0.6, step = 8)     | 47.58  | 48.36  | 48.80   | 49.12    |\n| System-1.5 (depth = 0.6, step = 32)    | **48.67**  | 48.48  | 48.92   | 49.27    |\n\n\nFrom these results, CoT shows clear gains ($\\sim$5%) with increasing $k$ (1, 5, and 10), benefiting from diversity across multiple sampled reasoning chains. In contrast, System-1.5 and Coconut exhibit much smaller improvements (typically <1%), as top-$k$ sampling is only applied at the final answer token, while all intermediate reasoning steps remain deterministic. This highlights a structural property of latent-space methods: increasing $k$ primarily affects the surface-level output, rather than altering the internal reasoning trajectory.\n\nFinally, we note that large-$k$ sampling (e.g., $k = 100$) in CoT can be computationally expensive and yields diminishing returns. For instance, pass@10 improves from 51.23% to only 51.92% at pass@100, suggesting performance saturation. This indicates that, even with significantly increased sampling budget, language-space reasoning has limited ability to improve its correctness beyond a certain point.\n\nIn contrast, System-1.5 remains both effective and efficient without depending on high-$k$ sampling. Notably, the configuration with depth = 0.6 and step = 32 achieves a reasoning length and computational cost comparable to pass@1 CoT, while matching CoT’s pass@5 accuracy (48.67% vs. 48.71%). This demonstrates that System-1.5’s latent, shortcut-based reasoning offers a more robust and structurally efficient alternative to traditional CoT sampling-based approaches."}, {"review_id": "zDCc19VK02", "reviewer": "Reviewer_zDgS", "summary": "This paper carries out an important and innovative study to decrease the inference cost of reasoning LLMs by training LLM with the mechanisms of depth and step shortcuts. By varying the inference cost on each token by weighing their significance, the model focuses on more important reasoning token and step, and skip the trivial tokens early.\n\nOverall, the proposed method is novel and the writing is clear. However, I have questions on the construction of the dataset for shortcut learning. The authors do not specifically illustrate the rationality of this process and the trustworthiness of the method. Meanwhile, I am curious why the sub-step criticality can be used to determine the criticality of the token in this step?", "questions": "As listed above,", "limitations": "In appendix. Need to be moved to main content.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "I think this paper has many strengths. Hence, I will not list the strengths specifically since I think the paper can be accepted if the questions on dataset construction can be answered,\n\nThe main weakness is: how the dataset for shortcut learning is constructed, Why atom-of-thought to construct directed acyclic graph is one decent way? Is there any analysis or methods to verify the correctness? Meanwhile, why the importance assessment on the sub-step can be used to assess the token importance?", "was_revised": true, "final_justification": "I think this paper is novel and solid. I recommend acceptance in this case.", "rebuttal": "We sincerely thank the reviewer for their encouraging comments and thoughtful questions, and we appreciate the opportunity to clarify the construction process of the shortcut learning dataset and elaborate on the motivations and insights behind its design.\n\n> **R3-Q1**: Why is atom-of-thought and DAG construction a justified strategy? and \n\n> **R3-Q2**: How is sub-step importance meaningfully mapped to token-level importance?\n\n**R3-A1** and **R3-A2**: The reason we adopt atom-of-thought and DAG construction is that **we indeed require higher-level criticality annotation beyond token-wise labels**, in order to bridge the reasoning pattern gap between language-space and latent-space.\n\nSpecifically, the goal of shortcut learning is to adaptively allocate computation in latent space based on reasoning difficulty estimated from language-space supervision. However, latent-space reasoning operates fundamentally differently from language-space reasoning: in language space, **each step corresponds to a discrete token**, whereas in latent space, **each step represents a continuous thought vector that may encode multiple tokens** or partial reasoning states. Prior work such as Coconut has empirically and theoretically shown that these latent representations often exhibit a \"superposition\" property, representing multiple potential paths or intermediate states simultaneously, akin to breadth-first search (BFS) in the solution space.\n\nThis structural mismatch makes token-level supervision too fine-grained and misaligned with the latent reasoning pattern. Instead, we adopt sub-step–level supervision that better reflects the hierarchical nature of multi-step reasoning. To construct these sub-steps, we apply atom-of-thought decomposition, which organizes a chain-of-thought sequence into a Directed Acyclic Graph (DAG), where each node corresponds to a minimal, self-contained reasoning unit. Nodes that depend on others (i.e., derived steps) are labeled as critical, while logically independent or restatement-type nodes are labeled as non-critical.\n\nThis labeling offers a structured and interpretable form of reasoning supervision. During shortcut learning, we assign each token the criticality label of its enclosing sub-step, resulting in segment-wise supervision: all tokens in a critical sub-step are labeled as critical, while those in non-critical sub-steps are not. **This design naturally allows the same token to receive different labels depending on its functional role in different reasoning contexts**. Importantly, since shortcut learning operates entirely on hidden states, rather than through the LM head or vocabulary output, this label propagation does not introduce bias tied to token identity and remains fully compatible with latent-space modeling.\n\nTo empirically validate this decision, we compare our step-level supervision with a fine-grained token-level labeling strategy based on **teacher model confidence**, where low-confidence tokens (confidence < 0.5) are treated as critical and others as non-critical. The results are summarized below:\n\n| Labeling Strategy   | Accuracy (GSM8K) | Speedup    | Notes                           |\n| ------------------- | ---------------- | ---------- | ------------------------------- |\n| Coconut             | 36.75            | 11.98×     | No shortcut supervision         |\n| Token-level (Conf.) | 35.94            | 20.25×     | Token-wise difficulty heuristic |\n| Step-level (Ours)   | **46.66**        | **20.27×** | Based on atom-of-thought DAG    |\n\nAs shown, the token-level labeling strategy leads to significantly weaker performance, even underperforming the Coconut baseline. A possible reason would be the noise and instability inherent in token-wise difficulty estimation, which lacks the structural coherence needed for guiding shortcut optimization in latent space. In contrast, our step-level supervision, derived from atom-of-thought decomposition, provides more stable and meaningful guidance for adaptive reasoning.\nWe will incorporate these clarifications on design motivations and the comparative analysis in the revised version to further support the trustworthiness and rationale behind our shortcut learning supervision."}, {"review_id": "rN7w7FbEfP", "reviewer": "Reviewer_zebB", "summary": "This paper mitigates the inefficiency of Chain-of-Thought reasoning in Large Language Models. While standard CoT improves reasoning accuracy (System-2), it is slow. Existing latent-space reasoning methods improve speed by avoiding text generation but are still suboptimal as they apply uniform computation to all reasoning steps, regardless of their complexity.  To overcome this, the authors propose System-1.5 Reasoning, an adaptive framework that dynamically allocates computation via Depth Shortcut and Step Shortcut. Experiments on mathematical (GSM8K, GSM-HARD) and commonsense (StrategyQA) reasoning tasks show that System-1.5 Reasoning achieves accuracy comparable to standard CoT fine-tuning while providing massive efficiency gains, including an over 20x inference speedup on GSM8K.", "questions": "The shortcut learning stage is guided by 'reasoning criticality' labels derived from atomic thought decomposition. How sensitive is the final performance of System-1.5 Reasoning to the quality and granularity of these annotations?", "limitations": "As mentioned above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The performance gains reported are impressive. Achieving a >20x speedup and >90% token reduction on benchmarks like GSM8K, while maintaining the accuracy of the much slower CoT baseline.\n2. The idea of combining both vertical (depth-wise) and horizontal (step-wise) shortcuts is novel.\n\n**Weaknesses**\n1. The shortcut learning stage relies on criticality labels generated by an external \"atomic thought decomposition\" process. This introduces a dependency and a potential point of failure. How sensitive is the performance of System-1.5 Reasoning to the quality of these annotations? Have the authors considered scenarios with noisy or imperfect criticality labels?\n2. The proposed framework is fundamentally supervised, relying entirely on distillation from a teacher model that generates explicit, pre-defined CoT paths. The critical supervisor signal—the reasoning criticality labels—is also derived from these static teacher traces. This makes it unclear how the method could be extended to a Reinforcement Learning (RL) setting like DeepSeek R1.\n3. The full framework is quite involved, requiring a pre-trained base model, a CoT-finetuned teacher, a System-2 student, a criticality annotation pipeline, and finally the System-1.5 model.", "was_revised": false, "final_justification": "", "rebuttal": "> **R4-Q1** and **R4-Q4**:: Concern about how sensitive System-1.5 is to the quality of criticality annotations used for shortcut learning.\n\n**R4-A1** and **R4-A4**: We thank the reviewer for highlighting this important concern. While the shortcut learning stage relies on criticality labels derived from atom-of-thought decomposition, we find that **System-1.5 Reasoning is relatively robust to label imperfections due to its use of coarser-grained, step-level supervision rather than direct token-level label**. This design mitigates the impact of noise and helps bridge the reasoning pattern gap between language space and latent space.\n\nSpecifically, the goal of shortcut learning is to allocate computation adaptively in latent space according to reasoning difficulty inferred from language supervision. However, latent-space reasoning differs fundamentally from language-space reasoning: in language space, **a decoding step corresponds to a discrete token**, whereas in latent space, **a step is a continuous thought vector that may encode multiple tokens or reasoning states**. Prior work such as Coconut has shown that these latent vectors often exhibit \"superposition\", representing parallel paths or intermediate states akin to breadth-first search (BFS).\n\nThis structural difference makes token-level supervision too fine-grained and misaligned with the nature of latent steps. Therefore, we adopt sub-step–level annotations to provide supervision at a more appropriate granularity. Using atom-of-thought decomposition, we convert multi-step reasoning traces into a Directed Acyclic Graph (DAG), where each node corresponds to a minimal reasoning unit. Nodes with logical dependencies are labeled as critical, and independent nodes as non-critical. This labeling is then propagated to all tokens within each sub-step.\n\nTo directly evaluate the sensitivity to annotation granularity and quality, we compare our step-level labeling against a fine-grained token-level supervision strategy, where criticality is estimated from the teacher model’s token-wise confidence scores, low-confidence tokens (confidence < 0.5) are treated as critical and others as non-critical. The results are shown below:\n\n| Labeling Strategy   | Accuracy (GSM8K) | Speedup    | Notes                           |\n| ------------------- | ---------------- | ---------- | ------------------------------- |\n| Coconut             | 36.75            | 11.98×     | No shortcut supervision         |\n| Token-level (Conf.) | 35.94            | 20.25×     | Token-wise difficulty heuristic |\n| Step-level (Ours)   | **46.66**        | **20.27×** | Based on atom-of-thought DAG    |\n\nAs shown, token-level supervision not only underperforms our step-level method but also falls below the Coconut baseline. This highlights that fine-grained supervision can introduce instability and noise that harms both performance and efficiency. In contrast, our step-level approach offers structurally coherent and stable signals that better guide shortcut optimization in latent space.\n\n\n> **R4-Q2**: Concern about extending System-1.5 to RL settings without ground-truth CoT.\n\n**R4-A2**: We thank the reviewer for this insightful and forward-looking question. While System-1.5 Reasoning currently follows a supervised fine-tuning paradigm, we agree that extending it to a reinforcement learning (RL) setting, particularly in the absence of explicit ground-truth CoT traces, is both valuable and non-trivial. We are actively exploring this direction by **reframing the self-distillation process into a self-reward paradigm**.\n\nSpecifically, at a high level, System-1.5 is trained in two stages:\n(1) distilling language-based CoT into latent-space continuous thoughts (language-to-latent alignment), and\n(2) distilling full-path latent reasoning into adaptive shortcut policies (shortcut learning).\n\nA potential RL-compatible formulation is to reinterpret the first stage as reward modeling, where the goal is to learn a reward function that captures the quality of latent reasoning trajectories, for example, their ability to reach correct answers with minimal computation. This learned reward model could then be used to guide the router policy through reinforcement learning, effectively enabling shortcut selection without explicit CoT annotations.\n\nIn this formulation, the main challenge lies in reward model training: acquiring high-quality signals that reflect step-wise reasoning value. Here, teacher models could still serve as a source of weak supervision, providing approximate scoring based on trajectory alignment or final output correctness. We view this as a promising direction for extending System-1.5 beyond supervised learning, and will include this discussion in the revised version to clarify the RL-compatible potential of our framework.\n\n\n> **R4-Q3**: Concern about the complexity of the full framework\n\n**R4-A3**: Thank you for the thoughtful question. While System-1.5 Reasoning adopts a two-stage training procedure, we emphasize that its overall complexity remains practical for both research and deployment. This is due to two key factors: (1) the **two stages are modular and loosely coupled**, and (2) the **overall training efficiency is competitive**.\n\nFirst, the System-2 student model is compatible with other latent reasoning architectures and can be swapped out without retraining the full pipeline. As shown in Section 3.2, we successfully replaced our student model with alternative latent reasoning backbones such as Coconut and CODI, while maintaining strong performance. This modularity allows the framework to generalize across reasoning strategies.\n\nSecond, the training process is efficient in practice. In Stage 1 (language-to-latent alignment), we adopt a distillation-based approach with **teacher forcing**, extracting the last-layer hidden states from a CoT-trained teacher model and using stop-gradient to guide the student model. This enables parallel training of the Transformer backbone without requiring step-wise autoregressive decoding or curriculum scheduling, as used in prior methods like Coconut or iCoT. As a result, this stage achieves significantly higher throughput.\n\nIn Stage 2 (shortcut learning), only **lightweight router and adapter** modules are trained, while the base Transformer remains frozen. These adapters introduce fewer than 1% trainable parameters and follow the efficiency principles of LoRA-style tuning, making this phase both fast and hardware-friendly.\n\nFollowing your suggestion, we provide a summary of training cost and inference speedup using GPT-2 as the base model:\n\n| Method               | Training Time (hrs) | # Trainable Params | Inference Speedup |\n| -------------------- | ------------------- | ------------------ | ----------------- |\n| CoT Fine-tuning      | 4.2                 | 100%               | 1.00×             |\n| Coconut              | 8.6                 | 100%               | 11.98×            |\n| System-1.5 (Stage 1) | 4.3                 | 100%               | –                 |\n| System-1.5 (Stage 2) | 0.6                 | \\~1%               | –                 |\n| System-1.5 (Overall) | 4.9 (4.3 + 0.6)     | –                  | **20.27×**        |\n\nThese results show that System-1.5 requires only 4.9 hours to train, nearly matching CoT fine-tuning (4.2 hrs) and significantly outperforming Coconut in both training time and inference efficiency. Despite its two-stage structure, the second stage updates only ~1% of parameters, keeping overhead minimal. Most notably, System-1.5 achieves a 20.27× inference speedup, confirming its practical advantages for scalable deployment."}, {"review_id": "0wyFSyQCPf", "reviewer": "Reviewer_hpM3", "summary": "This paper introduces System-1.5 Reasoning, a novel adaptive reasoning framework designed to optimize computational efficiency by combining latent-space reasoning and early-exit mechanisms. The framework utilizes two shortcuts, depth shortcut (DS) and step shortcut (SS), and trained through a two-stage distillation process to selectively allocate computational resources based on step criticality. Empirical evaluations on GSM8K, GSM-HARD, and StrategyQA show that System-1.5 significantly reduces computational overhead and inference time while achieving performance comparable to traditional CoT fine-tuning.", "questions": "- The reasoning pattern in Figure 1 is unclear, particularly regarding the process of transformer block computation. Given that computation at a transformer block requires hidden states from all previous steps from the same layer, the figure suggests certain layers are entirely skipped. Is the lightweight LoRA branch on those also skipped? How exactly is the transformer block, especially attention, computed in cases where layers or branches appear entirely skipped?\n- Following from the first question, if transformer block computation necessitates some computation for each layer, how does the proposed layer skipping in Equation 1 work during inference? Specifically, is skipping (using a lightweight branch) at an earlier layer independent, or does it influence the decision-making for skipping subsequent layers?\n- What is the motivation behind combining depth and step shortcuts in Equation 5 during training, given that they operate as independent branches during inference?\n-  Why does Equation 6 focus specifically on optimizing hidden space consistency only for the answer part? Would it not be more logical to optimize either exclusively the reasoning trace or a combined approach?\n- Why is joint updating of reasoning steps and answers necessary for the teacher model in Equation 7? How does this joint updating specifically contribute to the model's reasoning capabilities?\n- The rationale behind minimizing MSE for critical steps in early layers and minimizing non-critical steps with later layers in Equation 10 is unclear. \n- While the paper states that models are built on LLaMA 3.2 1B and GPT-2 124M backbones, it is not clearly specified which backbone is employed in the experimental results reported. Additionally, it would be helpful to know precisely how the baselines are configured with respect to these backbones to ensure fair comparison.", "limitations": "Please see Questions and Weaknesses above.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n- The paper effectively combines latent-space reasoning with early-exit mechanisms, proposing a well-motivated approach to reduce computational cost in reasoning.\n- Empirical results on benchmarks such as GSM8K, GSM-HARD, and StrategyQA consistently demonstrate that System-1.5 Reasoning maintains high performance while substantially accelerating inference.\n- Ablation study validates the effectiveness of the proposed architecture and highlights its flexibility in controlling computational budget at inference time.\n\n**Weakness:**\n- Several technical details regarding the computation process of each layer and the exact implementation of shortcuts are unclear (see Questions).\n- Motivation behind certain algorithmic choices and module designs remains insufficiently justified (see Questions).\n- The author does not clearly illustrate or examine the impact of the early-exit loss in Equations 10 and 11 on the labeled intermediate reasoning steps. \n- The evaluation methodology has limitations; specifically, using GSM-HARD as an OOD benchmark is questionable given its close relationship with the GSM8K training dataset. Additional evaluation on other datasets, such as MATH, would enhance the reliability of OOD generalization claims.", "was_revised": true, "final_justification": "This is a technically solid work. Through the rebuttal, I understood some of the approaches I previously misunderstood in my review. As the authors also present strong experimental results, I am raising my score to 4.\n\nMy main concerns remain the clarity of the writing and the limited scale of experimentation. I believe these issues can be better addressed in a later revision.", "rebuttal": ">  **R5-W1** and **R5-Q1**: Clarify how transformer block computation and attention are handled\n\n**R5-A1**: We would like to highlight that the computation flow of Transformer blocks in System-1.5 differs between training and inference: *training* uses a **soft mixture** of shortcut and full paths, while *inference* applies **hard gating** for deterministic execution.\n\n**During training, no computation is skipped**. As described in Eq. (5), the hidden state at each layer $l$ and decoding step $t$ is computed by combining three paths, the standard Transformer path, the depth shortcut from the previous layer, and the step shortcut from the previous step, weighted by a router-predicted gate. This means all layers, including the Transformer block and the LoRA adapter, are computed in every step. Attention computation proceeds as in a standard Transformer, since full hidden states are available for all layers and positions.\n\n**During inference, the router enables early exits based on a threshold**. When a token exits early, all computation beyond the exit layer is skipped, including both the Transformer and LoRA branches. No further attention is computed for that token at deeper layers.\n\nA key challenge in autoregressive decoding is handling attention when tokens exit at different depths, leading to the **missing hidden state** problem: if one token exits early while others continue deeper, the early-exited token lacks the key/value pairs required for attention in the deeper layers. This challenge has been widely studied in prior early-exit models such as LITE and LayerSkip. To address it, System-1.5 adopts the standard solution from these works, copying the hidden state of early-exited tokens forward to the deeper layers. This ensures no disruption in attention computation and preserves compatibility with mainstream inference systems such as HuggingFace Transformers and vLLM.\n\n\n> **R5-W1** and **R5-Q2**: Clarify how layer skipping works during inference.\n\n**R5-A2**: We clarify that Eq. (1) describes training-time behavior, where all branches (standard Transformer, depth shortcut, and step shortcut) are evaluated and combined via a **soft mixture**, weighted by a learned gate. No actual skipping occurs at this stage; instead, the model learns how to weigh different paths. During inference, the model switches to a **hard gating** mechanism. \n\nThe rationale is that the router's output weight, learned during training, serves as a confidence score during inference. At each layer, this score is compared against a predefined threshold. If it exceeds the threshold, the token follows the shortcut path and exits early, skipping all subsequent computation, including both the standard Transformer block and the LoRA adapter. Consequently, once a token exits at a particular layer within the current decoding step, it no longer proceeds to deeper layers at the current decoding step, and no further computation is applied. This means that early-exit decisions are final and do not influence or depend on subsequent layers. The resulting hidden state is then forwarded via the step shortcut to initiate the next decoding step.\n\n\n> **R5-W2** and **R5-Q3**: Why are depth and step shortcuts combined?\n\n**R5-A3**: The motivation for combining depth and step shortcuts during training lies in the fact that **they are not entirely independent**.\n\nSpecifically, the step shortcut (SS) relies on the hidden states produced by the depth shortcut (DS). SS reuses the intermediate hidden state from the current layer’s early exit (via DS) at the previous decoding step, rather than reinitializing from the first layer as in standard Transformers. Without DS, SS would always operate on final-layer outputs, losing its capacity to capture intermediate reasoning dynamics and effectively collapsing into a trivial skip connection.\n\nTo support this interaction, we use a unified router to jointly optimize both branches via Eq. (5), enabling dynamic allocation across both depth and step dimensions, improving flexibility at inference.\n\n\n> **R5-W2** and **R5-Q4**: Why apply consistency only on the answer.\n\n**R5-A4**: Applying consistency loss only to the answer portion is **both intentional and necessary** due to the fundamental nature of latent-space reasoning.\n\nEq. (6) distills a natural language CoT-finetuned teacher into a latent-space student model. However, latent reasoning operates fundamentally differently from language-space reasoning. In latent reasoning setting, the last-layer hidden states are not used to predict tokens at each intermediate step; instead, they are recursively passed forward as input embedding for subsequent reasoning. \n\nWhile language-space reasoning **generates one token per decoding step**, latent reasoning represents each step as a continuous thought vector that may **encode multiple tokens or partial reasoning states**. Prior work such as Coconut has demonstrated that these latent representations often exhibit a \"superposition\" property, encoding multiple intermediate paths simultaneously, akin to a breadth-first search (BFS) in the solution space. This capability is central to the expressive power of latent reasoning.\n\nIf we were to enforce consistency across the entire reasoning trace, we would force the latent-space model to mimic the discrete token-level outputs of the teacher. **Since feeding identical last-layer hidden states into the LM head would produce the same token prediction**, this constraint would collapse the latent reasoning process into a token-by-token imitation, thereby eliminating its flexibility and compositionality.\n\nTherefore, we apply the consistency loss only at the final answer step, where both the teacher and student decode into language space. This design anchors the student’s output to the correct answer, while preserving the expressive latent dynamics during intermediate reasoning.\n\n\n> **R5-W2** and **R5-Q5**: Why joint optimization of reasoning and answer in teacher?\n\n**R5-A5**: Eq. (7) represents a standard supervised fine-tuning objective for training a reasoning model, where the model learns to generate both the reasoning steps and the final answer conditioned on the input question. This objective corresponds to the initial training stage in Coconut, and is used in our framework to optimize the natural language CoT teacher model, which captures the full conditional distribution over the reasoning trace followed by the answer.\n\nJointly optimizing both the reasoning and the answer encourages the teacher to generate coherent and logically structured reasoning paths that directly support the final answer. This structured generation is crucial for providing rich, step-by-step supervision to the latent-space student model, to learn compact and effective latent reasoning representations.\n\n\n> **R5-W3** and  **R5-Q6**: The rationale behind early-exit loss.\n\n**R5-A6**: The rationale lies in the design of the weighting term $e_{l,t}$ in Eq. (10), which **assigns different loss weights** based on the criticality of each reasoning step to encourage the router to appropriately allocate computation.\n\nFor *non-critical* steps ($c_t = 0$), the **loss weight increases** with layer depth, as defined by $e_{l,t} = \\frac{\\sum_{i=1}^l i}{\\sum_{j=1}^L j} = \\frac{l(l+1)}{L(L+1)}$. This means shallow layers are assigned lower weights (losses) and deeper layers higher weights (losses), encouraging non-critical steps to exit earlier and avoid unnecessary computation.\n\nConversely, for *critical* steps ($c_t = 1$), the **loss weight decreases** with layer depth, given by $e_{l,t} = 1 - \\frac{\\sum_{i=1}^l i}{\\sum_{j=1}^L j} = 1 - \\frac{l(l+1)}{L(L+1)}$. This penalizes early exits at shallow layers more heavily, encouraging the model to delay exiting and allocate more computation for these important steps.\n\n> **R5-Q7**: Clarification on backbone models.\n\n**R5-A7**: Thank you for the helpful suggestion. We clarify that the results on GSM8K and StrategyQA are based on GPT-2 124M, following the setup used in Coconut. Specifically, we use the GPT2LMHeadModel from HuggingFace with the following architecture: $n_{\\text{emb}} = 768$, $n_{\\text{layer}} = 12$, $n_{\\text{head}} = 12$. The model is trained in float32 precision. Since GPT-2 was pretrained without a padding token, we set the pad_token_id to match the eos_token_id, as done in Coconut.\n\nFor GSM-HARD, we use LLaMA 3.2B, consistent with the LayerSkip baseline. The model is obtained from the official HuggingFace release, and we train System-1.5 using the same configuration: $n_{\\text{emb}} = 2048$, $n_{\\text{layer}} = 16$, $n_{\\text{head}} = 32$. Training is conducted in bfloat16 precision, as specified by LayerSkip.\n\n> **R5-W4**: Additional evaluation on MATH dataset.\n\n**R5-A8**: Thank you for the thoughtful suggestion. To better assess out-of-distribution generalization, we conducted additional evaluation on the MATH dataset using the LLaMA 3.2 1B backbone models. The results are shown below:\n\n| Method      | Acc. (%)  | # Steps | FLOPs r. | Speedup    |\n| ----------- | --------- | ------- | -------- | ---------- |\n| CoT         | 30.12     | 26      | –        | –          |\n| LayerSkip   | 27.92     | 32      | 1.77×    | 1.45×      |\n| iCoT        | 24.18     | 2       | 1.02×    | 13.45×     |\n| Coconut     | 25.31     | 2       | 1.02×    | 11.98×     |\n| CODI        | 27.04     | 2       | 1.02×    | 13.37×     |\n| Pause Token | 27.76     | 38      | 1.00×    | 0.80×      |\n| System-1.5  | 29.85 | 2       | 1.95×    | **20.27×** |\n\nwhile all models exhibit a performance drop on the MATH dataset, owing to its increased complexity in algebra and calculus compared to GSM8K, System-1.5 still achieves accuracy nearly matching its CoT teacher (29.85% vs. 30.12%), while significantly outperforming all baselines in inference efficiency. These results highlight the strong generalization of System-1.5, even on more challenging out-of-distribution reasoning tasks."}]

Author rebuttal:

Meta-review: **(a) Summary**:  
This paper proposes *System-1.5 Reasoning*, an adaptive reasoning framework for large language models (LLMs) that reduces inference cost while maintaining reasoning performance. The core idea is to introduce **two dynamic shortcuts** in the latent space:  
- **Depth Shortcut (DS):** early exit at intermediate transformer layers to avoid unnecessary computation.  
- **Step Shortcut (SS):** skipping reasoning steps by reusing intermediate hidden states.  

The framework combines these shortcuts via a routing module that adaptively allocates computation depending on reasoning difficulty. The method is trained in two stages: (1) distillation from a CoT-finetuned teacher into latent reasoning, and (2) shortcut learning guided by reasoning criticality labels derived from atom-of-thought decomposition.  

Experiments on reasoning benchmarks (GSM8K, GSM-HARD, StrategyQA, MATH) show that System-1.5 achieves **accuracy comparable to chain-of-thought (CoT)** while providing **substantial speedups (20–30×)**. The approach is also shown to scale effectively to 7B–8B models, outperforming prior latent-space reasoning baselines such as Coconut, iCoT, and CODI.

---

**(b) Strengths**:  
1. **Strong motivation and novelty**: Introduces a dual-shortcut mechanism (depth + step) that adaptively balances computation vs. reasoning complexity.  
2. **Impressive efficiency gains**: Achieves up to 20–30× inference speedups while retaining accuracy, with benefits growing at larger model scales.  
3. **Thorough ablations and clarifications in rebuttal**: Authors provide DS-only vs. SS-only analyses, showing complementary contributions, and analyze training throughput, shortcut routing, and router warm-up behavior.  
4. **Compatibility with inference engines**: Demonstrates that the method is compatible with vLLM/sglang and standard KV caching optimizations, ensuring practical relevance.  
5. **Robust supervision strategy**: The step-level criticality supervision is shown to outperform noisy token-level heuristics, validating the atom-of-thought decomposition design.  
6. **Generalization evidence**: Additional results on OOD datasets (GSM-HARD, MATH) confirm robustness, with consistent efficiency gains even under more challenging settings.  

---

**(c) Weaknesses**:  
1. **Initial clarity issues**: Reviewers consistently noted that the paper was difficult to follow, especially regarding the computation process, shortcut routing, and training objectives (though rebuttal clarifications helped).  
2. **Dataset construction concerns**: Reliance on atom-of-thought decomposition and criticality annotations raised questions about subjectivity and trustworthiness, though authors defended the design with empirical comparisons.  
3. **Limited ablations in the original submission**: Key insights (DS vs. SS contribution, curriculum, optimization strategies) were missing from the initial draft and only added during rebuttal.  
4. **Framework complexity**: Requires multiple components (teacher, student, criticality pipeline), which may hinder ease of adoption.  
5. **Limited experimental diversity at submission time**: The main results focused on smaller models and benchmarks. Larger-scale validation (7B/8B) and additional datasets (MATH) were only introduced during rebuttal.  
6. **Writing quality**: Some reviewers (esp. R5) found explanations of equations and algorithmic choices unclear, which could limit accessibility.  

---

**(d) Reasons for Acceptance**:  
The paper makes a **meaningful and novel contribution** to efficient reasoning in LLMs. Its combination of depth and step shortcuts introduces a new mechanism for dynamic compute allocation in latent space. Despite initial weaknesses in clarity and evaluation scope, the rebuttal provided convincing evidence that System-1.5:  
- Achieves **state-of-the-art efficiency–accuracy trade-offs**,  
- Scales well to larger models,  
- Remains compatible with modern inference engines, and  
- Demonstrates robustness across multiple benchmarks.  

The efficiency gains (20–30× speedups) are particularly significant given the cost of reasoning with large models, making this work timely and impactful. While the framework is complex, its modularity and practical compatibility mitigate concerns. Overall, the **strengths outweigh the weaknesses**, justifying acceptance.

---

**(e) Summary of Rebuttal and Reviewer Discussion**:  
- **Reviewer YTmj (R1):** Initially concerned about missing ablations (DS vs. SS), unclear routing, small model scale, router warm-up, and inference compatibility.  
  - **Authors’ response:** Added DS-only and SS-only ablations, clarified routing with thresholds, scaled to 7B/8B models, showed router warm-up curves, and explained vLLM compatibility.  
  - **Outcome:** Reviewer raised score to 4, satisfied with clarifications.  

- **Reviewer ws8G (R2):** Wanted ablations on curriculum/optimization, training throughput, difficulty-based analysis, and pass@k results.  
  - **Authors’ response:** Highlighted existing ablations, provided training time comparisons (System-1.5 nearly as fast as CoT, faster than Coconut), analyzed GSM-HARD vs. GSM8K, and reported pass@k despite limitations in latent reasoning.  
  - **Outcome:** Reviewer acknowledged improvements, maintained score of 4.  

- **Reviewer zDgS (R3):** Concerned about dataset construction and trustworthiness of shortcut learning labels.  
  - **Authors’ response:** Defended atom-of-thought DAGs as structurally coherent supervision, compared against noisy token-level annotations (showing superiority of step-level).  
  - **Outcome:** Reviewer remained positive, kept score of 4.  

- **Reviewer zebB (R4):** Concerned about dependency on annotation quality, lack of RL compatibility, and framework complexity.  
  - **Authors’ response:** Showed robustness to noisy labels, discussed RL extensions via reward modeling, and highlighted modularity + low parameter overhead.  
  - **Outcome:** Reviewer accepted rebuttal, maintained positive score.  

- **Reviewer hpM3 (R5):** Raised technical clarity issues on equations, shortcut design, and backbone setup. Also questioned GSM-HARD as OOD evaluation.  
  - **Authors’ response:** Provided detailed clarifications on computation flow, training vs. inference behavior, DS–SS interaction, consistency loss design, and added MATH experiments.  
  - **Outcome:** Reviewer acknowledged better understanding, raised score to 4, but still noted clarity issues and limited scale at submission time.  

**Final Decision Considerations**:  
All five reviewers converged on **borderline accept (4)** after rebuttal, with improved confidence in the work’s novelty and empirical strength. While weaknesses in clarity, dataset construction, and initial evaluation scope remain, the rebuttal addressed these convincingly. The demonstrated efficiency gains and scalability make this a **worthy contribution for acceptance**.
Decision: Accept (poster)