Latent-Guided Reasoning: Empowering Small LLMs with Large-Model Thinking
Authors: Hanzhu Chen
Lin Yang
Jie Wang
Junhao Yan
Zhe Wang
Xize Liang
Jianye HAO
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Efficient Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, but their high computational costs limit their widespread practical application. We argue that this inefficiency arises from the tight coupling of high-level cognitive planning (devising the solution strategy) and low-level linguistic realization (generating step-by-step text). To address this challenge, we propose a novel collaborative framework that decouples these two processes through Latent Guidance. Our approach implements a division of labor: a large model acts as an Implicit Thinker, performing high-level cognitive planning and compressing its solution strategy into a set of compact latent guidance vectors. A small, efficient model then serves as an Explicit Executor, which receives this latent guidance to generate a concise and effective reasoning chain. This process is enabled by a dual-loss training objective, grounded in information-theoretic principles, where a reconstruction loss explicitly compels the latent guidance to become a high-fidelity representation of the full reasoning chain. Extensive experiments on 8 diverse reasoning benchmarks demonstrate that our method substantially enhances the reasoning capabilities of small models across various scales (from 0.5B to 8B), allowing them to outperform strong baselines and exhibit superior generalization. Notably, our framework boosts small model accuracy by up to 13.9% over its standalone baseline. Our work introduces a new, theoretically-grounded paradigm for empowering small models with large-model thinking, substantially improving the performance-cost trade-off for complex reasoning.

Published as a conference paper at ICLR 2026
LATENT-GUIDED REASONING: EMPOWERING SMALL LLMS WITH LARGE-MODEL THINKING
Hanzhu Chen Lin Yang"t, Jie Wang Junhao Yan', Zhe Wang', Xize Liang', Jianye Hao²
I 'MoE Key Laboratory of inspire Intelligent Perception and Cognition, University of Science and Technology of China
Huawei Technologies Co., Ltd.
*College of Intelligence and Computing, Tiajjin University chenhz @ma 11. uste. edu.cn,
ABSTRACT
Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, but their high computational costs limit their widespread practical application. We argue that this inefficiency arises from the tight coupling of high-level cognitive planning (devising the solution strategy) and low-level linguistic realization (generating step-by-step text). To address this challenge, we propose novel collaborative framework that decouples these two processes through Latent Guidance. Our approach implements a division of labor: a large model acts as Implicit Thinker, performing high-level cognitive planning and compressing its solution strategy into a set of compact latent guidance vectors. A small, efficient model then serves as Explicit Executor, which receives this latent guidance to generate a concise and effective reasoning chain. This process is enabled by a dual-loss training objective, grounded in information-theoree principles, where a reconstruction loss explicitly compels the latent guidance to become high-fidelity representation of the full reasoning chain. Extensive experiments on 8 diverse reasoning benchmarks demonstrate that our method substantially enhances the reasoning capabilities of small models across various scales (from 0.5B to 8B), allowing them to outperform strong baselines and exhibit superior generalization. Notably, our framework boosts small model accuracy by up to 13.9% over its standalone baseline. Our work introduces new, theoretically- grounded paradigm for empowering small models with large- model thinking, substantially improving the performance-cost trade-off for complex reasoning.
INTRODUCTION
The emergence of step-by-step reasoning techniques, such as Chain-of-Thought (CoT) (Wei et al. 2022] Kojima et al. 2022, Xia et al.] (0255) has enabled Large Language Models (LLMs) to complex multi-step reasoning problems. Due to its high degree of interpretability, step-by-step reasoning has been widely adopted in many applications that require detailed and verifiable solutions (Nye et al. 2021 Wang et al. 2022p. However, the combination of massive parameter counts and long CoT generation processes leads to high computational costs (Achiam et al.) 2023 Zhou et al. 2022 Xia et al.) 2025a). This directly hinders their application in real-time or resource-constraines scenarios (Yin et al. [0233 Mei et al., 2025).
demonstrate strong capabilities
To address these high computational costs, existing research has primarily explored two directions. 1) CoT Compression for Large Models aims to shorten the reasoning chains or perform reasoning in latent space reduce generation costs (Goyal et al. 2023 Shen et al. 2025). However, these methods inevitably compromise the readability and interpretability of the original CoT, which is a significant drawback in fields where solution transparency is critical. 2) Knowledge Distillation to Small Models focuses on transferring reasoning capabilities from a large model to a smaller one,
Work conducted during the research internship of Hanzhu Chen (chenhz @ www.l.sttc..e..com Huawei.
I Equal contribution.
Corresponding author. Email: fiewangx@ustt.cdie

---

Published as a conference paper at ICLR 2026
2025 more advanced optimization objectives (Hsich iet al. [2023] Li et al.] (024). While these methods can alleviate the cost issue, the small model's performance is often constrained by its limited parameter count. leading to poor generalization complex, unseen reasoning tasks (Kang et al. 2023 Liao et al. 2025a). Therefore, how to efficiently transfer the reasoning capabilities of large models to smaller counterparts, achieving both high performance and low inference cost, remains a critical challenge.
often via supervised fine-tuning (Chen et al.
To address this core challenge. we argue that this issue arises from fundamental issue: the tight coupling of two distinct functions within the standard autoregressive framework. These are: I) high-level cognitive planning, the process of devising a solution strategy, and 2) low-level linguistic realization, the generation of step-by-step text. In the standard reasoning paradigm, these two functions are intertwined, such that high-level cognitive planning can only be achieved through the medium of costly, step-by-step text generation.
In this work, we propose Latent Guidance, a new collaborative framework designed to empower small models with the advanced planning capabilities of large ones. At its core is mechanism we term Cognitive Distillation, which decouples these two functions through a two-stage process. First, large model, acting an Implicit Thinker, is trained generate a compact set of latent guidance vectors by processing special thought tokens. This training is guided by a dual-loss objective: a task loss ensures the cognitive plan correctly solves the problem, while crucial reconstruction loss compels the latent vectors to encode the entire original reasoning chain. Subsequently, small, efficient model, the Explicit Executor, receives this latent guidance and learns to generate concise and effective textual solution. Our main contributions are as follows:
A Novel Framework for Efficient Reasoning. We propose Latent Guidance, framework empowering small models by decoupling cognitive planning from linguistic realization. Instead of distilling final text, our approach distills the high-level solution strategy, enabling a 7B model to gain up to a 13.9% accuracy over its baseline, approaching the performance of much larger teacher models.
A pnporeticallyGrounnde Training Objective. We introduce a dual-loss training objective where reconstruction loss encourages the latent guidance is a high-fidelity representation of the full reasoning chain. Our formal analysis, grounded in information theory, provides explicit bounds connecting latent capacity to reasoning fidelity, supporting the formation of robust and complete cognitive plan.
Comprehensive Empirical Validation. Across 8 diverse reasoning benchmarks and multi- ple small models (0.5B 8B). Latent Guidance consistently outperforms strong distillation baselines, demonstrating superior generalization. Furthermore, on challenging long- form QA benchmark spanning over 12 domains, our method yields qualitatively superior explanations, highlighting its robust applicability to complex, unseen tasks.
In-depth Analysis of the Latent Cognitive Plan. Through t-SNE visualizations and quantitative probing, we provide concrete evidence that latent guidance vectors organize into distinct clusters of high-level reasoning strategies. This analysis indicates the small model executes structured, abstract plan, rather than just learning feature correlations.
2 RELATED WORK
2.1 KNOWLEDGE DISTILLATION FOR SHAIN-OF-THOUGHE REASONING
Distilling Chain-of-Thouuhh (CoT) reasoning from large language models (LLMs) to smaller, more
efficient ones is a significant research area. Approaches range from Supervised Sine-Tuning (SFT) on teacher-generated chains (Magister
2023) to more advanced methods using distinct optimization
objectives (Hsich et al. 22023; Li et al. 2024). curriculum learning (Liang et al.) (2025). cascaded
learning stages Dai et al. 2024). or blending neural networks with symbolic KB (Liao et al.) 2025b
Bai et al. (026). Furthermore, recent studies have explored enhancing reasoning capabilities through instantiated logical data (Wang et al., 2025). However, these methods can be viewed as a form of outcome distillation, training the small model to replicate the large model's final textual output. This often struggles by requiring the student to replicate a complex reasoning process for which may lack the necessary parametric capacity specialized knowledge (Kang al.) [023] Liao et al. (025a). In sharp contrast, our Latent Guidance framework introduces Cognitive Distillation. Instead of imitating the final text, our method distills the high-level solution strategy itself. The goal is not
2

---

Published as a conference paper at ICLR 2026
for the small model to learn how to reason from scratch, but to learn the linguistic realization of a pre-computed cognitive plan. This shifts the burden of cogmitive planning to the capable large model, allowing the small model to focus solely the linguistic realization task.
2.2 REASONING IN LATENT SPACE
distinct line of research explores latent-space reasoning to accelerate a single large model's inference by circumventing costly autoregressive text generation. Methods include iterative latent processing (Geiping et al. 2025 Kong et al. (0255, using placeholder tokens (c.g., <pause>) for added computation (Goyal al. 2023 Plau al., 2024), or aligning explicit tokens with implicit states via self-distillation (Deng el al. 2023 Shen et al. 2025). Recently, foundational works like COCONUT (Hao et al., 2024) and Light (Zhang et al.]|2025 have validated the feasibility of performing reasoning in continuous latent space. often trading off some accuracy or interpretability for higher throughput and lower memory usage. Our work, however, repurposes latent computation for different objective: empowering small model by transferring the cognitive planning capabilities of a large one. While prior works focus on making large models more efficient, we focus on making small models more capable. This difference in purpose motivates unique design. Crucially, unlike latent reasoning approaches that may lack strong supervision for the internal "thought" process, our framework's reconstruction loss (Crcon) provides a powerful mechanism to ensure the latent guidance faithfully represents the full reasoning chain, yielding a robust, structured cognitive plan for the small model execute.
METHODOLOGY
Latent Guidance framework decouples the complex reasoning process into two specialized stages: cognitive planning, performed by a large model, and linguistic realization, handled by a small model. This section details the framework's architecture and the two-stage training procedure that enables this collaboration.
Our proposed
A bat and a ball cost $1.10 in total. The costs S1.00 more than the ball. How does the
Okay, let's break this down. The tocal cost is $1.10. The problem
Small
states the costs S1. more than the ball. So, if we take the
Language Model
the bul, we are left with $0.10. This amount must be the cost of the ball. So the
total cost of $1.10 and subtruci the extm $1
Latent Tokens'
ball 10 cenis,
double
The logic
to
lead to contradiction. Let me re-rend. Total is $1.10 Bat is
Large Language Model
Hidden State
$1.00 than the ball. The initial calculation
the me
direct. Maybe should stick with it. The answer is $0,10.
Projection
be cost ball.
b+$1.00.
<haart thoughe?
qend, thodgrhoghhp-
The total cest the sum: (b- $1.00)-11.00.
simpliifies 26- 11.00-11.10 Solving for b. 2b-50.10 b-50.05.
Small
Latent Tokens for Thought Guidance
Language
The answer is $0.05.
Figure 1: An overview of the Latent Guidance framework during inference. The large model (Implicit Thinker) generates compact latent guidance vectors from the input question by processing special thought tokens. The resulting hidden states are then passed through a projection layer the small model (Explicit Executor). The small model, conditioned on both the question and the guidance, produces a concise and effective reasoning chain. As illustrated, without this guidance, the small model alone may generate flawed reasoning path, highlighting the critical role of the latent guidance.
3.1 OVERALL FRAMEWORK
The core of our framework is the division of cognitive labor illustrated Figure [ The large model, acting the Implicit Thinker, is trained to produce sequence of compact latent vectors instead of generating lengthy text. These vectors, which we term latent guidance, serve as a high-level cognitive plan for the solution. The small model, acting the Explicit Executor, then receives this guidance. Its sole task to perform the linguistic realization of this plan, generating a step-by-ster reasoning chain and the final answer. The entire process is enabled by two-stage training strategy.
3

---

Published as a conference paper at ICLR 2026
3.2 STAGE 1: TRAINING THE IMPLICIT THINKER TO FORMULATE COGNITIVE PLAN
The primary objective of this stage to train the large model (Implicit Thinker) to take a question Q as input and produce a sequence of information-rich latent vectors that encode its solution strategy. To create explicit anchor points for these latent representations, we introduce a set of special thought tokens: '<start thought>'. '<end thought>', and K placeholder tokens '(< thoughtt >,....< thoughtx >}'
(ri,..., TM and an answer A (a1..... OL), we reformat the target sequence. Instead of predicting the full CoT, the large model trained on a template that replaces the reasoning text with our latent tokens: < start_thougle >< thought, < thought >< end und_thh > A'.
Given a training example consisting of question Q, reasoning chain
Question
Answer
Reasoning
$1.
more
$0.05
probahility problem, we consider Therefore, the cost the ball $0.05.
the bull. How much does the ball cost?
A
Loss, for Task
Loss, for Reconstruction
Question
LLM
Answer
LLM
Reasoning
Figure 2: The dual-loss training for the Implicit Thinker (large model). The model is optimized on two objectives simultaneously: task loss to ensure the correctness of the cognitive plan and reconstruction loss to compel the latent vectors to encode the full reasoning chain.
To ensure the hidden states corresponding these tokens meaningfully encode the complete solution strategy, we employ dual-loss training objective, as shown in Figure
LILM = Linck + Lrecon-
Task Loss (Cask): Grounding the Plan in Correctness. This is standard autoregressive language modeling loss focused solely on predicting the final answer A. This loss ensures that the model's cognitive plan is grounded in the ultimate goal of solving the problem correctly. Let LLEM be the parameters of the large model. The task loss is:
Cussk
log P(ay Q, (thought, a LLLM).
(2)
Reconstruction Loss (Crecon): Encoding the Reasoning Chain. This loss is the cornerstone of our method. After forward pass, extract the last-layer hidden states (h,...., hy } corresponding to the thought tokens. These states form our latent guidance, H guidance- The reconstruction loss then compels these latent vectors contain sufficient information to fully reconstruct the original reasoning chain R:
M log P(r; H guidamce, T'g 0LLM)
Lreson
This objective not just empirically effective; we provide a principled informatiinn-heeorticc analysis in Appendix A Specifically, minimizing Creoon is equivalent to maximizing the mutual information between Hguidance and the reasoning chain R, i.c.,
I(R; H Huuidance
[) 2 H(R Q) Crecon-
This compels that the guidance encodes a complete and faithful representation of the cognitive plan.
3.3
STAGE 2: TRAINING THE EXPLICIT EXECUTOR FOR LINGUISTIC REALIZATION
Once the Implicit Thinker is trained, we use it to generate and store the latent guidance vectors H Hguidance for every problem in the training set. The goal of this second stage is to train the small model (Explicit Executor) to take the question Q and the latent guidance as input, and generate the full, human-readable reasoning chain and answer (R, A).
4

---

Published as a conference paper at ICLR 2026
Bridging Latent Spaces with Projection Layer. Since the large and small models may have different hidden dimensions, their latent spaces are incompatible. We therefore introduce a lightweight projection layer to bridge the two spaces. In practice, this is implemented as:
(5) where the MLP consists of two linear layers with an intermediate dimension of 2048, a GELU activation, dropout layer with rate 0.1, and final LayerNorm. This design provides both capacity and regularization, ensuring stable transfer information.
H guidance MLP (Mgiiaguida
Realizing the Cognitive Plan. The small model then fine-tuned with standard language modeling objective. It learns to generate the complete solution sequence (Rt A) conditioned on both the question and the projected guidance vectors. Let ASLM be the parameters of the small model, the loss is:
Training Objective:
LSLM Σ log P(4 Q, H guidance lecs 0SLM),
where ...... is the concatenated token sequence of the reasoning chain R and the answer A. This training effectively teaches the small model to perform linguistic realization on the cognitive plan provided by the large model.
3.4 DECOUPLED INFERENCE PROCESS
During inference, the framework operates in a highly efficient, two-step manner:
1. Cognitive Planning:
question, the trained large model (Implicit Thinker) performs a single forward pass to generate the latent guidance vectors Hguidance. This step is extremely fast it avoids any autoregressive text generation.
Given
2. Linguistic Realization: The question and the generated Hgnidance are passed to the trained small model (Explicit Executor), which then autoregressively generates the final reasoning chain and answer.
This decoupled process leverages the large model's powerful planning abilities while capitalizing the small model's efficiency for text generation, achieving a superior balance of reasoning accuracy and computational efficiency.
EXPERIMENTS
To comprehensively validate our proposed method, we structure our experimental evaluation to systematically demonstrate our framework's effectiveness, generalization, and core mechanisms. Our experiments organized follows:
First, we establish the Broad Effectiveness of our framework, showing that consistently improves reasoning performance across wide range of models (from 0.5B 8B) on diverse reasoning benchmarks against strong contemporary baselines (Sec 4.2).
Second, demonstrate the Superior Generalization of method. We move beyond standard accuracy metrics to show that our method produces qualitatively superior and more concise explanations challenging long-form QA benchmark that spans over 12 distinct domains (Sec|4.3).
Third, we conduct In-depth Mechanism Analysis dissect the principles underlying these results. Through ablation studies, visualizations, and quantitative probing, we provide direct evidence that the framework learns a structured, abstract cognitive plan rather than superficial feature correlations (Sec 4.4).
To supplement these findings, we provide extensive appendices that substantiate our core claims. This begins with formal theoretical analysis (Appendix ( grounded in information theory, rate-distortion theory, and Fano's inequality, which justifies our training objective and proves the robustness of the learned cognitive plan. To empirically validate these theories, we then provide extensive diagnostics (Appendix B including neural mutual information estimates, latent capacity scaling experiments, and decoder robustness measurements. Notably, these diagnostics confirm that our latent guidance captures a substantial amount of quantifiable information about the complete reasoning chain, estimated at approximately 3.1 nats.
5

---

Published as a conference paper at ICLR 2026
4.1 EXPERIMENTAL SETUP
Datasets To rigorously evaluate performance and generalization, we partition our benchmarks into In-Domain (ID) and Out-of-Domain (OOD) categories. For ID training, we use the splits from GSM8K (Cobbe et al. 2021) and BBH (Suzgun al. 2022), evaluating on their respective test sets. To assess OOD generalization, we test on a wide array of unseen datasets without task-specific fine-tuning: AGIEval (Zhong al. 2023), ARC (Clark et al. 2018) (easy and challenge), Odyssey- Math (Netmind.A] 2024). SVAMP (Patel et al. 2021). ). and AQuA (Ling et al., 2017). We use
ELI5-Test (Fan et al. 2019) for qualitative evaluation of generated rationales.
Models and Baselines Our experiments utilize open-source models for reproducibility, with Qwen2.5-32B-Instruee (Hui et al.] (2024) as the large model (Implicit Thinker). For small models (Explicit Executors). we evaluate on range of base models (LLaMA2-7B (Touvron et al. (023).
LLaMA-3-8B (Dubey et al. 2024). Qwen2-0.5B/1.5B/7h (Team] 2024g) and use the powerful Owen2.5-7B-Instruee (Hui et al. (024) for deeper analysis. We compare against comprehensive suite of baselines including SFT (Supervised Fine-Tuning) Knowledge Distillation (KD), and strong contemporary methods: Std-CoT (Magister et al. 2023), MT-CoT (Li ct al. 2024), Step-by- step HHiih et al. 2023). KARD (Kang et al. 20023) CasCoD (Dai et al. 2024), and NesyCD (Liao
et al. 2025b). Our focus on iistillation-baee baselines ensures fair comparison. Please refer to
Appendix D for detailed experimental setup.
4.2 BROAD EFFECTIVENESS ACROSS DIVERSE MODELS
To establish the general applicability of our approach, we first evaluate its effectiveness across various model families and scales against established distillation methods. The results, presented in Table [ show that Latent Guidance consistently improves the reasoning performance of small models. For example, when applied to LLaMA-3-8B, our framework achieves an overall average accuracy of 73.1% %, an increase of 2.8 percentage points over the strongest competing baseline, NesyCD (70.3%). This performance gain is particularly driven by its superior generalization OOD datasets, where our method achieves an OOD average of 71.9%, significantly outperforming NesyCD's 68.1%. This trend is consistent across other model families; with Qwen2-7B, our method leads the OOD average by 3.6 points. These findings support central hypothesis that decoupling cognitive planning from linguistic realization is broadly effective strategy for enhancing small models' reasoning capabilities. Full results for LLaMA2-7B -B are available in Appendix C
4.3 ANALYSIS OF REASONING QUALITY AND GENERALIZATION
Having established the framework's 's quantitative improvements, we now investigate the drivers behind its strong out-of-domain (OOD) performance. We analyze both the quantitative efficiency and qualitative richness of the generated reasoning chains, using the powerful - as the small model.
Quantitative Performance and Generalizable Conciseness Table 2 reveals two key findings that explain our method's OOD success. First, Latent Guidance consistently boosts accuracy on challenging OOD benchmarks like Odyssey-Math (+7.2% over KD). Second, it produces substantially more concise reasoning chains, reducing the token count compared SFT while simultaneously improving accuracy. This demonstrates that the high-level cognitive plan enables the small model to generate more focused and generalizable reasoning paths, avoiding the verbose, exploratory steps often seen in less-guided generation on unseen problems. The model learns to execute a direct strategy rather than overfitting the stylistic artifacts of the training data.
Qualitative Proof of Broad Generalization We further test our framework's 's generalization on the completely OOD, long-form question-amee -answeinn benchmark, ELIS-Test. As detailed in Table 3. the results show a clear and consistent advantage. Across more than 12 diverse domains, from Biology to Economics, our method's outputs are judged by GPT-40 as substantially superior in both Correctness and Relevance, consistently outperforming both SFT and KD. This strong performance across numerous unseen topics demonstrates that the distilled cognitive plan is not task task-specific but imparts robust, broadly applicable reasoning structure. The small model is not just producing more accurate answers; it is generating qualitatively superior and more insightful explanations, confirming the powerful generalization capability of our framework.

---

Published as a conference paper at ICLR 2026
In- In-Domain
Out-Of-Domain
Methods
Overall Avg.
BBH-test
GSM8K
AGIEval
ARC-E
ARC-C
OOD Avg.
LLaMA-3-8B based
Std-CoT
79,4
61.6
41.3
83.2
71.9
65.5
67.5
MT-CoT
62.8
13.1
43.9
83.6
72.3
66.6
55.1
Step-by-step
64.0
11.5
43.7
84.3
74.6
67.5
55.6
KARD (BM25)
81.4
64.3
43.4
85.6
70.2
CasCoD
32.1
59.1
23.6
34.6
27.7
28.6
35.4
NesyCD
82.2
64.9
44.1
84.7
75.4
68.1
70.3
Ours
82.5
67.4
45.2
91.8
78.6
71.9
73.1
# Qwen2-0.58 based
Std-CoT
65.8
26.7
25.6
43.6
32.0
33.7
38.7
MT-CooT
47.2
5.3
27.7
46.0
35.1
36.3
32.3
Step-by-step
5.2
26.2
41.8
36.2
34.7
30.7
KARD (BM25)
66.3
30.9
23.9
48.9
37.2
36.7
41.4
CasCoD
37.6
27.7
15.6
21.5
14.8
17.3
23,4
NesyCD
68.7
32.2
28.4
46.8
36.7
37.3
42.6
Ours
34.8
29.1
49.1
37.6
38.6
44.3
# Qwen2-1.55 based
Std-CoT
68.2
52.7
34.0
69.3
56.4
53.2
56.1
MT-CoT
58.0
6.7
72.7
54.8
45.8
Step-by-step
48.4
5.8
34.4
72.1
57.6
54.7
43.7
KARD (BM25)
72.2
55.4
31.2
74,0
62.2
55.8
59.0
CasCoD
31.7
53.4
24.7
57.1
47.8
43.2
42.9
NesyCD
74.6
55.8
35.1
73.6
58.2
55.6
59.5
Ours
78.4
56.0
36.1
77.2
60.6
58.0
61.7
# Qwen2-7B based
Std-CoT
71.5
49.9
90.5
80.3
73.6
74.6
MT-CoT
70.0
15.2
49.4
90.9
80.2
73.5
61.1
Step-by-step
68.8
15.2
49.1
72.1
71.8
64.3
55.4
KARD (BM25)
80.2
75.3
49.6
92.1
83.5
75.1
76.1
CasCoD
35.7
72.3
37.4
70.1
63.1
56.9
55.7
NesyCD
80.9
76.3
49.9
91.9
82.9
74.9
76.4
Ours
82.1
77.5
54.4
94.8
86.3
78.5
79.0
Table 1: Performance (%) of different methods various small base models across five reasoning benchmarks. The OOD Avg. column highlights generalization performance. Bold indicates the best performance among distillation-bai based methods for each model. Our Latent Guidance framework consistently outperforms baselines.
Table Comprehensive Performance Evaluation using an instruct-tuned Qwen2.5-7B small model. The table shows quantitative accuracy (%) and average token counts reasoning benchmarks.
Accuracy (%)
Avg. Tokens
Type
Dataset Category
SFT
KD
Ours
SFT
KD
Ours
Quantitative Benchmark Performance
tu-Domain
GSM8K
73.0
80.5
235.4
253.7
128.9
in-Domain
BBH
79.2
77.6
79.8
335.3
337.8
192.9
Out-of- oo--oni
AGIEval
47.1
55.8
56.7
430.8
537.6
211.2
Out-of-Domain
ARC-E
94.5
89.1
96.4
230.3
261.5
122.3
of-ommnin
ARC-C
89.4
83.2
252.3
294.4
138.3
-oooann
Odyssey-Math
15.5
17.9
22.7
522.2
563.1
343.7
Out-of- Domain
SVAMP
83.0
82.9
85.9
155.1
171.8
68.6
Out-of-Domall
AQuA
65.4
69.5
71.7
316.2
427.3
186.4

---

Published as a conference paper at ICLR 2026
Table 3: Detailed GPT-4o evaluation results on the ELIS-Test dataset, focusing on core reasoning quality. We report scores (1-10) for Correctness (Corr.) and Relevance (Rel.). Our method consistently produces more correct and relevant explanations.
SFT
KD
Ours
Category
Corr.
Rel.
Avg
Corr.
Rel.
Avg
Corr.
Rel.
Avg
Biology
7.65
8.20
7.93
7.90
8.10
8.00
8.25
8.65
8.45
Chemistry
7.60
8.20
7.90
7.95
8.05
8.00
8.45
8.65
8.55
Culture
7.95
8.20
8.08
8.40
8.60
8.50
8.50
8.60
8.55
Earth Science
7.45
7.70
7.58
7.85
8.25
8.05
8.75
8.75
8.75
Economics
8.05
8.25
8.15
8.10
8.30
8.20
8.75
9.00
8.88
Engineering
7.90
8.35
8.13
8.35
8.70
8.53
8.40
8.85
8.63
Mathematics
8.45
8.50
8.48
8.20
8.40
8.30
8.95
9.15
9.05
Other
7.65
7.80
7.73
7.85
8.05
7.95
8.40
8.60
8.50
Physics
7.60
8.20
7.90
8.10
8.35
8.23
8.55
8.95
8.75
Psychology
8.45
8.65
8.55
7.80
8.05
7.93
8.75
9.20
8.98
Repost
7.60
7.90
7.75
8.05
8.35
8.20
8.65
8.65
8.65
Technology
8.25
8.55
8.40
7.95
8.40
8.18
8.80
9.15
8.98
4.4 UNPACKING THE MECHANISM OF LATENT GUIDANCE
To better understand why our framework is effective, we now analyze its internal mechanics. We conduct an ablation study to validate our core design choices and use visualizations to gain insight into the learned cognitive plan.
Ablation Study: Validating Core Design Choices We performed ablation study to isolate the impact of our core design choices: the reconstruction loss (Crecm). ). the number thought tokens (K), and the architecture of the projection layer. As shown in Table removing the reconstruction loss leads to a clear performance degradation across all datasets (e.g.. -2.3% on GSM8K), confirming its critical role in ensuring the latent guidance faithfully encodes the cognitive plan. The study on the number of thought tokens shows that reducing to K = 3 results in a slight decline, while increasing K 10 yields only marginal gains, suggesting our default of K is a reasonable trade-off.
The ablation on the projection layer architecture further validates our design. Replacing our 2-layer MLP with a simpler 1-layer linear projection leads consistent performance drop (e.g.. -1.5% on GSM8K). This suggests that a simple linear mapping is insufficient to fully bridge the two models' latent spaces, creating an information bottleneck. Conversely, a more complex 3-layer MI.P provides no meaningful improvement and even results in slight performance decrease on some datasets, likely due to the additional capacity leading to overfitting on the training data. This strongly indicates diminishing returns and reinforces our choice of the 2-layer design. These results demonstrate that our proposed configuration effectively balances model capacity and performance.
Table Ablation study on the key components of our framework using the Qwen2.5-7B-Insse small model. Performance is reported as accuracy (%). Our default 2-layer MLP projection with K=5 proves to bell-balanced configuration.
Configuration
GSM8K
SVAMP
ARC-C
Full model (2-layer MIP, K=5)
80.5
85.9
90.9
Ablation on Core Loss and Tokens
w/o Reconstruction Loss (Crecon)
78.2
83.9
89.5
Fewer thought tokens (K=3)
79.4
85.5
89.8
More thought tokens (K=10)
81.1
86.5
91.4
Ablation on Projection Layer
Simple Projection (1-layer MLP)
79.0
84.1
89.5
Deeper Projection (3-layer MLP)
80.4
84.4
89.8
Visualizing the Structure of the Cognitive Plan The ablation study confirms the functional importance of our training objective. To visually inspect what this objective imparts to the latent space, we visualized the H guiduncc vectors from the GSM8K test set using t-SNE (Figure The visualization reveals that the vectors form distinct clusters without any explicit clustering supervision.
8

---

Published as a conference paper at ICLR 2026
Grouped by Clusters
Probing Task
Ace,
ACcr
Reasoning Step Count Prediction
Exact Step Match
36.0
26.3
F-SNE Dimension
Tolerance =1 Step
74.4
50.5
Center 1
Center
Tolerance =2 Steps
94.2
76.9
Mathematical Operator Prediction
Presence of '+' (Binary)
85.4
50.0
Center
Presence of '-' (Binary)
87.0
50.0
Presence of ** (Binary)
84.5
50.0
Presence of 'f" (Binary)
82.1
50.0
(374 points)
Cluster (435 pontsl
Full Operator Set (Multi-class)
54.7
14.6
t-SNE Dimension
Table 5: Accuracy (%) of probing MLP. 'Acct' and 'AccR' denote accuracy from our guidance (Latent) and random baseline, respectively.
Figure 3: t-SNE visualization of the latent guidance vectors for the GSM8K test set.
qualitative review of samples within these clusters showed a strong semantic coherence, with each cluster corresponding a different high-level reasoning paradigm:
Cluster (Sequential Multi-Step Reasoning): Problems requiring sequence of dependent calculations (e.g.. tracking value over time).
Cluster (Direct Calculation Aggregation): Problems solvable by performing parallel calculations on independent inputs followed by final aggregation.
Modeling): Problems defined by algebraic or logical relationships that require setting up and solving equations.
Cluster 3 (Relational & Constraint-Band
This emergent structure provides strong evidence for our hypothesis that the latent guidance represents a high-level, structured cognitive plan, rather than an unstructured feature representation. This offers compelling explanation for framework's success: the small model is not merely mimicking token patterns but is executing a pre-computed, well-structured cognitive plan.
Probing for Semantic Structure in Latent Guidance While the t-SNE visualization suggests high-level organization of reasoning strategies, we conducted series of probing tasks to quantita- tively verify whether the latent guidance vectors, H Hguidance encode specific, fine- greingrinnd semantic information. We trained a simple Multi-Layer Perceptron (MLP) classifier directly the frozen latent guidance vectors to predict distinct properties of the ground-truth reasoning chain. We compare its performance against random baseline, where an identical MLP is trained randomly initialized vectors of the same dimension. The results, presented in Table 5 offer compelling evidence that the latent guidance is semantically rich. Predicting complex properties like the exact number of reasoning steps is inherently challenging, yet our probing model achieves a notable 0.360 accuracy, significantly outperforming the 0.263 baseline. The model's grasp of overall reasoning complexity is even more evident when allowing for minor tolerance, reaching 0.744 accuracy for +1 step and an impressive 0.942 for +:2 steps-both substantially higher than their respective baselines. This quantitative analysis strongly suggests that the large model, guided by the reconstruction loss, learns embed structured and detailed cognitive plan into the latent vectors, capturing not just the general strategy but also specific procedural elements required for the solution.
5 CONCLUSION
We introduced Latent Guidance, a novel framework that empowers small language models with the advanced reasoning of large ones through Cognitive Distillation. Grounded information theory, our dual-loss objective ensures the distilled guidance is high-fidelity representation of the teacher's reasoning process. By decoupling high-level cognitive planning from linguistic realization, our method significantly improves the performance-cost trade-off for complex reasoning. Extensive experiments demonstrate that our framework not only yields substantial accuracy gains with strong OOD generalization but also produces more concise and qualitatively superior reasoning chains. Our in-depth analysis, using t-SNE visualizations and probing tasks, provides direct evidence that method successfully distills structured, high-level cognitive plan, rather than superficial token patterns. This work opens promising new direction for achieving efficient, high- fidelity reasoning through specialized collaboration between large and small models.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by National Key R&D Program of China under contract 2022ZD01 19801, National Nature Science Foundations of China grants U23A20388 and 62021001. We would like to thank all the anonymous reviewers for their insightful comments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2rX17:2303.08774, 2023.
Yinqi Bai, Jie Wang, Lei Chen, Zhihai Wang, Yumeng Li, Mingxuan Yuan, Jianye HAO, Defu Lian, and Enhong Chen. Evolving graph structured programs for circuit generation with large language models. In The Fourteenth International Conference on Learning Representations, 2026. URL https: http///pppeeeeeeeee/eee/e.
n /
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018.
Toby Berger. Rate-Distortion Theory: A Mathematical Basis for Data Compression. Prentice-Hall, Englewood Cliffs, NJ, 1971.
Hanzhu Chen, Xu Shen, Jie Wang, Zehao Wang, Qitan Lv, Junjie He, Rong Wu, Feng Wu, and Jieping Ye. Knowledge graph finetuning enhances knowledge manipulation in large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1XX3v:88030054533 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.:::2/10.144 14168. 2021.
Thomas M Cover and Joy A Thomas. Elements of Information Theory, volume 1. John Wiley & Sons, 2006.
Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. Improve student's reasoning generalizability through cascading decomposed cots distillation. arXiv preprint arXiv:2405.19842, 2024.
Yuntian Deng. Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stu- art Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.0144 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. ur0iv-2407, 2024.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv: 1907.0990..9/11 2019.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05/71, 2025.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.0222 0...02022222.... 2023.
10

---

Published as a conference paper at ICLR 2026
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
David Haussler and Manfred Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics, 25(6):2451-2492, 1997.
Cheng- Yu Hsieh, Chun-Liang Li, Chin-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02300 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICI.R 1(2):3 2022.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.122 12186, 2024.
Minki Kang, Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. Knowledge- augmented reasoning distillation for small language models in knowledge-intensive tasks. In Advances Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, December 10-16, 2023, New Orleans, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22211 2022.
Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, et al. Scalable language models with posterior inference of 2025.
latent thought vectors. arXiv preprint arXiv:2502.01556
Shiyang Li, Jianshu Chen, yelong shen, Zhiyu Chen, Xinlu Zhang. Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. In 2nd Workshop on Sustainable AI, 2024. URL https www.riddrrrruUUSSSSS
7/openreview
Xize Liang, Lin Yang, Jie Wang, Yiyang Lu, Runyu Wu, Hanzhu Chen, and Jianye Hao. Boosting multi-domain fine-tuning of large language models through evolving interactions between samples. In International Conference on Machine Learning, pp. 37427-37441. PMLR, 2025.
Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang. Jun Zhao, and Kang Liu. Skintern: Internalizing symbolic knowledge for distilling better cot capabilities into small language models. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 3203-3221, 2025a.
Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, and Jun Zhao. Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24567-24575,
2025b.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 700.04055411 146, 2017.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adanek, Eric Malmi, and Aliaksei Sev- eryn. Teaching small language models to reason. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1773-1781, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 20.18653/v1/2222 aacr0/22...clooor.15555 URL 2ttp://222....c3acoc 151
atttps://acLantho...o
Kai Mei. Wujiang Xu, Shuhang Lin, and Yongfeng Zhang. Eccos: Efficient capability and cost coordinated scheduling for multi-llm serving. Available at SSRN 5/59339, 2025.
11

---

Published as a conference paper at ICLR 2026
Netmind.AL.
Odyssey-matee
tree / mmain
2024. URL
Cree/main
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03788 2018.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? arXiv preprint 0rX1v:2103.07191, 2021.
Jacob Pfau, William Merrill, and Samuel Bowman, Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing ofain-of-thought thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoe V. Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether offhnnof- can solve them. In Annual Meeting of the Associa- tion for Computational Linguistics, 2022. URL httpss//pp...smmiiccccc...aa org/
api
(CorpusDD:25291763)
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv: 2023.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2008
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou Self-consistency improves chain of thought reasoning in language models. 2022.
arXiv preprint
Zehao Wang, Lin Yang, Jie Wang, Kehan Wang, Hanzhu Chen, Bin Wang. Jianye Hao, Defu Lian, Bin Li, and Enhong Chen. Logictree: Improving complex reasoning of ulms via instantiated multi-step synthetic logical data. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:244244-48877 2022.
Heming Xia, Chak Tou Leong, Wenjie Wang. Yongqi Li, and Wenjie Li. Tokenskip: Control- lable chain-of- of-thhutttttt compression in Ilms, 2025a. URL htttt:////aaxi htttos//aaaxx...oc//////////3/ wttp/bo///aaa/eeee
12067
Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond chain-of-thought: A survey of chain-of-x paradigms for 1lms. In Proceedings of the 31st International Conference on Computational Linguistics, pp. 10795-10809, 2025b.
Huan Xu and Shie Mannor. Robustness and generalization. Machine learning, 86(3):391 4233,,334333311 2012.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuan-Jing Huang. Do large language models know what they don't 't know? In Findings of the Association for Computational Linguistics: ACL 2023, pp. 8653-8665, 2023.
Bin Yu. Assouad, fano, and cam. In Festschrift for Lucien Le Cam: research papers in probability and statistics, pp. 423 435. Springer, 1997.
12

---

Published as a conference paper at ICLR 2026
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502.15589, 2025.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint 2rXiv:2004.06364 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS AND PROOFS
This appendix supplies a self-contained theoretical perspective and detailed proofs that underpin the Latent Guidance framework presented in the main text. Our analysis builds directly on classical results in information theory and statistical learning, including cross-entropy decomposition and KL. non-negativity (Cover & Thomas, 2006), Fano's s inequality (Cover & Thomas 2006) Yu, 1997).
rate-distortion theory (Berger, 1971]. the data-procsssing inequality (Cover & Thomas 2006), and
robustness bounds based on Lipschitz continuity and Pinsker's inequality (Xu & Mannor [012
Villani et al. 2008). We also draw on covering-nune numberr arguments for continuous reasoning spaces (aausserr & Opper 1997), as well as recent neural -nformmaioooo estimators such as MINE
(Belghazi et al. 20118) and InfoNCE (Oord al. 2018).
Our goals are fourfold:
1. Connect the reconstruction objective to mutual information between the chain-or taain-of-thone of-thought and the latent guidance (a precise information-theoreiiic lower bound).
2. Provide an exact-recovery (Fano-style) lower bound illustrating how a small reconstruction error forces the latent guidance to carry nearly the full reasoning information.
3. Prove robustness bounds showing that small perturbations (noise quantization) to the latent guidance induce only bounded changes in the decoder distribution; and discuss the projection (data-pun satta-proesssing bottleneck.
4. Close the loop by giving explicit, empirically grounded upper bounds and practical clarifi- cations on constants (e.g., Lipschitz constants, covering numbers) so that the full range of lalent-capacity requirements sandwich-bounded and experimentally verifiable.
A.1 NOTATION AND SETUP
Let (Q, R) denote data pair where Q is problem statement (context) and R (r1,... . .T) is the corresponding ch (CoT), modeled as a token sequence of length T (for brevity use T to denote the typical reasoning-chaie length). The Implicit Thinker (teacher) produces a latent guidance E xx (a sequence of K vectors in Rd). The decoder/Explicit Executor's reconstruction model denoted qo (R( H, Q); the training reconstruction loss is
Lrecon (0) EP(QOR,HY log go [ H, Q)];
where P(Q. R., H) is the joint distribution induced by the data and the teacher encoder (we allow the encoder to be deterministic stochastic). We use standard information-theoree quantities:
H(R Q), H(R H. H) I(R; H Q) = H(R Q) - H(R) | H, Q)-
A.2 MUTUAL INFORMATION LOWER BOUND FROM RECONSTRUCTION LOSS
The next theorem makes explicit the connection between the reconstruction loss and the conditional mutual information between the reasoning chain and the latent guidance.
Theorem A.I (Mutual-information lower bound via reconstruction loss). For any decoder distributinn
go R H,Q) Q) holds that
[RR; H ( Z H(R | - Lroeon (0). Equivalently, if Creson (0) < H (R| Q) then 1 (R:; H QQ) 2
Proof. This follows directly from the variational representation of cross-entropy and the non- negativity of KL divergence (see e.g., .overr & Thomas 2006).
By definition of conditional mutual information,
I(R; H Q) H(R Q) H(R H,Q).
We relate Erecom (0) to H(R | H, Q). For fixed (H, let PRIH.Q denote the true conditional distribution of Rt given (H, Q). Then for any candidate decoder qe(- H, Q) the decomposition using KL divergence gives
E log go H,Q)) H(PRIQQ (L | H, Q))-
14

---

Published as a conference paper at ICLR 2026
Taking expectation over (H,Q) P(H, Q) yields
Crecon (0) = E H [[PMHH) + Ee(H.)) KL o H, Q1)
Since KL is non-negative, we have
Lrecon (0) ( [P(RHH..)] H( H,Q).
Combine with the mutual information identity to obtain
T(R: H Q) = H(R Q) H(R H,Q) 2 H(R Q) - Lrecon (0),
which is equation 7
Remark A.2. This theorem shows that minimizing Creon (with sufficiently expressive decoder) forces H carry more mutual information about R. In words: low reconstruction loss is certificate that H encodes (nearly) the full chain-of-thought information.
A.3 FANO-STYLE EXACT- RECOVERY LOWER BOUND
The following Fano-type bound shows that the decoder can reconstruct the chain-of-thought with high probability, then / [R; H Q) must be large; this gives bit-level intuition / necessary condition relating latent capacity to the CoT's information content.
Theorem A.3 (Fano-type lower bound). Assume R takes values in a finite set R. Suppose there exists a deterministic decoding rule R(H,Q Q) such that the exact reconstruction probability satisfies Pr[ Q) + R] < E. Then
IRR; H Q) 2 H(R () h(e) - log[[R] - -1),
(8)
where h(l)) = - log - (1- 5) log(1 - =) is the binary entropy function (natural logarithms) and [R] denotes the carlinality of the set of possible reasoning chains.
Proof. This direct application of Fano's inequality (see Cover & Thomas 2006 Yu 1997
Define the error indicator random variable
E I( + R}. Note E is (deterministic) function of (R, Q), hence H(E R. H. 0. Consider the conditional entropy HRR H, Q). Using the chain rule:
HR H) = HE,, R H, Q) H(F R. H. Q) H(E H, Q) + H(R E. H, where we used H(F) R. H, Q) 0. Now expand the second term by conditioning F: H (R) E, Q) = Pr(E
H, H (R E H,) When 0 we have exact reconstruction, i.c. R RRH, Q) and so H(R E H, Therefore
H, (RR E 0, ()
H(R = H ((R E
= Pr(E
But conditional entropy is bounded by the cardinaii,,,, H(R = Q) < log[[ - (since when the possible values for exclude at least the decoded value and are at most [R] - 1). Hence
H(R H2Q) H(F H, + Pr(F 1 H, Q) log(RR) 1).
Taking expectation over (H, Q) gives
HR H, Q) < H(E) Pr(E = log R|
By assumption Pr(E = 1) < and the entropy of a Bernoulli variable with mean Pr(E = 1) is at most h(E). Thus
H(R H,Q) s h(E) + log([R) Combining with T(R: Q) H(R Q H(R H, Q) yields the desired bound equation g
Remark A.4. The Fano bound gives necessary condition in terms of bits: to achieve small error the latent guidance must capture nearly ((R Q) bits (up to the h(z) + e log(|R] 1) slack). For continuous very large R. uses other distortion-oriented analyses (see distoriso below).
15

---

Published as a conference paper at ICLR 2026
A.4 RATE-DISTORTION PERSPECTIVE (INFORMAL BUT CONSTRUCTIVE)
This viewpoint follows the classical rate distortion framework (Berger 1971 Cover& & Thomas 2006).
The rate-distortion viewpoint complements the previous bounds by asking: what is the minimum mutual information [R; H Q) required to ensure an expected distortion (reconstruction loss) not
exceeding D? Formally, for distortion measure di R, R) (we may take d log qo (R H, Q) as probabilistic distortion), the conditional rate-distortion function defined as
(D)
inf ((
[R; H [Q).
In partieular, in the limit of vanishing distortion the minimum rate approaches H Q). Although deriving closed-form R (D) is in general intractable for natural- CoTs, the conceptual implication is that the latent capacity (measured in bits, approximated by K d) must exceed the rate o D) required for desired reconstruction fidelity D. This makes the capacity sweep (varying K and d) a natural empirical diagnostic.
A.5 PATA-PROCESSING/ PROJECTION BOTTLENECK
Let H' f be any (possibly deterministic) projection (for example the MLP projection from LLM hidden dimension into SLM embedding space). The data- cata-processing inequality ensures TR; H' IQ) S T(R: H Q) (Cover & Thomas, 2006). Thus a projection can only reduce the available reasoning information. "This tormalizes the intuition that projection width and expressivity are potential bottlenecks and should be accounted for in ablation studies.
A.6 DECODER ROBUSTNESS: DETAILED, CONSTANT-TRACKES BOUNDS
We now give a detailed proof (with explicit constants) showing that if the decoder's per-step logits are Lipschitz in the latent guidance H, then small perturbations of H induce bounded changes (in KL and coallvariation) to the decoder's full-sequence distribution. This supports the claim that latent guidance is stable to quantization small noise.
Lemma A.5 (Robustness of decoder under Lipschitz logits). Consider an autoregressive decoder that defines a conditional distribution qo [ H, Q) over a token sequence Rt = (ri,..., TT) given latent guidance and context Q. For each time step I. let u((H) (H) E RIV denote the pre-sofimax logits produced by the decoder for the next-token distribution conditioned on the prefix rei
(r Tet, H, Q) =oftmax(u, (H)).
Assume that there exists L > O such that for all and all H1, H2,
(Hi) u, (H2)llo S L H - H2ll-
(9)
Then for the sequence distributions:
KL H1, Q) go H2, Q)) 2LT | - Hall.
(10)
TV H1. ge(= H2, Q)) S LT H1 -H2ll
(11)
Practical estimation of the Lipschitz constant T., While Lemma A.5 assumes the existence of a Lipschitz constant for the decoder logits, a reviewer may question whether Transformer logits satisfy such a property in practice. To address this concern empirically approximate L via small, controlled perturbations the latent guidance vectors and report the procedure here (see Appendix B.5 for exact hyperparameters used in experiments).
Concretely, given a set of N latent guidance samples (H()), (each (D)) E RR RRdd and collection of perturbation offsets A E D, we compute an empirical Lipschitz estimate 7. by
|/ (H()) A) u, H (H
f = max n.A.t
|
where (-(-) E I denotes the pre-softmax logits at time f. In practice we take D to be a small grid of isotropic perturbations (and/or small Gaussian samples) with norms spanning set of magnitudes (e.g.. (10-4, 10-3, 10-2)); Appendix B.5 reports the exact grid, number of samples, and the resulting T,
16

---

Published as a conference paper at ICLR 2026
values observed in experiments. We find that Σ is stable across reasonable choices of perturbation magnitudes, supporting the applicability the Lipschitz- based robustness bounds in Lemmal A.5]
We emphasize that the Lipschitz constant L used in Lemma A.5 should be viewed but idealized quantity. While exact global constants for Transtormers are generally intractable, our empirical estimation via perturbation grids (see Appendix [.5)) provides stable and reproducible proxy. More importantly, our experimental results confirm the predicted trend: models with smaller empirical indeed exhibit stronger robustness to latent perturbations. Thus, although the theoretical bound itself approximate, it captures the qualitative relationship between Lipschitz continuity and robustness that we observe in practice.
informative
Proof. Our robustness analysis builds on Lipschitz continuity arguments (Xu & Mannor 2012) and Pinsker's inequality (Villani al., 2008).
We prove the two incqualities in several steps.
Per-step log-probability difference. Fix time step and denote u = N4 (H1)), ue (H2) and the corresponding softmax distributions p softmax (u). q = softmax(u) over the vocabulary V. For any index we have
logp: log (u4 02) [og e" - LogE, e")]
(u: 0z) log [Riing (e"))=)))) -
Since Ej~gle" 1e- lies in the interval (exp(ming (u, - uj)) exp(max (2, U; }] its log is bounded by min, (Ug Uz) and max, (u, - 22). Therefore
||u v I log E Eyngle 10 [b] ||u 20
and it follows that
llog:: pt log $ lus vsl + |/u o/10c S 21/uu e 20.
Hence
sup log log qel S 221 uu vlloo.
(12)
Per-step KL bound.
The KL divergence between and g is
KL(pllq) Σ p:(logp: - log
Using the bound equation 12
we get
KL(pl]g) I max log - log q:1 < 21/u vlloo. i
Sequence KL via chain rule. poses as:
For autoregressive sequence models the full-sequence KL decom-
KL(go(: (1, go H2, Q))
Σ E H, (L rec H1, () Ten H2, Q1)
Applying the per-step KI. bound and noting that I| (H1) ue (22) oloc is independent of the sampled prefix, obtain
I KL(.) <
Σ u (H₂) u4 (H2)
Using the Lipschitz assumption equation this yields
KL(.) 2LT ||l| - H2ll.
which is equation 10
17

---

Published as a conference paper at ICLR 2026
Total variation via Pinsker. Pinsker's inequality states that for any two distributions P. Q
TV(P.Q) (VP.))><RKI))) (TT((P1I))
Applying Pinsker and substituting the KL bound gives
(V go(
H2, Q)) V 2LT H1 - H2ll HIT H2ll
which is equation
This completes the proof.
Remark A.6.
The Lipschitz assumption equation 9 is natural for decoders with bounded weights and smooth activations, when / is fed through linear MLP layers whose operator norms are controlled.
The constant factors are explicit: the KL changes at most linearly with || H2ll (and with T",, while TV scales most the square root of || H2ll: These explicit dependencies are useful when designing quantization communication schemes for H.
The bound is conservative (we used sup-norm and crude inequality); tighter bounds are possible under stronger smoothness assumptions or when one tracks lower-order terms. leave tighter bounds under stronger smoothness assumptions to future work.
A.7 PRACTICAL DIAGNOSTICS: ESTIMATORS AND EXPERIMENTAL PROTOCOLS
The theoretical results above suggest concrete diagnostics to empirically verify that H captures sufficient reasoning information and to design capacity choices. Below give practical estimators and recommended experimental workflow.
(A) Estimating (R Q) (CoT information requirement). Two pragmatic estimators are:
LM negative log-likelihood (NLL). Use strong pretrained language model и (e.g., the teacher LLM or a separate high-quality LM) and compute the per-sample NLL:
N
1
HNLL
- log (R, Qi).
N
measured in nats (divide by In to convert to bits). Practical tips: use the same tokenization for R and Q; compute on the training set; batch the LM calls; report mean and standard deviation.
2. Compression-basee based estimate (gzip). Use general purpose compressor distribution- agnostic proxy for entropy:
Å Agzip
R
Q
bytes(gzipcconca((,, R.))) x 8.
N
Tips: optionally subtract the gzip size of Q alone approximate conditional size; repeat with other compressors (bzip2) to check robustness.
adopt standard neural MI estimators, including MINE (Beighazi al. 2018) and InfoNCE (Oord et al., 2018). Use one of the standard lower bounds (MINE7 Donsker- Varadhan NWJ InfoNCE). practical choice the DV bound:
(B) Estimating [RR; Q) (neural estimators).
pD = EP(Q. (H, R, Q) log E Q) Iy
where To is critic (e.g.. MLP) optimized by gradient ascent. Practical hints:
Use minibatches and importance sampling for the negative term; maintain moving-average baseline for numerical stability.
Critic architecture: 2-3 layer MLP with layer norm, hidden width 512-1024.
Learning rate: 1c-4-5e-4, train 1k-10k steps depending on dataset size.
Report the estimated MI across epochs and overlay with Creron and downstream accuracy curves.
18

---

Published as a conference paper at ICLR 2026
(C) Capacity sweep protocol. Vary the number of thought tokens K and/or the latent dimension d' (or equivalently K d), and for each configuration:
Train the Implicit Thinker with fixed À and save per-sample H.
2. Train the Explicit Executor using the projected H'.
3. Record: (i) reconstruction loss Lrocon. (ii) estimated H(R Q) (NLL/gzip), (iii) estimated MI I(R; H Q), and (iv) downstream accuracy EM.
Plot downstream accuracy and I vs and mark the empirical estimate of H Q) as vertical reference line. Expect phase transition behavior: accuracy/MI collapse below capacity threshold and saturate above it.
(D) Robustness/ projection ablations.
Noise test: Add zero-mean Gaussian noise A o²) to saved H; evaluate downstream accuracy function of a.
Quantization test: Uniformly quantize H to b bits and measure accuracy drop vs b.
Projection ablation: Vary MLP projection width/depth and measure T (R; H') and accuracy;
small projection width that reduces I indicates projection bottleneck per data fata-processing inequality.
(E) Toy / synthetic experiment for controlled validation. Construct synthetic dataset where R is generated by a known finite-state procedure from Q (so HR Q) known or easily computable). Use the synthetic setting to validate the capacity threshold predicted by rate-distortion and Fano bounds; this is particularly persuasive to reviewers skeptical of purely empirical claims on natural language data.
A.8 DECODER ROBUSTNESS AND LIPSCHITZ CONTINUITY
We assume that the student model's decoder, which maps latent states H' output logits, L- Lipschitz continuous. This property ensures that small perturbations in the latent guidance do not lead disproportionately large changes in the output distribution. For any two latent guidance vectors H' and Hi this is formally expressed as:
||oggis(( Hi) - logits (I))ll < L || - H:ll.
This assumption is standard in robustness analysis. To ground this theoretical assumption in empirical measurement, we approximate via perturbation experiments. We inject bounded Gaussian noise into the latent guidance and measure the maximum induced change across the entire sequence of output logits. This rigorous test yields a stable empirical estimate of Rs 0.263 for the model we consider (see Appendix B.5 for details). Thus, the Lipschitz constant is not left as abstract quantity but is validated experimentally.
A.9 FANO BOUND WITH FINITE COVERING ARGUMENT
In the derivation of the Fano inequality, the size of the answer space, RR), appears. For discrete reasoning tasks, R is finite and well-defined. For continuous or combinatorially large reasoning spaces (c.g., open-ended chains of thought), one may worry that (R] is infinite. To address this, we adopt the standard E-net covering argument (see, e.g., Yu 1997 Haussler Opper 1997): let N₂ denote the covering number of the reasoning space at resolution e. Intuitively, the covering number N represents the effective size of the space, quantifying the minimum number of distinct regions of radius needed cover all possible reasoning chains. Since Fano's 's inequality relies on the cardinality of distinguishable candidates, N₂ serves as the natural replacement for [R] when an approximation error of up to is tolerated. Replacing [R] with No yields
I(R; H Q) > log N: H (pe) Pe log( N 1),
where Pe is the probability that the reconstructed reasoning chain has distance greater than from the original. This substitution preserves the form of the Fano bound and ensures that the inequality remains valid and meaningful in both discrete and continuous settings.
19

---

Published as a conference paper at ICLR 2026
A.10 UPPER BOUND ON MUTUAL INFORMATION
The main bound in the paper establishes a lower limit on the information that latent variables must encode. For completeness, also state trivial but important upper bound:
TR; H Q) S ( (R| Q), K d log 2).
Here H(R | Q) is the conditional entropy of the reasoning outputs, and d log 2 corresponds to the bit capacity of the latent representation. This bound formalizes the intuitive saturation effect: even as latent capacity increases, the mutual information cannot exceed the entropy of the reasoning outputs themselves. This theoretical ceiling explains the flattening trend observed in capacity scaling experiments (Figure 5).
A.11 SUMMARY: SANDWICH BOUNDS AND EMPIRICAL GROUNDING
Combining the above, the mutual information between latent thoughts and answers is bounded
log N H (pe) Pe log(N, - 1) < I(R; H Q) min(H Q), K d log
The lower bound connects error probability with required latent capacity, while the upper bound explains why information saturates capacity grows. Both bounds empirically validated by our diagnostic estimators (Appendix B
A.12 ALGORITIMIC DIAGNOSTIC WORKFLOW (PSEUDOCODE)
1. Train Implicit Thinker with I_task lambda recon
2. For each train sample i, save H_i = encoder_outeee (Q_1)
3. Estimate (R Q) via:
- LM_LLL mean (-log_pi (R Qi))
gzip_ bits_bits__ mean (bytes (gzip
4. Train MINE critic T T_p to estimate I (R;H10) using pairs
(H (___
5. For K in K_values:
for d_values:
train fine-tune Implicit Thinker with (K, save for train
train Explicit Executor from projected H' evaluate downstream accuracy, L_recon, Iah 6. Run robustness evaluations (Gaussian noise, quantization)
on saved H
A.13
CONCLUDING REMARKS
The theorems and lemmas above provide both conceptual and quantitative justification for the Latent Guidance approach:
The reconstruction loss directly lower-bounds H(R H, Q) and thus controls the mutual information (RR; H Q).
Exact recovery (small error) requires I (RR; Q) be close HR Q) (Fano bound), giving bit-level necessary condition latent capacity.
The decoder robustness lemma shows the latent guidance can quantized perturbed with predictable degradation, provided the decoder logits are sufficiently smooth in Ht
Together with the practical diagnostics and capacity sweeps described above, these results give clear roadmap to (i) choose K, d, (ii) validate that H encodes sufficient planning information, and (iii) ensure robustness of the transfer from Implicit Thinker Explicit Executor.
B EXTENDED DIAGNOSTICS: EMPIRICAL ESTIMATORS AND VALIDATION
In this section, provide extended empirical diagnostics to complement the theoretical analysis in Appendix A We employ three complementary estimators SiGoNCE-based mutual information, capacity scaling, and eatrepy-based measures to evaluate whether the latent thought representations indeed satisfy the robustness and boundedness properties derived in theory. The following figures and tables present detailed results and interpretations.
20

---

Published as a conference paper at ICLR 2026
B.1 INFONCE-BAN BASED MI ESTIMATION
To empirically estimate the mutual information (MI) between the latent guidance H and the reasoning chain R, we implemented robust estimator based on the InfoNCE framework. A simple linear critic often learn superficial correlations, leading to spuriously inflated MI values. To mitigate this, critic architecture leverages the pre-trained components of the student model itself. Specifically, we the student model's powerful projection layer encode the latent guidance sequence H, and its token embedding layer to encode the reasoning chain R. On top of these rich representations, we train new linear projection heads to map them into contrastive space.
Figure 4 iillustaate the training history of this robust MI estimator. The estimated MI lower bound steadily increases from near-zero values and stabilizes after approximately 150 epochs, converging to final value of approximately 3.10 nats. Unlike initial simpler estimators which produced theoretically inconsistent results, this powerful estimator yields a reliable and stable measurement that respects the fundamental bounds of information theory, as discussed in Section B.4
InfoNCE Training History
3.0
Mutual Information (nats)
2.5
2.0
1.5
1.0
0.5
0.0
0
25
50
75
100
125
150
175
200
Epoch
Figure 4: Training history of the robust InfoNCE MI estimation. The lower bound converges to 2 3.10 nats after sufficient training, indicating stable and reliable estimate.
B.2 CAPACITY SCALING OF LATENT THOUGHTS
To investigate how the representational capacity of latent thoughts affects MI, we varied the number of latent thoughts K (3,5, 5, 10) while keeping dimensionality fixed. We used the same robust estimator described above for each configuration.
The results, shown in Figure 5 confirm two key theoretical expectations. First, the estimated MI increases monotonically with capacity, rising from 2.71 nats at K = 3 to 2.87 nats at K 5, and further 2.94 nats K 10. This provides strong empirical support for the claim that larger latent space offers greater bandwidth encode the cognitive plan. Second, we observe a clear diminishing returns effect. The MI gain from K to K is substantially larger than the gain from K 5 to K 10. This saturation suggests that K = 5 captures the majority of the useful information, justifying our choice of 5 in the main experiments as an effective trade-off between capacity and performance.
B.3 ENTROPY-BASED UNCERTAINTY ESTIMATES
We next examine the entropy characteristics of the reasoning chains, measured via neural negative log-likelinne log-likelhhood (NLL) as an estimate for the conditional entropy H(RQQ). This value serves as practical upper bound for the mutual information [(R; H that can be encoded in the latent guidance.
Figure 6 shows the NLL distribution: values range from 1.8 to 6.5 nats, with clear unimodal peak around 3.1 nats. The mean NLL is 3.47 nats (std 0.69), indicating that most reasoning chains fall within relatively narrow band of uncertainty. Figure presents the corresponding perplexity
21

---

Published as a conference paper at ICLR 2026
Capacity Sceling Mutual Information
2.00
Estimaaod Munail Information (nans)
2.55
2.00
K--
10000
19000
20000
25000
30000
25000
Latent Capacity (K d. student)
Figure 5: Capacity scaling of mutual information using our robust estimator. Increasing K leads to monotonic increase estimated MI but with diminishing returns, supporting as an efficient configuration.
distribution, which consistent with these findings. These entropy estimates provide the necessary context for interpreting the MI estimation results presented in the final summary.
Distribution
60
Count
50
40
10
Negative Log- Likellhod (NLL. nots)
Figure 6: Distribution conditional NLL (H Q)) across samples. The mean is 3.47 nats, serving as the empirical upper bound for our MI estimation.
B.4 SUMMARY OF DIAGNOSTIC STATISTICS
Table 6 agggreaates the key statistics from our final, robust estimators. The joint interpretation resolves the contradictions observed with simpler estimators and provides clear, quantitatively supported picture of our framework's 's information dynamics:
The average conditional entropy of the reasoning chains, A (R(Q), is estimated to be 3.47 nats. This value represents the total information required to specify the reasoning chain, given the question.
The estimated mutual information, İ(R; H (Q),, converged to 3.10 nats. This result is now consistent with information theory, as fRR; HQQ) < (R[Q)).
22

---

Published as a conference paper at ICLR 2026
Distribution of Perplexity
400
350
300
Countt
200
150
100
50
100
200
300
400
500
Perplexity (exp(NLLI)
Figure 7: Distribution of perplexity (exp(NLL)). The distribution is highly skewed, consistent with the NLL results.
Based on our empirical estimations, the latent guidance successfully encodes a substantial portion of the reasoning information. Our measurements indicate that the captured mutual information (i(;; H [) x 3.10 nats) accounts for a large fraction enstite-estimatte to be approx- imately 89%-of the total information required by reasoning chain, as measured by its conditional entropy (ff (R[Q) = 3.47 nats). While acknowledging that these values are estimations, the high ratio lends quantitative support to the core premise of our Cognitive Distillation framework.
The monotonic increase of MI with capacity (as shown in Figure 5 further supports our theoretical claims, while the diminishing returns justify our model configuration choices.
Metric
Value (nats)
Avg. Conditional Entropy A (H(Q)
3.47
Std. of Conditional Entropy
0.69
Final Estimated MI Ī(R; 17 Q)
3.10
Table Summary of final, validated diagnostic statistics.
B.5 EMPIRICAL ESTIMATION OF DECODER ROBUSTNESS
To empirically validate the Lipschitz continuity assumption of the decoder (Lemma A.5 we con- ducted perturbation study. The experiment was performed on 100 test samples, with 50 random Gaussian perturbations applied to the latent guidance for each sample. The computation was carried out using full float32 precision to ensure numerical stability. We measured the maximum sensitivity by observing the largest change induced across all token positions in the output logits.
The comprehensive results are presented in Table 7 and the distribution of 5,000 measured sensitivity values is visualized in Figure X The empirical maximum sensitivity was found to be Re 0.263. This small value provides strong evidence that the student model is robust to minor fluctuations in the latent guidance, supporting the stability of our framework.
DISCUSSION OF THEORETICAL ASSUMPTIONS
Our thcoretical analysis is grounded in established information -theoorttitt principles provide formal basis for our framework. To clarify the context our results, it instructive to discuss the key assumptions made when bridging these classical theories with the practical setting of deep learning. key challenge in this bridge, as noted in the Fano-style bound, is that information theory often assumes finite sets, whereas the space of possible reasoning chains (R) is vast. We have explicitly
23

---

Published as a conference paper at ICLR 2026
Table Detailed results from the Lipschitz constant estimation experiment.
Experimental Parameters
Num Samples Tested
100
Num Perturbations per Sample
50
Perturbation Magnitude
0.001
Total Perturbations
5000
Computation Dtype
torch.float2
Sensitivity Statistics
Metric
Value
Estimated (Max)
0.263214
Mean
0, 0,105576
Sid Dev
0.019438
Median (50%)
0.102043
90th Percentile
O, 12822846
95th Percentile
0,141 144
99th Percentile
0.172615
Distribution of Observed Sensitivities (L)
420
100
Senstivity (L)
Figure 8: Distribution of 5,000 observed sensitivity values (L) from the perturbation experiment. The distribution is tightly concentrated around a low mean value (0. red dashed line), with 95% of values falling below 0.11 (green dotted line), confirming the decoder's consistent robustness.
addressed this in Appendix A.9 by employing standard e-net covering number argument, thereby adapting the bound to be meaningful for large-scale spaces. Furthermore, our analysis proceeds under several standard and empirically-supporeed assumptions:
Lipschitz Continuity of the Decoder: Our robustness analysis in Appendix A.5 and A.6 relies on the assumption that the decoder's logits are L-Lipschitz continuous with respect the latent guidance H. This is common assumption in the thcoretical analysis of neural networks. We have provided strong empirical estimates of L in Appendix B.5 that demonstrate the practical stability of our model.
The mutual information values in Appendix B.I are obtained via neural estimators (InfoNCE). These methods are known to provide reliable lower bound on the mutual information. We therefore interpret these values as strong empirical measurements that justify our claims.
Neural Estimation of Mutual Information:
Expressivity the Decoder Model: Our framework implicitly assumes that the decoder model has sufficient capacity accurately translate well-formed cognitive plan into
24

---

Published as a conference paper at ICLR 2026
reasoning chain. To this end, we powerful, ofat-oof-the-r pre- rreitraine models as decoders, making this a reasonable assumption our experimental context.
By pairing this formal analysis with the extensive empirical diagnostics presented in Appendix B we aim to bridge the gap between theory and practice, offering a cohesive and well-supported explanation for the principles underpinning our framework's success.
C ADDITIONAL EXPERIMENTAL RESULTS
C.I FULL RESULTS ON LLAMA2-7B
To further demonstrate the broad applicability of our Latent Guidance framework, we provide the complete experimental results on the LLaMA2-7B base model in Table The results confirm the trend observed in the main paper (Section 4.2) our method consistently outperforms all distillation baselines, achieving a 2.6 percentage point improvement in overall average accuracy over the strongest competitor, NesyCD. This advantage is largely due to superior OOD generalization, where our method leads by 1.3 points the OOD average.
In-Domain
Out-Of-Domain
Methods
Overall Avg.
BBH-test
GSM8K
AGIEval
ARC-E
ARC-C
OOD Avg.
LLaMA2-78 based
Std-CoT
58.1
20.5
23.6
73.4
55.9
51.0
46.3
MT-CooT
46.4
7.5
32.1
70.3
55.7
52.7
42.4
Step-by-step
53.9
8.3
32.4
74.9
60.0
55.8
45.9
KARD (BM25)
59.2
23.5
29.2
70.2
55.4
51.6
47.5
CasCoD
23.6
28.8
72.6
56.7
52.7
48.3
NesyCD
75.5
32.4
33.6
77.5
60.8
57.3
56.0
Ours
76.6
39.8
33.8
79.5
63.1
58.8
58.6
Table 8: Full performance comparison (%) the ILaMA2-7B base model. The OOD Avg. column highlights generalization performance. These results suppiement the main findings in Table Bold indicates the best performance among distillation-base methods.
D DETAILED EXPERIMENTAL SETUP
This section provides the detailed experimental setup referenced in Section 4.1
Datasets To rigorously evaluate performance and generalization, we partition our benchmarks into In-Domain (ID) and Out-of-Domain (OOD) categories.
In-Domain (ID): We use the training splits from GSM8K (Cobbe al. 2021 and BBH (Suzgun et al. 2022). These datasets are used for training all models, and their corresponding test sets are used for ID evaluation.
Out-of-Domain (OOD): To assess generalization, evaluate models wide array of unseen datasets without any task-specific fine-tuning. This includes: AGIEval (Zhong et al. 2023), ARC (Clark et al.) 2018) (easy and challenge sets), Odyssey-Math (Netmind.Al
2024), SVAMP (Patel et al. 2021), and AQuA (Ling et al. 2017).
Qualitative Evaluation: To assess the quality the generated rationales, we use ELI5-
Test (Fan et al. 2019), an OOD benchmark for multi-domain long-form question answering.
Models and Baselines Our experiments utilize open-source models ensure reproducibility. Fren2.5532B-Instruue (Hui et al. (024)) serves as the large model (Implicit Thinker). For the small models (Explicit Executors), we conduct two sets of experiments:
Base Models: We evaluate on several standard base models: LLaMA2-7B Touvron
et al. 2023), L.L.MA-3-BB (Dubey et al. 2024), Qwen2-0.5B, Qwen2-1.5B, and Qwen2.
7B (Team 2024).
Instruct- Tanrre Model: For a deeper analysis, we use Qwen2.55-B-IInsrrue (Hui et al. 2024] powerful small model.
25

---

Published as a conference paper at ICLR 2026
We compare method against comprehensive suite of baselines:
SFT (Supervised Fine-Tuning): A standard baseline where small models are directly fine-tuned (Question, Ground-Truth CoT, Answer) triplets.
Knowledge Distillation (KD): A representative distillation method where the small model is trained on the large model's output logit distribution for the final answer.
Other Contemporary Methods: We also compare against several strong reasoning dis-
tillation baselines, including Std-CoT (Magister et al. 2023), MT-CoT (Li al., (024). Step-by-step (Hsieh et al. 2023). KARD (Kang et al, 2023). CasCoD (Dai et al. 2024).
and NesyCD (Liao al. 2025b).
We focus primarily on distillation-basee baselines as they share goal of transferring reasoning capabilities from large to small models, ensuring a fair and controlled comparison.
Implementation and Evaluation Details All models are fine-tuned using Low-Rank Adaptation (LoRA) (Hu et al., 2022) for parameter efficiency. Quantitative performance is measured by Exact Match (EM) accuracy. For the qualitative evaluation on ELI5-Test, we employ GPT-4o as an auto- mated judge score the correctness, comprehensiveness, and relevance of generated explanations. All experiments utilize consistent hyperparameters to ensure fair comparisons.
E IMPLEMENTATION DETAILS
This section provides key information regarding our experimental setup, including hardware, software, core hyperparameters, and evaluation protocols to ensure reproducibility.
TRAINING ENVIRONMENT
We used the PyTorch framework in conjunction with Hugging Face's transformers and accelerate libraries. All training was performed using bfloat16 mixed-precision.
STAGE 1: IMPLICIT THINKER TRAINING
The large model (Implicit Thinker, Dwen2.5-33BBIInsiie fine-tuning (LoRA) with a rank (r) of 8, alpha of 32, and dropout of 0.05 applied to all linear layers in the attention blocks. The model was trained for 3 epochs using the Paged AdamW (8-bit) optimizer. We set the learning rate to 2e-5 with cosine scheduler, per-device batch size of 4, and 8 gradient accumulation steps.
fine-tuned using parameter-efficient
STAGE 2: EXPLICIT EXECUTOR TRAINING
The small models (Explicit Executors, e.g., Qwen2.5-7B-Inm --nstrruct) were trained for 2 epochs using the same LoRA configuration as in Stage 1. For this stage, used a learning rate of 2e-5 with cosine scheduler and 30 warmup steps. The training was configured with a per-device batch size of 8 gradient accumulation steps, and weight decay of 0.05.
EVALUATION PROTOCOLS
QUALITATIVE EVALUATION WITH GPT-40
For the qualitative evaluation the ELI5-Test dataset (Table 3), which contains reference answers for each question, we used GPT-4o as an automated judge. To ensure grounded and objective assessment, each model's generation evaluated independently. The prompt provided the reference answer alongside a single gener-ggee answer and explicitly instructed the evaluator to score the model based on its adherence the reference.
26

---

Published as a conference paper at ICLR 2026
Prompt
You are an impartial expert evaluator. Your task is to assess the quality of an Al-generated explanation by comparing it against provided reference answer.
Question: /Question from ELI5-Test dataset is inserted here] [The ground-truth answer from the dataset inserted here]
Reference Answer:
Answer to Evaluate: [A single generated explanation from a model is inserted here]
Evaluation Instructions:
Please evaluate the Answer to Evaluate on the following criteria on a scale of 1 to 10, using the Reference Answer as the ground truth.
1. Correctness: How factually accurate and logically sound is the explanation when compared to the Reference Answer? Does it contain information that contradicts the reference? (1 Completely incorrect, 10 = Perfectly aligns with the reference answer). 2. Relevance: How well does the explanation cover the key points of the Reference Answer without including superfluous details? Does it successfully address the question as the reference does? (1 Not relevant at all, 10 Highly relevant and captures the essence of the reference answer). Provide your scores in the following format: Evaluation Score: Correctness: [Score]/10- Relevance: [Score]/10
F EXTENDED DISCUSSION ON METHODOLOGICAL POSITIONING AND DESIGN CHOICES
In this section, we provide a deeper analysis of the methodological positioning of our framework. clarify the rationale behind key hyperparameter choices, and discuss the relationship between our method and concurrent latent reasoning works.
F.1 METHODOLOGICAL POSITIONING: FROM OFFLINE DISTILLATION TO COLLABORATIVE INFERENCE
While our framework shares the ultimate goal of standard knowledge distillation empowering small models with the capabilities of larger ones-it fundamentally differs in its operational paradigm. We propose that our framework functions, to certain extent, as Collaborative Inference system facilitating Large-Small Cognitive Planning Transfer, rather than strictly adhering to the traditional Offfine Distillation paradigm.
Traditional distillation (c.g.. SFT, KD, sequence-level distillation) aims compress all teacher capabilities into the student's parameters, removing the teacher entirely at inference time. While efficient, this often hits a performance ceiling due to the limited parameter capacity of the small model. In contrast, our approach implements division of labor:
The large model acts as an Implicit Thinker, focusing solely on high-level cognitive planning.
The small model acts as an Explicit Executor, focusing on linguistic realization.
By retaining the large model for minimal forward pass (generating only latent guidance), we bridge the gap between static compression and dynamic collaboration. This allows the small model to exceed its inherent parameter limitations by leveraging real-time, high-level cognitive signals.
F.2 DESIGN RATIONALE: THE PERFORMANCE- EFFIICEECY TRADE-OFF (K = 5 vs. K = 10)
The number of latent thought tokens, K, is a critical hyperparameter governing the information bandwidth between the Implicit Thinker and the Explicit Executor. Our ablation studies explored configurations ranging from K
K 10.
We explicitly selected K = 5 as the default configuration to optimize the Cerformance-Efficcein Trade-off, rather than pursuing marginal accuracy gains disproportionate costs. Our rationale two-fold:
1. Diminishing Returns in Accuracy: Increasing K from 5 to 10 yields only a marginal accuracy improvement (e.g.. +0.6% on GSM8K), which does not justify the increased computational overhead.
2. Information Saturation: Our mutual information (MI) analysis demonstrates that the first
5 latent tokens capture approximately 95% of the recoverable information gain regarding
27

---

primary motivation for this work is to improve the performance-to-con trade-off for complex reasoning. In this section, quantify the computational efficiency of our framework by examining the estimated inference overhead (Table D). Our evaluation focuses on the intrinsic computational footprint; the reported metrics derived from average generation throughput (tokens/s) and sequence Published as a conference paper at ICLR 2026
the reasoning plan. Beyond K 5, the marginal information contribution of additional tokens diminishes rapidly.
3. Computational Cost:
10 would nearly double the computational load of the projection layer and the teacher's s forward pass inference cost (generating twice as many latent tokens), undermining the efficiency goals of the framework.
Setting
Thus, K represents the "swect spot" where the system achieves robust reasoning transfer while maintaining high inference speed.
G ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
In this section, we present comprehensive additional experiments conducted to validate the scalability, efficiency, and structural integrity of our Latent Guidance framework.
G.1 EVALUATING THE PERFORMANCE- EFFICIENCY BALANCE
length, representing the theoretical efficiency limit by focusing on computational savings from reduced token generation.
The results demonstrate substantial improvement in reasoning density. On the challenging ARC-C benchmark, our method (32B large + 7B small model) achieves an accuracy 90.9%. This represents remarkable 13.9 percentage point accuracy gain over the 7B small model baseline (90.9% vs. 77.0%), while significantly reducing the computational overhead. Notably, the estimated processing time is reduced to 3.6s, which is approximately half of the 7B baseline (7.2s) and nearly 4 times less than the 32B teacher (15.2s), all while retaining competitive accuracy. This gain is primarily driven by the generation of more concise reasoming chains, analyzed in Sec 4.3, which effectively minimizes redundant linguistic realization. This favorable balance of high accuracy and low sequence complexity is consistent across all tested datasets. These findings indicate that Latent Guidance is environments, achieving
practical paradigm for deploying advanced reasoning in resource-constraing near-teacher performance with significantly minimized computational burden. We emphasize that the reported latency is a conservative estimate derived from token counts and throughput, intended to compare relative efficiency trends rather than absolute wall-clock performance.
Table 9: Inference efficiency comparison on multiple datasets. Ours latency denotes the total end- to-end time (Teacher's 's forward pass + Student's generation). For a consistent evaluation, latency is calculated based on total token counts and the average inference speed (tokens/s) of the respective models, assuming negligible transmission and communication overhead between the teacher and student.
Dataset
Method
Model Configuration
Accuracy (% f
Tokens
Lateney
Large Model
84.1
265.8
14.4
GSM8K
Ours
32B- 7B-Inst
80.5
128.9
3.5
Small Model
Qwwn225-77-Innsruct
75.4
291.3
7.2
Large Model
Owen2.5-32B-Instme
95.5
280.7
15.2
ARC-C
Ours
32B-Inst 7B-Inst
90.9
138.3
3.6
Small Model
Qwen2.5-7B-Instree
77.0
293.8
7.2
Large Model
Owen2.5-32BBImm
86.2
178.8
9.8
SVAMP
Ours
32B- 7B-Inst
85.9
68.6
2.2
Small Model
Qwenn255-7-IInsrreet
79.1
216.2
5.3
G.2 ANALYSIS OF THE PROJECTION LAYER INFORMATION BOTTLENECK
To verify whether the projection layer Z (Hguideeee introduces harmful information bottleneck. conducted an ablation study the projection architecture and measured the Mutual Information (MI) Retention Ratio: Ratio T(R; Z|Q)/T(RL gusddance IQ)-
Table 10 demonstrates that default design (2-layer MI.P) achieves optimal balance.
28

---

Published as a conference paper at ICLR 2026
A single linear layer (I-layer) causes significant information loss (71% retention), degrading accuracy.
The 2-layer MLP preserves 93% of the planning information. Increasing depth further (3-layer) yields diminishing returns in MI retention (94%) without meaningful accuracy gains.
Table 10: Impact of Projection Layer Architecture on Information Retention and Accuracy.
Projection Arch.
Est. MI (nats)
Retention Ratio
GSMSK Acc (%)
Reference (H ((guddnee))
3.10
100%
1-layer MLP (Linear)
2.21
71%
79.0
2- 2-layer MLP (Ours)
2.88
93% %
80.5
3-layer MLP
2.91
94%
80.4
H USE OF LARGE LANGUAGE MODELS IN MANUSCRIPT PREPARATION
accordance with ICLR 2026 policy, we disclose the use of Large Language Models (LLMs) as an assistive tool in the preparation of this manuscript. The primary application of LLMs was to aid in improving the clarity and quality of the writing.
Our process involved using an LLM to perform the following specific tasks:
Grammar and Spelling Correction: Identifying and correcting grammatical errors and spelling mistakes.
Clarity and Readability Enhancement: Rephrasing sentences and suggesting alternative phrasings to improve the overall readability and flow of the text.
Conciseness: Assisting in shortening sentences and paragraphs to make the writing more direct and concise.
The core scientific contributions, analyses, and claims presented in this paper are the work of the human authors. have ensured that the use of LLMs in the writing process was conducted responsibly and in line with academic and ethical standards.
29


Reviews:
[{"review_id": "QSYKcoBhIa", "reviewer": "Reviewer_8Hqv", "summary": "This paper proposes Latent Guidance, a framework that combines large and small language models to perform complex reasoning for improved cost-effectiveness. The approach uses a two-stage training process: in the first stage, the large model is trained to compress (via auto-encoding) the reasoning steps into some latent vectors. In stage two, the small model (augmented with some learnable projection layers) is trained to generate the reasoning steps conditioned on these latent vectors. Experiments across 8 reasoning benchmarks show that a 7B model achieves up to 13.9% accuracy gains over its standalone baseline while being 2x faster, and 4x faster than the 32B teacher model.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a well-motivated framework addressing the inefficiency of tight coupling between high-level cognitive planning and low-level text generation in current reasoning systems.\n\n- The experimental evaluation covers multiple model scales (0.5B to 8B) and diverse reasoning benchmarks, with ablation studies demonstrating the necessity of key design choices. Overall, the generalization of the proposed framework to OOD tasks is impressive. The appendices provide extensive diagnostics such as MI estimation and capacity scaling experiments, that attempt to validate theoretical claims.", "weaknesses": "- The paper seems to mischaracterize the proposed method as \"distillation\" when it's closer to an autoencoder with active inference-time involvement of the large model. This makes comparisons with true distillation baselines (SFT, KD, NesyCD) fundamentally unfair, as those methods train the small model to reason independently without the large model at inference time, while this approach requires the 32B model to generate latent guidance for every test query. Relatedly, it's good to report standalone 32B performance for reference.\n\n- The results that the proposed method produces substantially more concise reasoning chains than other baselines is quite difficult to understand. If the small model is trained to reconstruct the teacher's reasoning, the lengths should be comparable to other distillation methods - how could they be dramatically shorter? In lines 326-329 the authors explain this via: *\"This demonstrates that the high-level cognitive plan\nenables the small model to generate more focused and generalizable reasoning paths, avoiding the verbose, exploratory steps often seen in less-guided generation on unseen problems. The model learns to execute a direct strategy rather than overfitting to the stylistic artifacts of the training data.\"* which I'm not convinced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NHnGbmWXie", "reviewer": "Reviewer_hvKj", "summary": "The paper proposes Latent Guidance, a framework that decouples cognitive planning from linguistic realization to empower small LLMs. A large model compresses its solution strategy into latent guidance vectors, which a small model then uses to generate concise reasoning chains. The dual-loss training (task loss + reconstruction loss) ensures the latent vectors faithfully encode the reasoning process.", "questions": "How does performance scale with even larger gaps between teacher and student sizes, say 70B to 2B?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The theoretical foundation using mutual information and Fano bounds provides solid justification for the approach. I'm impressed by the consistent OOD improvements across diverse models from 0.5B to 8B parameters. The comprehensive diagnostics in Appendix B, especially the MI estimation converging to 3.10 nats, strengthen the empirical validation. The emergence of distinct reasoning strategy clusters without explicit supervision is fascinating.", "weaknesses": "THe presentation could be improved. Also, not tesetd on larger models. \n\nThe framework requires storing latent guidance for all training examples which could be memory intensive for larger datasets. While the projection layer design is ablated, I think more analysis on the information bottleneck it creates would strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksovWo7H4d", "reviewer": "Reviewer_gdf4", "summary": "The paper proposes a novel framework named \"Latent Guidance\" (LG) designed to empower small language models (SLMs) with the complex reasoning abilities of large language models (LLMs). The authors term this new paradigm \"Cognitive Distillation,\" which decouples high-level cognitive planning from low-level linguistic realization. In this framework, a large \"Implicit Thinker\" model performs the complex planning and compresses its strategy into a set of latent vectors ($H_{guidance}$). A small \"Explicit Executor\" model then receives this guidance and generates the final textual reasoning chain. The framework is optimized via a dual-loss training objective.\n\nThe authors prove their method's effectiveness and efficiency via both empirical experiments and theoretical analysis. Comprehensive ablations like the probing experiments of the in-depth analysis of the latent cognitive plan indicates the small model executes a structured, abstract plan, rather than just learning feature correlations.", "questions": "1. About the latency in Table 6, does the reported time for \"Ours\" represent the total end-to-end latency ($T_{total} = T_{step\\_{(32B\\_{fwd})}} + T_{step\\_{(7B\\_{gen})}}$)?\n\n2. See weakness 1\n\n3. Have the authors considered an additional baseline that involves providing the full text of the 32B teacher's reasoning chain ($R_{LLM}$) directly into the 7B small model's prompt at inference time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The manuscript is well-organized and easy to follow.\n\n2. The proposed idea of cognitive distillation is sound.  The information theoretical analysis is solid.\n\n3. The appendix includes almost all theoretical and experimental details like the training hyperparameters and computational resources.\n\n4. The paper goes beyond accuracy tables to explain the inner mechanisms. The t-SNE visualizations and quantitative probing tasks provide compelling evidence that the $H_{guidance}$ vectors are not just feature containers but successfully encode high-level, abstract reasoning strategies (e.g., \"Sequential Multi-Step Reasoning\").", "weaknesses": "1. There is a contradiction between the core methodological design and the empirical results. \n The reconstruction loss  and  SLM loss explicitly train the models to reconstruct the original, full-length reasoning chain.\nHowever, the paper claims a key benefit is that the SLM can generate substantially more concise reasoning chains (e.g., 128.9 tokens vs. SFT's 235.4). The paper cannot train the model to do one thing (reconstruct the full 235-token chain) and then claim its success for doing the opposite (generating a 129-token chain) without a clear explanation. Is conciseness an emergent property? Or is the method description in Eq 6 inaccurate?\n\n2. The paper calls L_recon the \"cornerstone\" of the method. However, the ablation study (Table 4) shows that removing it causes only a minor performance drop (e.g., -2.3% on GSM8K). This small drop does not support its \"cornerstone\" status for in-domain performance.\n\n\n3. Line 152, Line 157: the author should use correct right single quote.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "na7ih6WtdM", "reviewer": "Reviewer_bGgN", "summary": "This paper addresses the high computational cost of complex reasoning in LLMs. The authors argue this inefficiency stems from the tight coupling of \"high-level cognitive planning\" (strategy) and \"low-level linguistic realization\" (text generation). To solve this, they propose \"Latent Guidance,\" a collaborative framework that decouples these processes. The framework uses two models: an **Implicit Thinker** (a large LLM, e.g., Qwen2.5-32B) processes the input question and special \"thought tokens\" to produce a set of compact \"latent guidance vectors\" ($H_{guidance}$); an **Explicit Executor** (a small LLM, e.g., Qwen2.5-7B) receives both the question and $H_{guidance}$ to autoregressively generate the final reasoning chain and answer. The core of this \"Cognitive Distillation\" is a dual-loss training objective for the large model. It includes a task loss for the correct answer, and, crucially, a reconstruction loss $\\mathcal{L}\\_{recon}$ that compels the latent vectors $H_{guidance}$ to contain enough information to reconstruct the entire original, detailed reasoning chain $R$. The authors provide a theoretical analysis showing that this $\\mathcal{L}\\_{recon}$ maximizes the mutual information between the plan and the reasoning chain. Experiments across eight benchmarks show the 7B model, when guided, gains up to 13.9% in accuracy, runs 2x faster than its 7B baseline, and is 4x faster than the 32B teacher, while also showing superior OOD generalization.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.  The paper's primary strength is its novel \"Cognitive Distillation\" framework. The conceptual move to decouple planning from realization is intellectually appealing. It reframes distillation from mimicking *outcomes* (final text) to transferring a *process* (a high-level plan), which is a contribution to the field.\n\n2.  The $\\mathcal{L}\\_{recon}$ loss is not merely empirical; it is rigorously justified with an information-theoretic analysis in Appendix A. The derivation showing that minimizing $\\mathcal{L}\\_{recon}$ is equivalent to maximizing the mutual information $I(R;H_{guidance}|Q)$ provides a solid theoretical foundation for why the latent vectors should be information-rich.\n\n3. The method achieves strong results, particularly in OOD generalization (Table 1). The fact that it consistently outperforms strong distillation baselines (like NesyCD) on unseen datasets lends weight to the hypothesis that distilling a \"strategy\" is more robust and generalizable than distilling text.\n\n4.  The results in Table 6 are noteworthy. Achieving a 2x speedup over a 7B baseline while simultaneously boosting its accuracy by +13.9% (on ARC-C) represents a clear Pareto improvement and a practical achievement.", "weaknesses": "1. The paper's central thesis—that $H_{guidance}$ represents an *abstract, high-level \"cognitive plan\"* —is not sufficiently supported by the provided evidence.\n    *   **t-SNE Visualization (Fig. 3):** This evidence is weak. t-SNE is known for creating illusory clusters. The observed clusters, which are given post-hoc labels like \"Sequential Multi-Step Reasoning,\" are far more likely to be simple high-dimensional clusters of \"problem types\" (e.g., all two-step addition problems) rather than proof of abstract, disembodied \"strategies\".\n    *   **Probing Analysis (Table 5):** This analysis is tautological. The $\\mathcal{L}\\_{recon}$ loss *explicitly trains* $H_{guidance}$ to be able to reconstruct the full text. Therefore, \"proving\" that a probe can predict properties of that text (like \"Reasoning Step Count\" or \"Mathematical Operator\")  from $H_{guidance}$ is not evidence of a cognitive plan; it is merely a sanity check that the loss function *worked*.\n\n2. The ablation in Table 4 shows that using $K=10$ thought tokens yields better performance on GSM8K (81.1%) than the $K=5$ (80.5%) used for the main results. This suggests the paper may not be reporting its strongest possible configuration.\n\n3. The qualitative analysis in Table 3 relies entirely on GPT-4o as an evaluator (Appendix E). This is not a sufficient substitute for human evaluation (or stronger models like GPT-5), especially when judging nuanced qualities like \"Relevance.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UjxTWJx9Pg", "reviewer": "Reviewer_bKFV", "summary": "A new method is introduced to use latent reasoning from a large model (implicit thinker) to guide a smaller model to generate conventional CoT conditional on the latents, in order to improve the smaller model (which is trained under the inference setting).", "questions": "I don’t know all those baselines well, and they aren’t described in detail (MT-CoT, Step-by-step,  KARD, CasCoD, NesyCD) – although I think MT-CoT would be straight training on the larger models CoT? Did you try on-policy distillation  https://arxiv.org/abs/2306.13649 from the larger model? Or, what about RL of the small model after distillation? Those two variants sounds like some of the strongest standard small model approaches I would try first, and I'm not sure they are compared to?\n\nFor which of those baselines do you have to keep around the large model for inference time, I’m assuming many do not, which yours has to, right? So -- this looks like a disadvantage of yours just in terms of complexity...? (That is, the gains would have to be substantial in that case I guess, given the extra complexity).", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The method is clearly presented and, to the best of my knowledge, different to other methods -- although sharing many commonalities (see below). Results improve over a number of small model distillation baselines (although see questions below).", "weaknesses": "While to the best of my knowledge the method is different to others -- it is a mix of some existing things (which can be ok). Learning latent vectors by making them predictive of CoT is a pretty standard approach in continuous latent reasoning approaches, e.g. iCoT and COCONUT. In those cases, I believe they show that better latent vectors can be learnt by iteratively removing the CoT to be predicted to compress more of the information, although this is more complex than your approach -- but could it be better?\n\nOther methods also aim to mix latents with CoT text tokens, such as LightThinker (https://arxiv.org/abs/2502.15589) which I believe performs pretty well without needing to keep around a larger and smaller model, I think this work is not mentioned/compared?\n\nOverall, the need to keep both a large and a small model around for inference seems complex, and I think would need to give significant wins to make it worth on it. On the other hand, the counter argument is that frontier models are often these days deployed as mixtures of experts, so this isn't too far off something like that either..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)