CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
Authors: Li Li
Ziyi Wang
Yongliang Wu
Jianfei Cai
Xu Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Chain-of-Thought (CoT); Task Vectors; Model Steering; Large Language Models (LLMs)
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Chain-of-Thought (CoT) prompting has emerged as a powerful approach to enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing implementations, such as in-context learning and fine-tuning, remain costly and inefficient. To improve CoT reasoning at a lower cost, and inspired by the task vector paradigm, we introduce CoT Vectors, compact representations that encode task-general, multi-step reasoning knowledge. Through experiments with Extracted CoT Vectors, we observe pronounced layer-wise instability, manifesting as a U-shaped performance curve that reflects a systematic three-stage reasoning process in LLMs. To address this limitation, we propose Learnable CoT Vectors, optimized under a teacher–student framework to provide more stable and robust guidance. Extensive evaluations across diverse benchmarks and models demonstrate that CoT Vectors not only outperform existing baselines but also achieve performance comparable to parameter-efficient fine-tuning methods, while requiring fewer trainable parameters. Moreover, by treating CoT Vectors as a probe, we uncover how their effectiveness varies due to latent space structure, information density, acquisition mechanisms, and pre-training differences, offering new insights into the functional organization of multi-step reasoning in LLMs. The source code will be released.

Published as a conference paper at ICLR 2026
COT VECTORS: TRANSFERRING AND PROBING THE REASONING MECHANISMS OF LLMs
Li Ziyi Wang' Yongliang Wu' Jianfei Cai²
Xu Yang'
1 School of Computer Science & Engineering, Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China 2 Data Science & AI Department at Faculty of IT, Monash University, Australia l lilyliie seu. edu. en, xuyang-palm@s seu. edu.cn
ABSTRACT
Chain-of-Thougee (CoT) prompting has emerged powerful approach to hancing the reasoning capabilities of Large Language Models (LLMs). However, existing implementations, such as in-context learning and fine-tuning, remain costly and inefficient. To improve COT reasoning at a lower cost, and inspired by the task vector paradigm, we introduce CoT Vectors, compact representations that encode task-general, multi-step reasoning knowledge. Through experiments with Extracted CoT Vectors, we observe pronounced layer-wise instability, man- ifesting as U-shaped performance curve that reflects a systematic three-stage reasoning process in LLMs. To address this limitation, we propose Learnable CoT Vectors, optimized under teacher-student framework to provide more sta- ble and robust guidance. Extensive evaluations across diverse benchmarks and models demonstrate that CoT Vectors not only outperform existing baselines but also achieve performance comparable to parameter-effice while requiring fewer trainable parameters. Moreover, by treating CoT Vectors a probe, we uncover how their effectiveness varies due to latent space structure, information density, acquisition mechanisms, and pre-training differences, offer- ing new insights into the functional organization of multi-step reasoning in LLMs. The source code will be released.
tne-ttnnng methods,
INTRODUCTION
Chain-of-Though (CoT) prompting (Wei et al. 2022 has emerged as a powerful technique to un- lock the complex reasoning capabilities of Large Language Models (LLMs) (Zhao al.
2023) By
reasoning step-by-step. CoT enables models to decompose problems, mimic human- like logic, and
improve performance several challenging tasks (Imani et. 2023 Huang & Chang 2022 How-
ever, how to effectively harness the power of CoT in practice remains an open problem. Existing
approaches generally fall into categories: (1) In-Context Learning (ICL.) (Brown et al., 2020)
with few-shot CoT examples, which can enhance reasoning, but it requires longer prompts and slows inference; (2) Fine-tuning LLMs (Ziegler al., 2019) with CoT-annotated data, which demands large amounts of high-quality reasoning traces and computational resources, while often yielding only limited improvements for models that already equipped with CoT abilities. These challenges prompt critical question: can we transfer the essence of CoT, i.e., the general "problem-solving mindset" of a task, into LLMs in a way that is compact, reusable, and efficient?
Recent advances in Task Vectors (Ilharco et al. 2022) offer promising direction. Task-specific knowledge can be distilled into compact vector, often represented as the difference in activa- tions (Hendel et al.) 2023 Todd et al., 2023 Liu et al. 2023) parameters (Ortiz-Jimenez et al.) 2023b, Li et al.J 2025) between fine-tuned and base models. Such vectors can steer model be- havior toward adaptation. However, current applications of Task Vectors have been limited to simple adaptation scenarios, leaving it unclear whether this paradigm can be extended to complex multi-step reasoning.
tasks without modifying model weights, thereby enabling parameter-efficie
"Corresoonding author.

---

Published as a conference paper at ICLR 2026
Frozen
Trainable
Question: Carmen outbids three single bidders for $200 desk with increments-whie does pay?
Question: Carmen outbids three single bidders for a $200 desk with increments- pay?
Question:
Carmen outhids three
single bidders for $200 desk with $50 increments- what does she pay?
Support Set
Support Set
o0
Q+CoT
00
LLM
CoT Vector
LLM
4
LLM
1
CoT Thinking:
Three rivals add 3 $50-$1 50, Carmen counters with $50=$150 so the desk costs 11:00-$150+$50-1500
CoT Thinking:
Carmen makes 1 opening plus counter-bids, while the 3 rivals add 3 more, totaling bids. (7 1) $50 $200 = $500, pays $500.
CoT Thinking: Carmen and three rivals each bid once, pushing the price up: $200 $250 $300 $350 $12300
The is: 12300
X
The answer is: 500
The answer is: 500
(a) Complex Reasoning
(b) Fine-tuning
(c) CoT Vector
Figure I: Overview of our approach. (a) Standard LLM may struggle to produce a correct reasoning
chain for complex problem. (b) Conventional fine-tuning adapt the model to such tasks by training on a support set, but requires updating model parameters, incurring high computational cost. (c) Our proposed CoT Vector leverages the support obtain a compact reasoning representation, which can be injected into the forward process of model to guide reasoning efficiently.
Through mathematical analysis, we observe that the effect CoT can be formalized as a consistent shift in the internal activations of model (He et al.) 202Ip, suggesting that extending task vectors to reasoning both feasible and promising. In this work, we introduce CoT Vectors, task-general reasoning representations that adapt the task vector framework CoT reasoning. CoT Vectors compactly encode the critical reasoning knowledge from support set of (Question, CoT, Answer) triplets, and can be directly injected into the forward process during inference. This approach not only enables portable reasoning enhancement without costly retraining or significant inference over- head, but also offers a new probe into how LLMs internalize and apply COT. Specifically, we begin with Extracted CoT Vectors, directly derived from activation differences between reasoning and non-reasoning traces, line with the traditional task vector approach NLP. Our studies reveal that Extracted CoT Vectors are effective but highly unstable across layers, with a striking U-shaped performance curve. This pattern suggests a systematic functional organization in LLMs, which characterize as three-stage reasoning process spanning perception, reasoning, and expression. Shallow and deep layers show relatively consistent representations, whereas middle layers contain highly variable, sample-specific structures that cause extracted vectors to fail.
To improve robustness, we introduce Learnabl CoT Vectors, optimized via a teacher- student frame- work. Mathematically inspired by the additive shift formalization of COT, our method distills more robust and generalizable reasoning signal into a single, reusable vector. By actively learning reason- ing knowledge rather than passively averaging activations, it achieves greater stability and stronger performance, overcoming the layer-wise volatility of extracted vectors. We conduct comprehen- sive evaluation across various models and benchmarks, comparing extracted and learnable vectors against baselines. Our analyses further elucidate the sources of variability in CoT Vector effective- ness, highlighting how differences acquisition mechanisms and -edel-specific latent structures shaped during pre-training impact reasoning performance. This perspective offers valuable lens for understanding how LLMs organize and apply multi-step reasoning internally.
Overall, we summarize the main contributions of this work as follows:
We introduce CoT Vectors, extending task vectors to multi-step reasoning. Experiments with traditional extracted vectors uncover their layer-wise instability, which in turn reveals systematic three-stage reasoning process in L.LMs.
2

---

Published as a conference paper at ICLR 2026
To address the limitations of extracted vectors, we propose novel Learnable CoT Vectors, op- timized via a sucher--ttuent framework, which provide more robust, stable, and task-general reasoning representations.
We conduct comprehensive evaluation across benchmarks and models. Using CoT Vectors as a probe, analyze their variability from multiple perspectives, including latent space structure, in- formation density, acquisition mechanisms, and model pre-training differences, thereby providing new insights into the mechanistic organization of reasoning in LLMs.
2 RELATED WORK
Enhancing Chain-of- Reasoning in LLMs. Our work builds upon the foundational paradigm of CoT prompting (Wei et al., 2022). which significantly improves reasoning perfor- mance by eliciting step-by-step rationales. Numerous subsequent efforts have refined this idea through improved prompting strategies (Kojima et al. [022 Khot et al. 2022), search-base
rea- soning frameworks (Yao et al. | (023). program-aide -ided execution (Gao al., 2023). and iterative self- refinement (Madaan et al. (023). Another common strategy to improve performance is fine-tuning. Typical approaches include Supervised Fine-tuning (SFT) on (Question, CoT, Answer) triplets, as well as reinforcement learning techniques such as RLHF (Ouyang et al.] (022). PPO (Schulman) ct al. 2017). and GRPO (Rafailov et al. 2023). While effective, these techniques often demand substantial amounts of high-quality rationales. significant computational resources for training or alignment (Zao et al. 2025), making them prohibitively expensive relative to the incremental gains.
A parallel line of work seeks to compress explicit reasoning steps into fewer, or invisible, latent representations, often referred to as Implicit CoT (Deng et al.] 2023 2024). These methods aim internalize the reasoning process improve efficiency and performance. Some approaches modify model architecture (Geiping et al. 2025) or use placeholder tokens in prompts (Pfau et al., 2024) to extend latent reasoning depth, effectively condensing multi-step thinking into compressed latent transitions that lead directly to answers. However, these methods typically require specialized model modifications and carefully engineered training regimes involving intensive post-training of model parameters (Hao et al., 2024, (hhen et al.) [025 Cheng & Van Durme 2024), which demand substantial resources while often delivering only modest gains. In contrast, our approach keeps the model architecture untouched: we distill reasoning into an external, plug-and-play CoT Vectors that
can be acquired and applied rapidly for new tasks, combining flexibility with strong performance.
Task vectors. Task vectors (Ilharco et al., 2022) have emerged compact representation of task- specific knowledge, typically obtained either by computing weight differences between fine-tuned and pre-trained models (Ortiz-Jimenc) et al.) 2023a Li ct al.] 2025), by capturing activation differences induced by distinct input prompts (Liu et al. 2023) Todd et al. 2023 Hendel et al, 2023). These vectors not only enable parameter-efficient task transfer but have also been leveraged to provide preliminary insights into the internal mechanisms of LLMs (Yang et al. 2025). However, existing studies largely focus on relatively simple scenarios such classifications or in-context learning, leaving the application of task vectors to complex multi-step reasoning underexplored. Several recent preliminary study (Azizi al., 2025 Tang et al. 2025 Zhang & Viteri) have tenta- tively explored steering vectors in CoT. suggesting the ieasibility of the paradigm. However, Azizi al. 2025) focuses on compressing CoT chains, while Tang et al. (2025) aims to stimulate longer reasoning trajectories, both focusing on controlling CoT generation rather than capturing task- general reasoning pattern. Meanwhile, Zhang & Viteri remains limited to conventional extraction techniques and offered only surface-level analysis. Our work moves substantially beyond this early exploration. Instead of relying solely on a basic extraction method, we introduce a novel learnable mechanism that actively optimizes CoT Vectors for better generalization and performance. Further- more, our study provides a comprehensive analysis absent from prior work.
METHODOLOGY
We first formalize the concept of CoT Vectors in Section 3.1
deriving it from the mechanistic
effect of CoT reasoning the model's attention outputs. This formulation not only establishes the feasibility of our approach but also provides the guiding principle for the subsequent development. Building on this, develop our two practical frameworks in Section 3.2 for acquiring CoT Vectors:
3

---

Published as a conference paper at ICLR 2026
Frozen
Trainable
Bap
Layer
Mutt-head
Layer
Muil-hhaad
(A (Q.
Layer
Mulu head
(COT.A (Q0)
Lyp Line
(CoT] [A]
(Q) IA]
Seacher (CaT] (A]
Student (C0) [A]
Text Question (q)
(>) Extracted CoT
Learnable CoT
(c) Vector Injoction
Figure 2: Methods for acquiring and applying COT Vectors. (a) Extracted CoT Vector is obtained by recording the activation gap at the k-th layer between inputs with and without COT. (b) Learnable CoT Vector is inserted into the k-th layer activations of student sequence without CoT,, and trained by aligning the student's final answer-token hidden states with those of a teacher sequence that includes COT. (c) At test time, CoT Vector is added to the activations at the k-th layer during each forward pass of auto-regressive generation, guiding the reasoning process.
a non-parametric extraction method and novel parametric learning-based method, along with how efficiently integrate the resulting vectors during inference steer the model's reasoning.
3.1 CONCEPTUALIZATION AND FORMALIZATION OF COT VECTOR
The effectiveness of CoT prompting highlights that inserting explicit reasoning sequence between the input question Q and the final answer A significantly enhances the model's reasoning capability. Our objective is to capture and quantify the effect of this reasoning process within the model.
In transformer-based language models, information flow is governed by self-attention. He et al. (02T) suggest that the effect of imput prefixes can be understood as shifts in the attention out- puts. In the context of CoT,, the CoT sequence can be viewed the generation of answer tokens. When reasoning with CoT, the generation of an answer token depends not only on the question tokens but also the intermediate CoT tokens. For each an- swer token a A, the single-head self-attention with and without the CoT sequence denoted as SA [Kq, Kc, KA,, (Vo, Vo, VAl) and Sa [KQ, KA), [Vq, Val) respectively, where the sub- scripts Q. C, and refer to the question, CoT, and answer. We then derive the following equation:
specialized prefix that modulates
SA(a, KQ2 Kc, KA), (e. Ve, VA)) = SA(a, K,), [Vo. VA))
(1)
Standard atteution
+ p (SA(a, Kcl. [Vo)) SA [Kq. KAl, [VQ, Val))
CoT shift
(2)
The introduction of CoT induces additional term in the attention output, whose influence is quan- tified by scalar coefficient y (see the supplementary material for the full derivation). This additional contribution reflects precisely the knowledge injected by the CoT sequence. We formalize this effect as CoT Shift, and denote the corresponding representation as the CoT Vector Uco- Accordingly, Equation can be reformulated as
SA(a, KQ. Kc. KA), VQ, Ve. VA)) = SA(a, KQ. KA), (Vo, Val) + ju Ucor
(3)
The CoT Vector serves as a compact representation of the reasoning knowledge compressed from the CoT sequence. We hypothesize that, for tasks of the same type, the CoT Vectors derived from individual examples reside a continuous semantic space. The centroid of this space, which we call problem,
the task task-general CoT Vector, encodes the shared reasoning strategy for that task. For injecting Veat into the model's 's forward pass, which reversely applics the Equation 3 can effectively guide the model toward appropriate reasoning trajectory and thereby improves task accuracy.
4

---

Published as a conference paper at ICLR 2026
3.2 TASK-GENERAL COT VECTORS
To leverage the advantages of task-genera CoT Vectors, we first acquire them from support set D. We propose two approaches for this acquisition: a traditional extraction-bars based method and novel parametric learning-based method. Once obtained, the task-general vector is injected into the model's 8 forward pass during inference to steer reasoning process.
3.2.1 EXTRACTED COT VECTORS
Given a support set of pairs (Q,A) and triplets (Q, CoT, A), we define the Extracted CoT Vector (!)
CoT the difference model activations of answer token a layer between these inputs, as shown in Figure (a):
Σ
(a) - a (an-Cor(a))
(4)
CoT
A
aEA
where a (a) is the hidden state of answer token a at layer for the input and |A| denotes the total number answer tokens.
For each instance (Qi, CoT, A.), compute instance-specifie CoT Vector UÇT,, A task- general Extracted CoT Vector eg then obtained by averaging across all N support instances:
ÜF
(5)
3.2.2 LEARNABLE COT VECTORS
Beyond extraction, we propose a novel parametric method that learns task-general CoT Vector through gradient-based optimization. As depicted in Figure 2 (b), ot is initialized as learnable parameters, added as a shift to the hidden state at a specific layer, and optimized on the support set D encode generalized reasoning knowledge. We adopt instance (Qi, A:), the teacher path processes the full triplet with frozen model parameters, providing the supervisory signal. The student path, in contrast, only processes (Qi, At). while en is injected to compensate for the missing CoT sequence. Through this process, गy distills essential reasoning signals from the teacher into a compact, transferable representation.
seacher-studens framework. For each
Throughout optimization, all original LLM parameters are kept frozen; only Ez are updated. The training objective combines two components: Prediction loss (Lcx) the cross-entropy loss on the student's predicted answer tokens, ensuring that the injected vector guides the model toward correct outputs. Representation alignment loss (CAliga) is the mean KL loss between hidden states of teacher and student paths at the answer tokens, enforcing alignment of internal reasoning representations. The final objective is:
C = CAlign + X LCB
(6)
where À is a hyperparameter balancing the two terms.
3.2.3 INTEGRATING THE CoT VECTOR TO REASONING
At inference time, given a new question, task-general CoT Vector öosT obtained from the support set at specific layer I, is then injected into the model at the same layer during every forward pass of CoT thinking, as shown in Figure? 2 (c).
()
(I) CoT
For Extracted CoT vectors, μ is an explicitly defined constant scaling factor. For Learnable CoT Vectors, however, μ is effectively interalized-since गz is optimized end-to-end, the scaling factor absorbed into the vector during training rather than being maintained as a separate constant. This integration incurs almost additional overhead: it does not increase the input context length, and the runtime cost is negligible since the operation reduces to a simple vector addition. As a result, approach provides extremely efficient mechanism for enhancing reasoning in L.LMs.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
Next, we explore In this section, we first outline the setup and implementation details (Section 4.1 the adaptation of task vectors to multi-step reasoning in LLMs by introducing and analyzing CoT This investigation extends beyond mere performance evaluation, utilizing
Vectors (Section CoT Vectors as a tool to probe the underlying functional mechanisms of reasoning within LLMs.
4.1 SETUP AND IMPLEMENTATIO DETAILS
Models and Datasets. We conduct experiments on two open-source LLMs: Qwen2.5-Matt 7B (Yang et al. 2024p, a model fine-tuned for mathematical reasoning tasks, and I.LaMA33 1-8B- Instruct (Grattafiori et al. 2024), an instruction-tuned model with broad domain coverage. We use three datasets with fround-truth CoT sequences: GSMSK (Cobbe et al. 2021). MATH (Hendrycks) et al. 2024). MMLU-Pro (Wang et al., 2024), CommonsenseQQ (Talmor et al. 2019). and Strate- gyQA (Geva et al. 202T). GSMSK and MATH cover mathematical reasoning, MMLU-Pro spans diverse subject domains, and CommonsenseeAA/Staatee A/StrategyyA focus natural-languaat logical rea- soning. For MATH, we divide the dataset into two subsets based on difficulty: MATH-Easy (levels 1-3) and MATH-Hard (levels 4-5), ensuring balanced difficulty in the support set. We sample 3,000 examples from GSM8K and the two MATH subsets as the support set. For MML.UPro, we use all 70 ground-truth annotated problems as the support set. For CommonsenseQA and StrategyQA, no official CoT traces are provided, so we rely on model-generated CoT sequences instead.
Implementation Details. We use standard zero-shot CoT prompting as baseline, where the model instructed to "think step by step." For CoT Vectors, both extracted and learnable variants are implemented following the procedures described in Section 3 For extracted vectors, the scaling factor y is fixed at 1.0. For learnable vectors, the loss balancing factor À in Eq. 6 is set to 0.5. We select LoRA (Hu et al., 2022) as the representative parameter-efffiient fine-tuning baseline, where LoRA adapters are trained on (Q, CoT, A) triplets. Following common practice, we apply LoRA to the projection matrices Wo. WK. Wv, and Wo in all attention layers.
4.2 EXPLORING COT VECTORS AND THE MECHANISM OF REASONING
This section explores the adaptation of task vectors to multi-step reasoning via CoT Vectors. Our analysis begins with examining COT Vectors obtained by the traditional extraction method, which prove effective but highly unstable performance across layers. This instability reveals consistent layer-wise patterns, uncovering three-stage reasoning process in LLMs. Building upon these in- sights, we introduce Learnable CoT Vectors that achieve greater stability and stronger performance. We comprehensively evaluate both CoT Vectors against baseline and LoRA across two models and four benchmarks, and interpret performance variations through analyses latent space structure, information density, acquisition mechanisms, and pre-training differences. Together, these studies not only extend the task vector framework to reasoning, but also reveal new perspectives on the underlying mechanisms of LLM reasoning.
4.2.1 THE THREE- STAGE REASONING PROCESS
To assess whether task vectors can be extended to reasoning, we first explore the applicability of conventional extracted task vectors in CoT setting. Table shows that Extracted CoT Vectors are indeed effective, improving over the baseline by an average of 2.4 and 1.1 points on the two models respectively and demonstrating the feasibility of task vectors in reasoning: however, their effective- ness is highly unstable across different layers (Figure
B (a)) with the layer-wise average performance even falling below the baseline. Interestingly, this instability follows a non-random pattern. We ob- serve sawtooth U-shaped pattern: despite the fluctuations, the overall trend shows that performance enhancements when vectors are injected into either the shallow and deep layers, whereas injections into the middle layers yield minimal gains or even degrade performance. This contrasts with prior task vector research on simpler tasks (Todd et al. 2023 Hendel et al., 2023). where middle-layer interventions are typically most effective. This divergence suggests that the functional organization of complex multi-step reasoning in LLMs differs fundamentally from that of simpler tasks, high- lighting the unique mechanisms involved in complex reasoning.

---

Published as a conference paper at ICLR 2026
Oweon2:5
LLaM631 dBinetnet
GareD 5 Nithh
ULMEC 1.00 Iosu
wise (a) Layer-wa wise Results of Two LLMs
(b) Information Density of Each Layer
(c) Qwen2.5
(d) 1.MMA3.-88-instruu.
(i Shallow (Layer 2)
@ Shallow (Layer 2)
Middle Layer [Layer
( Middle (Layer 17)
Deep Layer (Layer
Deep Layer (Layer
Figure 3: (a) Layer-wise performance of two LLMs with both extracted and Learnable CoT Vectors, averaged over four datasets. (b) Layer-wise information density curves of LLMs, obtained via PCA on 500 sampled instances across four datasets. Abbreviations: PC = principal component; var. variance; cum. cumulative; dims dimensions. (c-d) T-SNE visualizations of hidden states at shallow, middle, and deep layers on GSM8K (500 samples) of two LLMs. Left: sample distributions under non-CoT and baseline (with CoT) inputs. Right: same baseline with additional insertion of Extracted and Learnable CoT Vectors. Color scheme is consistent across (a, d): orange non- CoT, blue = baseline, green Extracted CoT Vector, red- Learnable CoT Vector.
This layer-wise variability naturally leads us to hypothesize that the underlying reasoning process in LLMs may itself be structured in stages. Building on insights from prior work on layer specializa- tion (Tenney et al. 2019 Chuang al. [023] Skean et al. 2025), we posit three-stage organization of perception, reasoning, and expression. In this view, Shallow layers primarily perform basic fea- ture extraction and semantic encoding, producing more linear and unified representations. Middle layers execute core reasoning process, leading to sample-specific, high-dimensional representations with dominant direction. Deep layers map internal reasoning states into surface-level linguistic outputs, where the representations again become more unified.
To test this hypothesis, we conduct an information density analysis via PCA on hidden states from 500 randomly sampled instances (Figure 3 (b)). We observe that mid-layers require significantly more principal components to explain the variance compared shallow and deep layers, while the variance explained by the top components drops sharply. This indicates higher representational com- plexity and the absence of dominant direction mid-layers, ultimately delineating three distinct stages across shallow, middle, and deep layers. Visualizing the latent space with t-SNE (Figures (c-d)) further reveal that middle-layer activations with CoT (baseline) form dispersed, input-specific clusters, reflecting highly complex and non non-linear structure that differs markedly from the non-CoT distribution. In contrast, shallow and deep layers exhibit more uniform activations, supporting that the middle layers serve core stage for sample-specific reasoning. These findings explain why Ex- tracted CoT Vectors fail in the middle layers: the mid-layer activations lack a coherent, task-general direction, making it difficult to extract compact and reusable CoT Vector.
Further cross- cross-layer transfer experiments support this conclusion. In Table injecting mid-layer vec- tors into shallow layers degrades performance, whereas shallow-layer vectors improve performance

---

Published as conference paper at ICLR 2026
Table 1: Comprehensive evaluation results. MATH-E = MATH-Fasy, MATH-H MATH-Hard, MMLU-P MMLU-Pro, CSQA CommonsenseQAA SQA StrategyQA, Reported CoT Vector results correspond to the best injection layer selected from layer-wise evaluation. We note that extraction-bass -based vectors are particularly dependent on this choice, whereas learnable vectors main- tain more consistent performance across layers.
Model
Method
#Params
GSM8K
MATH-E
MATH-H
MMLU-P
CSQA
SQA
Avg.
Baseline
74.6
69.9
47.9
33.2
53.8
23.7
50.5
Owen2.5-Mathe
Extracted
78.2
72.0
35.3
53.6
Learnalle
3.6K(x10)
83,5
71.9
50.9
35.1
58.2
31.2
55.1
LoRA
10.0M(x277788)
79.0
70.4
48.2
33.8
58.0
31.2
53.4
Baseline
77.4
62.0
34.6
44.6
72.7
60.8
58.7
LLaMA3.1-8 1-.A33.--B--BA 8B-nnsruct
Extracted
78.6
63.2
35.7
45.5
73.2
64.3
60.1
Leamable
4.2K(x1.0)
78.2
63.8
36.4
46.2
73.7
65.0
60.6
LoRA
13.6MG3338.0)
78.6
63,5
36.3
45.5
73.6
64.8
60.4
Cross-layer CoT Vector transfer results Qwen-GSM8K. Performance when injecting a CoT Vector extracted from a Source Layer (column) into a different Target Layer (row). The diagonal shows baseline performance (source target). A indicates the absolute change from the target layer's s baseline. Green arrows (f) indicate improvement, red arrows (J) indicate degradation.
Table 2:
Source: Shallow
Source: Middle (I.14)
Target Layer
Accuracy
Accuracy
Shallow (Layer 6)
J144
Middle (Layer 14)
75.3
19.0
66.3
when injected into middle layers. This indicates that the failure of mid-layer injection stems not from location, but from the intrinsically sample-specific and non-generalizable nature of mid-layer representations, which are ill-suited for capturing compact, task-wide reasoning direction.
4.2.2 LEARNABLE COT VECTORS
Learnable vs. Extracted CoT Vectors. Be- yond the conventional extraction- proach, we further introduce novel Learnable CoT Vectors, optimized via teacher-studen, architecture distill generalizable reasoning patterns. Experimental results reveal that the Learnable CoT Vector demonstrates two clear advantages over its extracted counterpart: (i) higher overall performance across benchmarks (Table [. and (ii) significantly greater stabil- ity across layers with higher layer-wise average accuracy (Figure (a)). Unlike the sawtooth U-shaped curve observed for extracted vectors, where gains concentrate in shallow and deep layers but diminish in the middle and with strong fluctuations, the learnable vector peaks at the first layers and maintains consistent plateau across all subsequent layers. Consequently, while extracted vectors show noticeable drops at mid-layers compared to baseline, learnable vectors con- sistently provide improvements across nearly all layers.
ap-
Figure 4: T-SNE visualization of over-fit and under-fit Learnable CoT Vectors (Layer 30 of LLaMA on GSM8K).
We attribute this divergence to the fundamental nature of each vector type. The extracted vector descriptive statistic, passively recording the average activation difference between CoT and non-CoT forward passes. Its efficacy is thus constrained by the representational properties of the source layer: strong when representations have clear dominant directions (e.g., shallow and deep layers), but fragile when such structure is absent. As consequence, it not only induces relatively mild shifts latent space (Figure 3 (c-d)) but also retains sample-specis noise. leading to sharp volatility where even adjacent layers at similar depths behave inconsistently. In contrast, learnable
8

---

Published as a conference paper at ICLR 2026
vectors optimized via gradient descent mimic the teacher model's reasoning. This results in a more directional and aggressive shift in the latent space (Figure 3 (c-d)), enabling it to overcome representational limitations of individual layers and avoid being intervened by sample-specific noise. Consequently, the learnable vector achieves stronger and more stable performance across layers.
These differences have important practical implications. Extracted vectors suffer from high instabil- ity, with the optimal injection layer varying across tasks and models (see Appendix for more details). In real-world deployment, where ground truth unavailable, such unpredictability severely limits their usability. Learnable CoT Vectors, however, produce consistent gains across all layers, with their strongest performance consistently emerging in the shallowest pnyers-ofter the very first layer. This stability permits simple and robust application: even on unseen tasks, injecting the vector the first layer suffices to achieve near-optimal performance.
However, the aggressive steering of Learnable CoT Vectors also brings risks. As visualized in Fig- 4 vectors applied to middle or deep layers are prone to overfitting, over-steering the latent space and collapsing diverse reasoning paths, which significantly degrades accuracy. This fragility stems from the representational nature of these layers that mid-layer activations are heterogeneous and sample-specific, while deep layers are closely tied to surface outputs, where even small perturba- tions can destabilize generation. To mitigate this, we employ early stopping or reduced learning rates, which produce mildly under-fitted vectors that still provide modest gains without catastrophic collapse. These findings reinforce our earlier conclusion that shallow layers are the most suitable for Learnable CoT Vectors, while mid and deep layers less amenable strong external guidance.
Learnable CoT Vectors vs. LoRA. From parameter-efficieet
perspective, our Learnable CoT Vector demonstrates advantages over LoRA fine-tuning. As illustrated in Table T it outperforms LoRA on most datasets while requiring orders of magnitude fewer trainable parameters. We attribute this to the fact that instruction-tun sttnedd LLMs already possess strong CoT priors, leaving limited room for LoRA to improve. In contrast, our approach adds an external guidance signal that efficiently steers the model's latent reasoning without altering the model's existing functional structure.
4.2.3 MODEL DIFFERENCES
As shown Table D the effectiveness of CoT Vectors varies across models: Qwen benefits more consistently and substantially from CoT Vector injection compared to LLaMA. For example, aver- aged across benchmarks, Qwen gains up to points over the baseline, whereas LLaMA yields a more modest improvement of 1.5 points with Learnable CoT Vectors. We trace this discrepancy to differences in the latent space structures of the two models throughout the three-stage reasoning process. In Figure (b), Qwen exhibits more distinet three-stage reasoning pattern than LLaMA. Notably, its top principal components explain more variance than those of LLaMA, suggesting a lower information density and a more structured latent space with clearer principal directions. This facilitates both extraction and optimization in capturing high-quality, task-general signals.
We conjecture that this structural disparity stems from differences in training data and procedures. Qwen has undergone more domain-focused and standardized fine-tuning, whereas LLaMA has been trained on broader and less curated corpora. a result, Qwen demonstrate a more distinct func- tional separation of layers. This structural clarity allows CoT Vectors to more easily capture task- general reasoning directions. In summary, the performance gap highlights that the efficacy of CoT Vectors is influenced by the inherent properties of the model's representations. Models with more structured latent spaces provide a more fertile ground for the CoT Vector intervention.
4.2.4 CROSS-DATASER AND
TRANSFERABILITY
We investigate whether CoT Vectors acquired from one source (model or dataset) can be effectively applied to another.
Cross-Model Transfer. As shown in Table 3 CoT Vectors gained from one model can be effee- tively reused in another. The vectors obtained from more powerfully instruction-tunan variant of the Qwen2.5 series 10wen..5-Math-77. - consistently improve performance when applied to Qwen2.5-Math-7B (74.6 + 77.5).
Cross-Dataset Transfer. Results in Table 3 further demonstrate transferability across datasets. 1) In-domain: COT Vectors obtained from the GSM8K dataset effectively enhance performance on the

---

Published as a conference paper at ICLR 2026
Source + Target
Baseline
Self
Transferred
Crosy-Modks Transfer
Owen2.5-Matte Math--a--------- IB-Intructt Qwen2.5-Math778
74.6
78.2
77.5
Crosy-Datasel Transfer
GSM8K + MATH
47.9
49.7
48.6
MMLU-Pro - MATH
47.9
49.7
48.5
Table 3: Cross-modes and Cross-dataset transfer results of CoT Vectors. Baseline refers to stan- dard zero-shot CoT prompting. Self means applying the CoT Vector obtained from the same model-dataset pair (no transfer). Transferred means applying CoT Vector obtained from dif- ferent source model or dataset.
MATH dataset (47.9 + 48.6). This confirms that the vector successfully captures a generalized mathematical reasoning strategy rather than merely memorizing dataset-specific features. 2) Cross- domain: vectors obtained from MMLU-Pro yield gains on MATH (47.9 - 48.5). This suggests that the CoT Vector may encode a meta-reasoning capability- as the ability decompose problems or follow logical steps-that is beneficial across distinct task domains.
These transferability experiments underscore central claim of our work: the CoT Vector is not merely a compressed set of features from a specific model or dataset, but portable, generalizable representation of a reasoning process that can effectively applied novel contexts.
4.2.5 ABLATION ON TRAINING SET SIZE FOR LEARNABLE COT VECTORS
We further conduct an ablation study on the size of the support set to compare the performance of the Learnable CoT Vector and LoRA under different data regimes. As shown in Table while both methods benefit from larger support sets, the Learnable CoT Vector consistently outperforms LoRA across all data scales. Notably, with very small support set (e.g., 100 examples), the learnable CoT Vector still yields noticeable improvements over the baseline, whereas LoRA offers only marginal gains. This highlights the strong data efficiency of our approach. As the support set grows, Learnable CoT Vector also demonstrates greater potential for performance improvement compared to LoRA. These phenomena all indicate that our Learnable CoT Vectors provide a more effective and scalable mechanism for enhancing reasoning performance than LoRA across diverse data conditions.
Table 4: Performance Comparison with Different Training Sample Sizes on Qwen-GSM8K.
Sample Size
Baseline
Learnable CoT Vector
LoRA
100
74.6
78.2
76.0
500
74.6
79.0
77.9
1000
74.6
82.3
78.5
3000
74.6
83.5
79.0
Overall, our results confirm that CoT Vectors are a highly efficient and effective means of enhancing reasoning capabilities. Due to space limitations, further ablation studies and robustness analyses of CoT Vectors are provided in the supplementary material.
5 CONCLUSION
We have presented CoT Vectors, extending the task vector paradigm to multi-step reasoning in LLMSs. Our analyses uncover a consistent three-stage reasoning process and show that the newly introduced Learnable CoT Vectors provide stronger and more stable gains than the traditional extraction-based approach, while also offering multiple perspectives on why their effectiveness dif- fers. These results demonstrate both the practical utility of CoT Vectors and their value probe into the mechanisms and organization of multi-step reasoning in LLMs. However, performance variability in intermediate layers highlights structural limitations, suggesting that task-level vectors may not fully capture intra-task diversity. Future work could explore finer reerrgg or adaptive vec- torization strategies to improve robustness and generalization.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by Jiangsu Province Carbon Peak Carbon Neutrality Science and Technology Innovation Special Fund Project (Grant No. BT2025029), National Natural Science Foundation of China (62576091), the Southeast University Big Data Computing Center, and Southeast University Kunpeng & AscendCenter of Cultivation.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study does not involve human subjects, per- sonally identifiable information, proprietary data. All datasets used, including GSM8K, MATH, and MMLU-Pro, are publicly available. The proposed method, CoT Vectors, is parameter-effiiiet technique for steering the reasoning process of pre-trained large language models. It does not in- troduce any new capabilities that could cause harm, nor does enable misuse beyond the standard capabilities of existing large language models. We are not aware of any potential risks related bias, fairness, or security that arise specifically from the method proposed. However, we acknowledge that the effectiveness and potential output of CoT Vectors are dependent on the base model and the support set data; such, they may reflect or amplify biases present in these sources. No conflicts of interest, legal compliance issues, or sponsorship influences are present in this work.
REPRODUCIBILITY STATEMENT
We have taken multiple steps ensure the reproducibility of work. All datasets used in experiments are publicly available and properly cited the main text and appendix. Training con- figurations, including hyperparameters, optimizers, and evaluation settings, are described in detail in Section and Appendix A.3 Theoretical claims, including the formalization of the CoT shift, are formally derived in Section 3. and Appendix A.2 Experimental results include multiple models, reasoning benchmarks, and various ablations to validate robustness in Section 4 and Appendix [A.4. A.5 We will release the full source code and pre-trained vectors upon publication further support reproducibility.
REFERENCES
Seyedarmin Azizi, Erfan Baghaci Potraghloo, and Massoud Pedram. Activation steering for chain- of-thought compression. arXiv preprint arXiv:2507. 044422, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint 21Xhv22412.13171, 2024
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint urXiv:2309.03888 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.1411 14.11........... 2021.
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stu- art Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.014660 2023.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to inter- nalize cot step by step. arXiv preprint arXiv:2405.1444 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aice aaddd language models. International Conference on Machine Learning, pp. 10794-10799, PMLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint 2:Xiv:2502.05171, 2025.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-366 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint ar.ii:2407.217888 2024.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint AXXi:22112.06669 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kink-Kiikattrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:21 10.04366, 2021.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. arXiv preprint (rxXiv23300159111 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv. 0rg/abs/2103.0388 2, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)33 2022.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:
2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05339 2023.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02440 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances neural information processing systems, 35:22199-2245 22213, 2022.
Hongkang Li, Yihua Zhang, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yo Chen. When is task vector provably effective for model editing? generalization analysis of nonlinear transformers. arXiv preprint a504v:2504. 10957, 2025.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint (.XXi:2311.0666 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 465944 2023.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- fems, 36:66677-666744 2023a.
12

---

Published as a conference paper at ICLR 2026
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, 36:66767-6677674 2023b.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 011533228--7444 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.2107, 2025.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint 20XX::25002.00133.. 2025.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume I (Long and Short Papers), pp. 4149 4158, 2019.
Xinyu Tang, Xiaolei Wang. Zhihao Lv. Yingqian Min, Wayne Xin Zhao, Binbin Hu, Ziqi Liu, and Zhiqiang Zhang. Unlocking general long sfain-of-thoughe reasoning capabilities of large language models via representation engineering, arXiv preprint arXiv:2503. 2025.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv 2019.
preprint
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. Function vectors in large language models. arXiv preprint arXiv:2310.15213 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances Neural Information Processing Systems, 37:95266-9522 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ofaiinof--tought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24487,, 2022.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.1 12122, 2024.
Haolin Yang, Hakaze Cho, Yiqiao Zhong, and Naoya Inoue. Unifying attention heads and task vectors via hidden state geometry in in-context learning. arXiv preprint arXiv:2505.18755 18752, 2025.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances neural information processing systems, 36: 1809-118222 2023.
Jason Zhang and Scott W Viteri. Uncovering latent chain of thought vectors in large language models. In Workshop Neural Network Weights New Data Modality.
13

---

Published as a conference paper at ICLR 2026
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of- of-thhughtt reasoning via preference optimization. arXiv preprint arXiv:2504.18397, 2025.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, al. A survey of large language models. arXiv preprint arXiv:2303. /8223, 1(2), 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.0859: 2019.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we use Large Language Models (LLMs) solely as writing assistant for grammar correction and minor phrasing refinement. The LLMs doesn't contribute research ideation, experimental design, analysis, or interpretation of results. All scientific content and claims entirely the responsibility of the authors. The LLMs are not involved in authorship and considered contributors to this work.
A.2 THE DERIVATION OF COT SHIFT
When the model reasons with CoT, the generation of an answer token E A depends not only on the question tokens but also on the intermediate CoT tokens. Formally, the single-head self-attention can be expressed as
exp(ak2)
exp(ak
exp(aK X)
Sa KQ. Kc. KAl, Vo, Vo. Val)
Vq
+
Vc
VA
Zuoill
Zuoull
Zuotal
ZQ Zuotll
exp(akK)
②
Zc
exp(ak
ZA
exp(ak
Ve
Vc
VA
ZQ
Zuttal
Zc
Zustal
ZA
SA(a,(Kg).[Vol)
SA(a,Ko),,
SA(a,(Ka)[[Va)
where Zo Σ exp(aKK)) Zc
cx ZA E exxp(aK)). and Zunal Zz Zo
A
In contrast, without CoT, the self-attention reduces to
ZA SA(a, [Kq, KAI, Vo, Val) = Zo SA(a, [Ko), + Zo + Z
SA(a, K [VA])
Zo AA
Zc
Therefore, through algebraic transformations and let Zuoal
can obtain,
SA(a, [Kq, Kc, Kc,
(Vo. Vo, Val) SA(a, [KQ. K Ka), VQ2 VAl)
(8)
Standard atlention
+ p. (SA(a, [c), (Vo)) SA(a, (Kq: KA), (Vo, VA)))
CoT shift
This is Equation 1 in the main text, which reveals that when LLM performs CoT inference, the actual content of CoT is prefix for the final answer given. Its role in inference ultimately comes from attention shift, which means that CoT information can be compressed into this shift. This discovery provides a theoretical basis for CoT Vectors.
A.3 MORE IMPLEMENTATION DETAILS
A.3.1 DATASET DETAILS
GSM8K. We randomly sample 3,000 examples from the training set construct the support set for CoT vector extraction/leaming and LoRA fine-tuning. For evaluation, we randomly selected 1. 1,000 examples from the test set.
MATH.
We have observed that the MATH dataset contains problems with five levels of difficulty, and the imbalanced distribution across these levels could lead to unstable training. To address this, we partition the dataset into two subsets based on difficulty: MATH-Easy (levels 1-3) and MATH- Hard (levels 4-5). For each subset, combine all categories and then sample 3,000 examples from the training portion to form the support set. Similarly, 1,000 examples were sampled from the test portion for evaluation.
15

---

Published as a conference paper at ICLR 2026
MMLU-Pro. Since ground-truie truth CoT sequences required, have used the official validation set as the support set. Although this set contains only 70 samples, still provides sufficient signal for CoT vectors to capture useful reasoning patterns. For testing, also sample 1,000 examples from the test set.
A.3.2 PROMPTS AND GENERATION CONFIGURATIONS
Prompts. We use two versions of prompts: CoT and Non-CoT, as shown Table 5
The CoT version is specifically designed for scenarios that require step-by-step reasoning. This prompt format consistently applied to all our baseline model tests and to the evaluation processes for both the CoT Vectors and LoRA methods. It also serves as the teacher input during the training of our Learnable CoT Vectors. Conversely, the Non-CoT version is used to directly obtain the final answer without any intermediate reasoning, serving exclusively the student input during the training of our Learnable CoT Vectors.
Additionally, for GSM8K and MATH datasets, instruct the LLM to directly generate the fi- nal answer, while for the multiple-choice dataset MMLU-Pro, we prompt the model to output the correct option letter. In the input construction for MMLU-Pro, the answer choices are appended immediately after the question.
Table 5: Prompt Templates.
Dataset
Reasoning Type
Prompt
You are helpful and precise assistant for solving math problems. Please reason step by step, and put your final answer within {boxed
CoT
GSM8K & MATH
You are helpful and precise assistant for solv- ing math problems. Put your answer within (boxed().
non-CoT
You are a helpful and precise assistant for solving problems. Please reason step by step, and put your final answer within \boxed(}- Your final output should be only the uppercase letter of the correct choice (e.g., A).
CoT
MMLU-Pro
You helpful and precise assistant for solv- ing problems. Put your answer within (boxed(): }- Your final output should be only the uppercase let- ter of the correct choice (e.g.. A).
non-CoT
Generation Configurations. All tests were conducted using a consistent set of generation parame- ters, as detailed in T
Table 6: Generation Configuration Parameters
Parameter
Value
Number of beams
Maximum new tokens
512
Length penalty
0.0
Do sample
False
Temperature
1.0
Top-p
1.0
A.3.3 HYPERPARAMETERS
We employee the AdamW optimizer (Loshchilov & Hutter 2017) with weight decay of 1e-3 and train using float16 precision for both Learnable CoT Vector and LoRA. Additionally, we set the accumulate gradient batches to 2 effectively increase the batch size.
16

---

Published as a conference paper at ICLR 2026
Learnable CoT Vectors. To train Learnable CoT Vectors, configure distinct hyperparam- eters for each model and dataset. All of these parameters are fine-tuned through a combination of systematic grid search and manual adjustments to ensure stable and efficient training. For the LLM, we adopt a unique tiered learning rate strategy. Using higher learning rate for the earlier layers of Qwen is intended to obtain vectors that better fit the task-general direction, as mentioned in Sec- tion 4.2 of the main text, where shallow layers exhibit better directional properties. In contrast, using smaller learning rates for the deeper layers of Qwen and the entire Llama model aims to ob- tain underfitting vectors, as they demonstrate better stability when information density is high.As detailed in Table
Table 7: Hyperparameters for Learnable CoT Vector
(a) Qwen
Dataset
Samples
Learning Rate
Warm-up
Note
GSM8K
3000
5e-3 and le-4
0.1
LR: First 4 layers used 5e-3, others le-4
MMLU-P
70
5e-3 and lc-4
0.1
LR: First 4 layers used 5e-3, others 1c-4
MATH-E
2000
5e-3 and le-4
0.5
LR: First layers used 5e-3, others le-4
MATH-H
2000
5e-3 and le-4
0.5
LR: First 4 layers used 5e-3, others 1e-4
(b) LLaMA
Dataset
Samples
Learning Rate
Warm-up
GSM8K
2000
le-4
0.5
MMLU-P
70
le-4
0.1
MATH-E
1000
lc-4
0.5
MATH-H
1000
1e-4
0.5
LoRA. We apply LoRA with set of shared, general hyperparameters across all models and datasets, with training set sizes consistent with those used for CoT vector training. Interestingly, we found that when using LoRA for fine-tuning, the loss was very low from the beginning of training. This aligns with our hypothesis in Section 4.2 that for models already fine-tuned with CoT and possessing CoT capabilities, using LoRA for further specific CoT fine-tuning would yield limited gains.As detailed Table
Table 8: General Hyperparameters for LoRA.
Parameter
Value
Learning Rate
5 x 10-5
Warm-up
0.1 16
Matrix Rank
Dropout Rate
0.05
Target Modules q-proj, k-proj, o.proj, v.proj
A.4 COMPLETE EXPERIMENTAL RESULTS
In Section 4.2 of the main text, we provide the comprehensive evaluation results for two types of CoT Vectors, baseline and LoRA cross models and benchmarks, in Table [ Especially for CoT Vectors, the final result is the report of the best layer after iterating and testing the results of each layer of LLMs. Here we present all the layer-wise results of each model and dataset in Figure 5
We can observe the following pattern. First, mentioned in the main text, extracted layers gen- erally exhibit performance troughs in the middle layer, with strong volatility. Relatively speaking, learnalle layers usually achieve their best performance in the first few layers, while the subsequent layers are generally stable around the baseline or with a small gain around the baseline. Second, in addition, the most crucial point is that for highly volatile extracted vectors, we find that even if
17

---

Published as a conference paper at ICLR 2026
Qwen-GSM8K
Qwen-MMLL-P
88
34
78
29
73
24
68
Extracted
Learnable
Learnable
19
Layer
Baseline
Layer
Baseline
Qwen-MATH-E
MMen-MMTTH-
74
51
72
70
48
68
45
Extracted
Extracted
Learnable
42
Learnalle
Layer
Baseline
Layer
Baseline
(a) Layer-wise results of Qwen2.5-Math-77
LLaMA-GSMBK
LLaMA-MMLU-P
80
47
46
45
70
44
Extracted
Learnable
65
Layer
Baseline
43
Layer
Baseline
Learnable
LLaMA-MATH-E
LLaMA-MATH-H
37
35
62
33
Extracted
Extracted
58
Learnable
31
Learnalle
Layer
Baseline
Layer
Baseline
(b) Layer-wise results of LLaMA3.1-8B-Insse
Figure 5: Layer-wise performance analysis of two L.MSs and four benchmarks.
the same model is used, the layers that achieve the best performance on different datasets are com- pletely different. Even certain layer can gain a lot on one dataset, but it will damage performance on another dataset, and there is basically no regularity this. This means that when using extracted CoT Vectors, layer wise search must be conducted first to determine the optimal shift layer, which almost impractical for practical applications without ground truth.
A.5 ABLATIONS AND ADDITIONAL ANALYSES
Due to space limitations in the main paper, provide extended analyses of CoT Vectors in this section. We first present alternative acquisition methods beyond the activation-gap based approach, including raw-activation (Section 1. and attention-level variants (Section A.5.2), which also im- prove over the baseline but yield less representative gains than the method reported in the main text. We then conduct ablation studies on key design choices, such the scaling factor μ for extracted vectors (Section A.5.4), the loss balance for learnable vectors (Section A.5.6), and the training set size (Section 4.2.5). through which uncover additional insights into their behaviors. We fur-
18

---

Published as a conference paper at ICLR 2026
Table Results of Raw Activation Extracted CoT Vectors.
Model
Method
#Params
GSM8K
MATH-E
MATH-H
MMLU-P
Avg.
Bascline
74.6
69.9
47.9
33.2
56.4
Owen2.5-Math--1
Raw Activation
77.6
71.5
49.2
35.8
58.5
Activation Gap
78.2
72.0
49.7
35.3
58.8
Baseline
77.4
62.0
34.6
44.6
54.7
LLaMA3. Instruct
Raw Activation
80.4
62.4
35.4
56.0
Activation Gap
78.6
63.2
35.7
45.5
55.8
ther analyze robustness and transferability, considering settings such multi-layer vector injection (Section A.5.3). varying the source of teacher CoT sequences (Section A.5.5), ) and cross-model or cross-dataset transfer (Section 4.2.4.. Finally, we complement these quantitative studies with analyses (Section A.5.7): illustrating both successful and failure scenarios to better understand when and why CoT Vectors are effective.
A.5.1 RAW ACTIVATION EXTRACTED COT VECTOR
This method is inspired by the work in ICL (Todd al. 2023 Hendel et al. 2023), using the direct model activations as a coarse-grained approximation of the CoT Vector. Specifically, we forward- pass the triplet (Q, CoT, A) and take the mean of the hidden states (activations) across all answer tokens A at a specific layer l.
Σ a aE
(a)
(10)
CoT
LAI
(1) where a ( () is the hidden state (activation vector) of the answer token a at layer under the CoT prompting input, and |A| denotes the total number of answer tokens.
Raw Activation vs. Activation Gap. Comparing two extraction strategies in Table 9 the Activation-Gai vector generally outperforms the Raw-Activation vector in most datasets, bet- ter isolates the reasoning-indus induced shift. Interestingly, in some cases raw activations match or even surpass gap-based ones, suggesting that hidden states encode not only the reasoning information but also auxiliary task-relevant features.
A.5.2 ATTENTION-LEVER CoT VECTOR
From a mathematical perspective, the most precise location for the CoT Vector should be within the attention computation, shown in Eq. in main text. Therefore, in addition to the Activation-level CoT Vector presented in the main text, we have also explored an intervention at the attention level in both Extracted and Learnable CoT Vectors. This method similarly improve performance over the baseline but is slightly less effective than the activation-level approach.
Attention-level Extracted CoT Vector. We extract the attention output at specific layer when the LLM processes the (Q. CoT, A) triplet and the (Q, A) pair separately during the forward pass, and compute the gap between them highly accurate pncentionouuputtleee
CoT Vector. For multi- head attention, we compute a distinct vector UCTT, head for each head h, enabling more fine-grained
_(i,h)
control. The final task vector is collection of all these head-level vectors. extracted CoT Vector in activation form inserted into the hidden states of the corresponding extraction layer, while the attention-gan gap form is inserted into the attention output of that same layer. During insertion, manually set constant μ is used to scale the magnitude of the CoT shift.
Attention-level Learnable CoT Vector. Similar to the activation level learnable method, but the attention-level vector added as a shift to the output of specific attention module. For multi-head attention, a distinct vector is learned for each head, allowing for fine fine-grained control.
In practice, although this method does lead to some performance improvement against baseline, the gains are relatively modest compared to activation-level variants. The details of these experiments
19

---

Published as a conference paper at ICLR 2026
Table 10: Results of Attention-level CoT Vectors.
Model
Method
#Params
GSM8K
MATH-E
MATH-H
MMLU-P
Ayg.
Baseline
74.6
69.9
47.9
33.2
56.4
Qwen2.5-Math-77
Extracted
76.8
71.1
49.7
35.2
58.2
Learnable
7.2K
76.1
71.9
48.6
34.1
57.9
Baseline
77.4
62.0
34.6
44.6
54.7
IJ.MMA.1-8B-Inmmm
Extracted
78.3
62.7
35.4
46.2
55.7
are listed in Table I0 This result suggests that the effectiveness of task-genera CoT Vectors does rely on fine-grained, head-specific attention patterns, but instead reflects broader representational shifts at the activation level.
A.5.3 MULTI-LAYER COT INJECTION
In our main experiments, our designed CoT Vector is exclusively obtained and inserted at a single layer of the LLM. However, we also curious about whether such operations could be applied simultaneously across multiple layers, and whether performing shifts in more layers would lead to greater performance improvements.
Extracted CoT Vectors. As shown in Table 11 we find that simultaneously injecting the extracted CoT Vectors into multiple layers produces different effects, depending on the functional character- isties of the selected layers. When vectors are injected into multiple layers that prove beneficial for CoT integration(c.g. Layer 3, 7, 15 for GSM8K), a positive synergistic effect is observed, re- sulting in greater performance improvement than single-layer injection. Conversely, when vectors are simultaneously inserted into multiple layers that are sensitive to CoT but unsuitable for external intervention(e.g. Layer 8. 14), additional interference effects arise, leading to a further decline in performance, which sometimes even significantly below the baseline. This phenomenon reaffirms which the significant functional specialization among different layers of the Transformer and also indicates that the injection effect of CoT Vectors can be further optimized through inter-layer coordination and combinatorial selection.
Table 11: Multi-layer Extracted CoT Injection on Qwen- Owen-GSM8K.
Model
Layers
Accuracy (%)
Baseline
74.6
15
77.6
7, 15
79.3
Extracted
3, 7, 15
80.1
8, 14
45.1
Learnable CoT Vectors. We further investigate multi-layer insertion of Learnable CoT shifts. In- terestingly, multi-layer insertion produces complementary effects for extracted vs. learnable vectors: extracted vectors benefit from multi-layer insertion (cumulative mild shifts yield larger gains), while learnalle vectors typically perform best when injected single shallow layer and degrade under naive multi-layer injection as shown T2] Mechanistically, this difference stems from the am- plitude and directionality of the shifts. Extracted vectors produce mild, descriptive shifts that safely accumulate across layers, whereas learnable vectors induce stronger, optimization-drive shifts that can over-steer representations applied repeatedly.
A.5.4 ABLATION ON THE SCALING FACTOR OF EXTRACTED CoT VECTORS
In the main experiments, the scaling factor μ for the Extracted CoT Vectors is set fixed value of 1.0. To further investigate the impact of this hyperparameter, we experiment with different values of
n.
20

---

Published as a conference paper at ICLR 2026
Table 12: Multi-layer Learnable CoT Injection on Qwen-GSM8K.
Method
Layers
Accuracy (%)
Baseline
74.6
83.5
Learnable
0,
81.0
0, 1.2
76.1
For single-layer injection. In Table 13 we find that the effect of the scaling coefficient μ varies significantly across different model layers, leading to the following observations: For layers where == 1.0 already yields improvement (e.g., Qwen on GSM8K at layer 6), increasing μ can further en- hance performance slightly, although excessively large l leads to over-steering and performance degradation. In contrast, for layers where 1.0 causes performance degradation (e.g., Qwen on MATH-Hard at layer 17), reducing μ can mitigate the negative effect, but no value of µ can bring improvement over the baseline. This indicates that the CoT Vectors extracted from these layers are inherently low-quality, and their deficiency cannot remedied simply by adjusting μ.
Table 13: Model Accuracy Comparison Across Different Values for single-layer injection
(a) Qwen--AITH--Hard-Laye 17
(b)
Accuracy (%)
Accuracy (%)
0.0
47.9
0.0
74.6
0.2
46,6
0.8
78.
0.5
46.7
1.0
78.2
1.0
45.8
1.2
78.4
1.2
44.4
1.5
78.0
For multi-layer injection. For many layers of dense shift insertion, necessary to reduce value. As shown in Table [4] when applying CoT Vector shifts to four layers of the LLM con- currently, scaling down μ to 0.5 leads to further performance improvement. When injecting CoT Vectors into all layers simultaneously, reducing y 0.2 is necessary achieve optimal results.
Table 14: Model Accuracy Comparison with Different Values for multi-layer injection
(a) Qwen- Bivn-GSMMK-all layers
WWMn-GGMMKK
Setting
Accuracy (%)
Model
Accuracy (%)
Baseline (μ 0)
74.6
Baseline
74.6
/ 0.05
74.8
Layers 3,7,15,23
1.0
79.6
/ 0.1
76.3
Layers 3,7,15,23
0.5
80.3
f = 0.2
76.7
μ 0.3
67.1
1 = 0.5
14.5
1 1.0
15.2
A.5.5 SOURCES OF TEACHER CoT
For the explicit CoT sequences used extraction or as the teacher signal in the learning process, experiment with different types of sources. These include using ground-truth CoT from the dataset (typically human-wiin (oran-written) as well as model-generated CoT sequences, obtained from the model itself.
As shown Figure [exxpeiiettt results indicate that the performance difference between using ground-truth CoT and model-generated Co'I is relatively minor. Both approaches yield comparable
21

---

Published as a conference paper at ICLR 2026
LLaMA3.1-6B-Instruct
Qwen2.5-Math-7B
79
79
74
74
64
64
1
7
10
13
16
10
22
25
28
31
10
13
16
19
22
25
28
Layer
Layer
ground-truth CoT
ground-truth CoT
mederage CoT
mederatee CoT
Figure 6: Performance comparison between ground-truth and model-generated CoT sequences of Extracted CoT Vectors on GSM8K.
improvements in the model's reasoning capabilities, suggesting that the benefits of CoT guidance are robust to variations in the specific wording or stylistic differences the reasoning chains.
A.5.6 ABLATION ON TRAINING OBJECTIVES FOR LEARNABLE CoT VECTORS
For training the Learnable CoT Vectors, we perform an ablation study on the loss design. In addition the hybrid loss (a combination of representation aligmment loss and cross-entropy loss) adopted in the main experiments, we evaluate two simplified variants: using only the alignment loss or only the cross-entropy loss. As shown Table [5] both simplified variants lead clear performance degradation compared to the hybrid objective.
We attribute this to the complementary roles of the two components. The representation alignment loss is crucial for distilling task general reasoning patterns: by enforcing the student's intermediate representations to align with the teacher's reasoning trajectory, it effectively transfers the underlying logical process. However, when used alone, it lacks explicit guidance for producing correct and coherent final answers. The cross- cross-entropy loss complements this by anchoring the model's I's output to accurate reasoning traces, ensuring that the learned vector supports both faithful reasoning dynamics and correct task execution.
Table 15: Ablation Results Training Objectives in the Qwen-GSM8K Setting
Method
Accuracy (%)
Baseline
74.6
Learnable CoT Vector
83.5
Learnable CoT Vector (only alignment loss)
82.9
Learnable CoT Vector (only CE loss)
78.4
A.5.7 CASE STUDIES
To qualitatively understand the mechanistic impact of CoT Vectors, we present two representative cases. Figure 7 (a) shows GSM8K problem where baseline (Zero-Shot CoT) fails. The model's reasoning becomes stuck in an infinite loop, repetitively counting the bids without making progress toward a solution. structure the problem correctly and produce valid solution. Notably, the learnable vector not only leads to the correct final answer but also produces reasoning chain that was more structured and closer the ground-truth CoT in the dataset. This highlights how CoT Vectors can reshape the model's 's internal reasoning trajectory, guiding it toward more faithful and efficient solutions.
contrast, both the Extracted and Learnable CoT Vector helps the model to
Conversely, also observe cases where CoT Vectors harm performance. In Figure 7 (b), the base- line arrives at the correct answer unaided. However, when CoT Vectors are injected, the reasoning becomes more aligned with the ground-truth style (especially for extracted vectors), but both vari-
22

---

Published as a conference paper at ICLR 2026
Question: Cannen wins the antique desk after outhidding other people who each bid starting from $200 opening bid with $50 increments. she pay?
Baseline
DO
Extracted
DO
Learnable
OO
Ground Truth
much Carmen
Let' badding process she
much Carmen paid
other people bleding
number
(óres parapapp asatyse
sewagraph analysin
Sindrste 120i 150 sokkedt coal
arroust she paid.
The bade raw by 150 each time, we can
Carmen beta ather wach the other
the dlek other people.
Carman
pacpr
additimnaa tits
also laid agaimett
biditen.
$200 $50 + $250
of tids ohe made 1 jratel but) 3 jacdiionnll bals)
per bid Soodf 100 160xo-100 to overal
Siace the rise by 500 each time, the
Carmen: $250
$300
from other
cod.
5300, $200,... $12200 dntermealere
$350
Sincee bids
fime.
150 $100 $ect100- -00-0000000000
sapp oraltedt
-Carman: $250 $58 + $480
can find the tocal amoune she paid
adtied to the inikal ooer of
ceek
Third other perton: $50 $450
muitiplying the number by adding
The theeedoe coos opentin
Carmen suution with 5500
bis
Siicht vo0
The trnal arsiver ix. (988)
Total amount (number bids 1)"
akoue 500
oakusadion 900
(a) Successful Guidance
items triplets-boe many skuins?
Bascline
DC
Extracted
Learnable
6
Truth
anabsijj
Nartha neoda, need caliciato
analysg)
skilns hat + akozis seart 12
HHll chrldron iioits por 9
lokal number skeans requi each
Huc 2 sholrs per hat hals
skalts per sercato 1 sboin per pail
-Sceevee: chiitwnn 1-1 sowine 3 sovete
graeekoiie needee
Scart divera
exded 12244 2122t12-22ood akains
22 sox ea for nat
3 12 shvine
So to lent suntie for of her grawden ldtwn, whe will
-Stenslen:: ct kiewe skwine per
soe for scurt
Saaweter
cowetier 3 per
-Aiitene:: childenn
mitters
Milleme:
mihheee 3 1-3 sduein
22 soo ea for pair andda
abwine
cren atvena
Toral aseams grwndenad
Now lofs odd the numbero
Sinco there are granduhiddion,
Now lofs add tocal number
skicnoo readed
mulills the total pioo grandoriid by rondinslronddiild Adron skalis
stotno redeed:
(Paal)) (scorves) (sweoatons)
(swearers)
68 sticino
[sookss) 72
So, Narth needs to (93) skeins
So. Martha needs to buy (boked(57)
So. Martha needo buy (700 skeins of wool
wool
skeins wool
(b) Harmful
Figure 7: Case studies.
ants introduce ballucinations or arithmetic mistakes that ultimately lead to incorrect conclusions. This demonstrates that while CoT Vectors can effectively shift latent reasoning, the perturbation may sometimes interfere with correct reasoning paths, underscoring the double-edged nature of such interventions.
23


Reviews:
[{"review_id": "JcUtvZrTlm", "reviewer": "Reviewer_YJ7G", "summary": "The paper introduces CoT vectors, an extension of the task vectors to encode and transfer multi-step reasoning in LLMs. The authors propose and study two initiation of the CoT vectors: extracted CoT vectors and learnable CoT vectors. The extract CoT vectors are computed as the activation difference between reasoning and non-reasoning traces at a given layer; the learnable CoT vectors are parametrized as a learnable shift added to the hidden state of a layer and optimized in a teacher-student framework to distill reasoning representations. Experiments on three benchmarks (GSM8K, MATH, and MMLU-Pro) across two model families (Qwen2.5 Math and Llama3.1 8B) show consistent improvement over zero-shot CoT and LoRA baselines using only 3-4K parameters. Furthermore, the authors use CoT vectors to uncover a three-stage reasoning process in LLM and provide new insights that contrast with prior task vector findings. The observation is supported by additional analyses on layer-wise performance and latent space visualization.", "questions": "- In equation 7, why is the intervention applied to the hidden state rather than directly to the attention output? How do the author explain the difference between formulation and practical use?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper has clear question formulation and motivation, is well-structured, and easy to follow. \n- By probing with CoT vectors, the authors discover a U-shaped layer-wise performance curve and provide a three-stage reasoning process interpretation, offering novel insights that are different from prior task vector research. Supporting analyses provide a valuable contribution to understanding model internals. \n- The proposed learnable CoT vectors address limitations of extraction-based methods (layer-wise instability), and effectiveness is supported through experiments and visualizations\n- Evaluations span across multiple models, datasets, and baselines. The results demonstrate consistent improvement, validating the effectiveness of the proposed method.\n- The method is parameter-efficient and requires negligible inference overhead, avoiding prompt lengthening and full-finetuning, which aligns with practical needs in efficient LLM deployment.", "weaknesses": "- The scope of \"task-general\" vectors is not fully supported by experiments. The CoT vectors are obtained per dataset, and the paper doesn't perform cross-task transferability tests to back the \"task-general reasoning\" claim. For example, the authors should consider applying vectors learned on GSM8K on MATH and vice versa.\n- The improvements on Llama are small, and MMLU-pro only uses 70 annotated questions. It is difficult to assess the reliability of small gains without variance across seeds and significance tests.\n- LoRA is the only parameter-efficient baseline, whereas other activation intervention methods are not compared or mentioned, including [1], [2], [3]. \n- The paper reports the best injection results. However, in practice, a comprehensive layer search could limit applicability.\n\n[1] Azizi, Seyedarmin, Erfan Baghaei Potraghloo, and Massoud Pedram. \"Activation Steering for Chain-of-Thought\"\n[2] Zhang, Jason, and Scott W. Viteri. \"Uncovering Latent Chain of Thought Vectors in Large Language Models.\"\n[3] Tang, Xinyu, et al. \"Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineering.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jk8YriKpCl", "reviewer": "Reviewer_wcrC", "summary": "This paper introduces CoT Vectors, a compact representation that encodes multi-step reasoning knowledge, enabling LLMs to enhance reasoning without modifying model weights.\nTwo approaches are proposed to obtain these vectors: extracted CoT vectors and learnable CoT vectors optimized via a teacher-student distillation framework, with analysis of their layer-wise effects.\nExperiments show that the proposed methods can improve performance on the studied tasks.", "questions": "see Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method proposed in the paper is overall clear in its conceptual approach and is presented in a way that is relatively easy to understand.\n2. Layer-wise analysis offers a novel perspective on how reasoning in LLMs is internally organized.", "weaknesses": "1. The main experiments in the paper focus on mathematical reasoning and some domain-specific tasks, and the performance of the proposed method on more natural language logical reasoning tasks remains to be further validated.\n\n2. The training of task vectors may depend on factors such as the number and quality of sampled support instances, as well as the balancing factor $\\lambda$ between the two losses. Providing additional experimental results on these aspects could better demonstrate the robustness of the proposed method.\n\n3. Although the authors provide some formula-based intuitive insights in Section 3.1, there remains a certain gap between the theoretical explanation and practical implementation. For example, the $\\mu$ and shift vector in Equation 1 should ideally be functions that vary with the input, whereas the task-general CoT vectors seem to assume a single shared reasoning chain prompt (i.e., the same $K_C$ and $V_C$) for all instances of the same task. Even under this assumption, $\\mu$ would still be expected to vary with the input.\n\n4. While the proposed method introduces relatively few trainable parameters, in practice an additional teacher model is required to perform CoT reasoning and align the output distributions with the student model. Reporting only the number of trainable parameters may therefore provide an incomplete picture of the overall overhead.\n\n5. The comparison in the paper is limited to LoRA as a parameter-efficient fine-tuning baseline, lacking evaluation against other fine-tuning or prompt optimization methods, such as Prefix-Tuning or Adapters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XEmopOTkD2", "reviewer": "Reviewer_QPGb", "summary": "This paper presents CoT Vectors, a novel extension of task vectors for modeling multi-step reasoning in LLMs. The approach is well-motivated and provides meaningful insights into reasoning dynamics, with Learnable CoT Vectors showing stronger and more stable gains than extraction-based methods. However, the variability across intermediate layers suggests limitations in capturing intra-task diversity. Further exploration of adaptive or hierarchical vectorization could enhance robustness and generalization. Overall, this is a valuable contribution to understanding structured reasoning in LLMs.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper’s strengths lie in its novel approach to vectorizing Chain-of-Thought (CoT) reasoning, offering a compact and efficient representation of multi-step reasoning processes. The introduction of a teacher–student framework for learning CoT vectors is noteworthy, as it effectively captures the influence and transfer of reasoning knowledge. In addition, the paper is supported by comprehensive experimental evaluations, which clearly demonstrate the advantages and robustness of the proposed method across different settings.", "weaknesses": "Here are some concerns and/or issues about this work:\n\nThe proposed method is intriguing, but its systematic assessment of effectiveness is lacking. Specifically, it lacks a measurable way to categorize different types of CoTs and demonstrate when the proposed method works well and when it doesn’t.\n\nThe authors claim that the method saves compute cost because fine-tuning is not required. However, the inference process involves invoking both the frozen LLM and the student model. Given that models are typically used for inference far more often than training, the overall efficiency of the proposed method in real-world scenarios remains uncertain.\n\nIf I understand the authors correctly, there may be a hidden issue in Eq. 4. In Eq. 4, the extracted CoT vector takes the average. This means that if the answer is long and there are only a few mismatched tokens, the model may accept the result. However, if the problem is similar to the one depicted in Figure 1, a few mismatched tokens can invalidate the entire answer.\n\nThe experimental study lacks evaluation of the end-to-end efficiency of the proposed method. The authors propose comparing it to fine-tuning, so they should include some experimental results to support the significance of the proposed method compared to fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)