Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Authors: Seungwook Han
Jinyeop Song
Jeff Gore
Pulkit Agrawal
Venue: ICML 2025 spotlightposter (2025)
Keywords: in context learning
task vectors
mechanistic interpretability
Primary area: deep_learning->large_language_models
TL;DR: We study how task vectors emerge in transformers through an encoder-decoder framework. We show that the ability to decode task from representations correlates with task vector effectiveness and predicts LLMs' in-context learning ability.

Abstract: Autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. Prior works have shown that transformers represent the ICL tasks as vectors in their representations. In this paper, we leverage the encoding-decoding framework to study how transformers form task vectors during pretraining and how their task encoding quality predicts ICL task performance. On synthetic ICL tasks, we analyze the training dynamics of a small transformer and report the coupled emergence of task encoding and decoding. As the model learns to encode different latent tasks (e.g., "Finding the first noun in a sentence.") into distinct, separable representations, it concurrently builds conditional decoding algorithms and improves its ICL performance. We validate this phenomenon across pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B) and over the course of pretraining in OLMo-7B. Further, we demonstrate that the quality of task encoding  inferred from representations predicts ICL performance, and that, surprisingly, finetuning the earlier layers can improve the task encoding and performance more than finetuning the latter layers. Our empirical insights shed light into better understanding the success and failure modes of large language models via their representations.

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Seungwook Han
Jinyeop Song"
Jeff Gore
Pulkit Agrawal
Abstract
to new situations. In artificial intelligence, autoregressive transformers are beginning to exhibit similar capabilities (Brown et al., 2020; Bubeck et al., 2023; Ajay et al., 2023; Han et al., 2024). Through in-context learning (ICL), they adapt to new tasks without parameter updates, suggesting they might also be forming internal abstractions (Raventós et al., 2024; Hong al., 2024; Zheng al., 2024; Kumar et al., 2024; Han et al., 2024).
Autoregressive transformers exhibit adaptive learning through in-context learning (ICL., which begs the question of how. Prior works have shown that transformers represent the ICL tasks as vec- tors in their representations. In this paper, we leverage the mucoding-decoding framework to study how transformers form task vectors dur- ing pretraining and how their task encoding qual- ity predicts ICI. task performance. On synthetic ICL tasks, we analyze the training dynamics of small transformer and report the coupled emer- gence of task encoding and decoding. As the model learns to encode different latent tasks (e.g., "Finding the first noun in a sentence.") into dis- tinct, separable representations, it concurrently builds conditional decoding algorithms and im- proves its ICL performance. We validate this phenomenon across pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B) and over the course of pretraining in OLMo-7B. Further, demonstrate that the quality of task en- coding inferred from representations predicts ICL performance, and that, surprisingly, finetuning the earlier layers can improve the task encoding and performance more than finetuning the latter lay- ers. Our empirical insights shed light into better understanding the success and failure modes of large language models via their representations.
Hendel et al. (2023); Merullo et al. (2023); Todd et al. (2023) introduce a mechanistic perspective on how pre- trained LLMs represent the latent concepts underlying the ICL task as vectors their representations. They empiri- cally demonstrate that these task-specifie vectors can trigger the desired ICL behavior in many cases, with the effec- tiveness varying across tasks. Although impactful first observation, there still remains unanswered questions of why these task vectors exist in the first place and why the ef- fectiveness varies by task. This necessitates a deeper mech- anistic understanding of this internal abstraction behavior of LLMs.
In our work, we leverage the encoding-decoding decoding framework to investigate the origin and formation of task vectors in transformers. To study their emergence during pretraining, we train a small transformer on a mixture of sparse linear regression tasks. We find that task encoding emerges as the model learns map different latent tasks into distinct, separable representation spaces. This geometric structuring of the representation space is coupled with the development of task-specific ICL algorithms namely, task decoding. Through causal analysis, we demonstrate that the model associates different algorithms to different learned concepts in the representation space and that ICL happens through the two-step process. Importantly, we see that the emer- gence of the two-stage process coincides with each other, suggesting mutual dependence between the two.
1. Introduction
Throughout history, humans have made sense of the world by distilling complex experiences into fundamental abstrac- tions, such as physics and mathematics. These mental mod- els enable us to learn quickly, predict outcomes, and adapt
Inspired by these findings, we investigate the task encoding- decoding phenomenon across different pretrained model families and scales (l.ama-3.1-88/777
Equal contribution 'Massachusetts Institute of Technol- ogy, Cambridge, USA. Correspondence to: Scungwook Han <swhan@ mi. Jinyeop Song <ycopjin mit.cdu>>
and Gemma-2 2B/9B/27B) on more natural ICL tasks, such as part-of- speech tagging and bitwise arithmetic. We introduce task decodability, geometric measure that quantifies how ef- fectively model can decode the task from its intermediate
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Task Vectors and Latent Concepts in LLM Represen- tations. Recent work by Todd et al. (2023) and Hendel et al. (2023) identifies task-specific vectors in LLMs that can induce desired in-context learning behaviors (e.g.. object- color mapping). Building on this foundation, our study ex- amines when and how task-specific representations emerge and how their quality (TD score) be measured and used to predict downstream ICL performance. Moreover, Park et al. (2024) study the different algorithmic stages for ICL under homogeneous Markov chain settings. In contrast, our work expands the study to heterogeneous tasks, a regime more reflective of LLM pretraining, and proposes a more framework to understand the Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Coupled Emergence of Task Decoling
Task Decodability predicts ICL Performance
Non sprecn signng
Broodina
Hovn, Adjectvv
Ho Sede
SAMB
C
Qu How do task vertors emerge?
ICL perfor performmree
Decocing
specifc cagereclgrrmmmm thm
spechicalgormm
CD-1
sperili algor thm
Task decocabiltty (T12)
Training
Figure An overview of our work. We study the task ancoding-decoding to explain why and how task vectors emerge in pretrained LLMs. We demonstrate that transformers concurrendy learn to map latent concepts into separable representations and develop task task-specific decoding algorithms. We validate the generality of this finding across model familics and scales, and show that the quality of task sicoding-decoding predict ICL task performance.
representations. We demonstrate that task decodability is causally related to ICL performance through ablation stud- ies on ICL examples, finetuning, and prompting. Overall, our study reveals that the encoder-decoder frameworks pro- vide valuable insights into the emergence of task vectors and why their effectiveness varies across tasks.
2. Related Work
Mechanisms of ICL. Astounded by LLMs' ability to per- form ICL, many have proposed theories to understand the mechanisms of ICL. Some works (Dai et al., 2023; von Os- wald al., 2023; Ahn et al., 2024; Akyürek et al., 2024) have proposed that LLMs, with linear attention (Katharopou- los et al., 2020), can implement stochastic gradient descent to perform ICL. Other works (Xie et al., 2021; Wang et al., 2024; Ye et al., 2024) have presented a Bayesian frame- work to theoretically explain the workings of ICL. This view implies that the model implements a two-stage algo- rithm to estimate the posterior P(=[D) and the likelihood Pty. D). In this work, adopt this framework and demonstrate how the model implements it through its in- termediate representations. More specifically, we study the emergence of task encoding and decoding.
Our main contributions are as follows:
1. We first study the emergence of task vectors during training under the encoder-decoder framework. By training a small transformer on synthetic ICL tasks ($3.3), we observe the coupled occurrence of task coding and task decoding, ultimately forming task vec- tors.
2. We introduce Task Decodability (TD) as geometric measure to quantify how well the model can infer the latent tasks from its representations and demonstrate that TD effectively predicts downstream ICL perfor- mance in pretrained LLMs ($4.2). We demonstrate our framework's generality across tasks, model families, and scales (Llama 3.1 8B/70B, Gemma 2B/9B/27B) and further study its evolution throughout pretraining using different checkpoints of OLMo-7B.
3. We establish the causal relationship between TD and ICI. performance in pretrained I.LMs through mech- anistic intervention ($4.1) and controlled finetuning ($4.4). Contrary common convention, we show that finetuning the earlier layers improves ICL performance more than finetuning the later layers (Wu et al., 2024; Kumar et al., 2022).
general encoding- mechanies of ICL.
Beyond the scope of task-specific vectors, several studies have explored how language models encode wide range of latent concepts (Dalvi et al., 2022; Merullo et al., 2023). including truthfulness (Marks & Tegmark, 2023), time, and space (Gurnee Tegmark, 2024). These investigations re- veal that such notions can be linearly separable the hidden
4. We offer an unifying perspective on how the learning signal of more in-context examples, finetuning, and prompting ($4.5) materializes in L.Mss.
2

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
representations, and that model scaling often yields more interpretable features (Bricken et al., 2023; Cunningham et al., 2023)
illustrate that task-specific vectors are then decoded and trigger the desired ICL task behavior. With simple encoder- decoder analogy, these findings suggest that the two-stage behavior of ICL, as described in Equation (1), is mediated by the encoding and decoding of latent variables within the representation space. Building on this idea, we begin our investigation with the following questions:
Mechanistic Interpretability. To study the causal rela- tionship between the quality of task ancoding-decoding and downstream ICI. performance, analysis techniques from Geiger et al. (2020); Vig et al. (2020); Todd et al. (2023); Heimersheim & Nanda (2024); Merullo et al. (2024). We specifically use the method of activation patching, where we replace the activations of an intermediate layer from a sample with another.
adopt causal mediation
How do task vectors emerge during training, and what drives their varying effectiveness across tasks?
2. How is the model's 's ability to accurately infer the latent tasks related to downstream ICL performance?
3. Understanding In-context Learning
3.3. Motivation: Synthetic Experiments
3.1. Notation and Background
As motivating experiment, we study the formation of task vectors during training dynamics of small autoregres- sive transformer on synthetic ICL tasks. We observe that, as the model "localizes" the latent task by building a dis- tinct representation from the others, it associates with a uniquely corresponding decoding algorithm. Building on this observation, we outline the task encoding and decoding framework to explain the formation and mechanism of task vectors.
We focus on ICL problems, where the goal is to pre- dict y, from a query Iv, given some in-context examples D = = [(x, 34) )= Each problem shares a latent task z that links inputs æ to outputs y. For instance, in an ICL task where the latent task is object-color mapping, we pro- vide demonstrations like (apple, red). (banana, yellow), and (grape, purple), and then ask for what comes after (lemon, 2). We employ this parameterization to accommodate latent tasks varying in complexity, from simple function regres- sion problems (Garg et al., 2022; von Oswald et al., 2023; Li et al., 2023) to Part-of--peeed (POS) tagging (Blevins et al., 2022; Banko & Moore, 2004) and bitwise arithmetic (He et al., 2024).
Task. We compose our task mixture of sparse linear regression. We follow the conventional linear regression setup from Garg et al. (2022); von Oswald et al. (2023)
and construct the input-output pair (æ:, ys) ) by sampling 24 ~r~ N0, ID) and y; = w x + where W is randomly generated from a standard normal distribution, (O, ID), and E ~ A((0, o2). We, however, add sparsity constraints to W with the sparsity pattern represented by the basis B Each B& has rank of In other words, the basis chooses the dimensions of W to turn on and off. The basis is sampled uniformly from B = { B2, B3, B,} and each basis is non-overlapping and orthogonal to each other. By default, we set 16 and r 4. By adding this layer of latent concept of B, we can explicitly control and interpret the latent concepts, and analyze their representations.
3.2. Theoretical Framework
Of the many different frameworks (Bai et al., 2024; Min al., 2022; von Oswald et al.. 2023; Akyürek et al., 2024) to understand the workings of ICL., we adopt the Bayesian view (Xie et al., 2021; Mittal et al., 2024; Wang et al., 2024; Ye et al., 2024). It proposes that transformers implicitly infer the latent variable z underlying the demonstrations and apply it to generate an answer. More formally,
ply+ | (x, D) = / Po x), Po(z D) dz (1)
Model and Training. We train a 12-layer GPT-2 architec- ture transformer (Radford al., 2019) with an embedding dimension of 128 and 8 attention heads. We train the model to minimize mean squared error (MSE) loss over the se- quence length of 20. We run 5 different random seeds for training and report observations that generalize across the runs. We detail the experimental setup in Appendix D.2.
This framework suggests ICL is a two stage process. First, latent concept inference. Latent concept is approxi- mated from D through the distribution ~ P=(DD) Sec- ond, selective algorithm application. The model applies an algorithm conditioned on å to predict y. as given by Po(
Although theoretically compelling, it was not until recently that Hendel et al. (2023); Todd et al. (2023); Merullo et al. (2023) showed empirical evidence of models encoding the latent concepts in the intermediate representations. They
Theoretical Error Bounds. The error bounds of regression depend on whether the model learns to infer the underlying bases. If the model can infer the bases, then the model can theoretically achieve r-dimensional
3

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Loss by Basis
(a)
(b)
(c)
1.0
UNA
(a)
Test MSE
0.8
(b)
1.0
1,0
0.4
(c)
MSE
0.5
3.5
0.5
0.0
0.0
0.0
10
Epoch
# of demonstrations
81
Bg
B,
Perfoct 4-dim
Perfect LSE 16-dim
Cptimal-wcighttdd Lasso 12-dim
Figure 2. Coupled emergence of task encoding and conditional decoding algorithms in mixture of sparse linear regression. The loss curve on the left-hand side shows different convergence dynamics per basis and show three phases of descent, which we mark with (a), (b). and (c). The test MSE is the mean squared error computed over sequence of 20 in-context examples. On the right-hand side, we plot geometric changes the representations and how they separate by basis these marked points. These points coincide with the algorithmic switching behavior. For the UMAP visualizations, we randomly draw 100 samples for each basis. Next, we collected y-token representations of 5th layer at 20th demonstrations and plotted the UMAP with the parameters neighhors= and min dist=0.1.
regression, where the MSE approaches with r in-context examples. If not, the model, in the worst case, can perform D-dimensional regression with r-sparsity, which has longer tailed error curve that approaches with D in-context examples. With these insights, we can better analyze which latent basis the model has learned and the associated algorithm from its error curve. Note that we define "algorithm" as a class of statistical methods for linear regression, as detailed in Appendix D.I.
to the optimal 4-dimensional regression. This observation suggests that model encodes the tasks into separated repre- sentations to conditionally apply decoding algorithms.
Causal Relation between Task Encoding and Perfor- mance. We conduct perturbation analysis validate that the model conditionally applies decoding algorithms based on the separated representations. Given an input of source basis, we patch the activations of layer 5 representations of the residual stream of the transformer layer - with the mean activations a target basis and analyze whether it will improve or degrade performance. We specify layer for this activation patching analysis because the separation of representations by concept is only clearly observed from that layer and afterwards based on the UMAP visualizations in Figure 11. We formally describe the activation patch- ing procedure in Appendix C. When the source is equal to the target selffpeetuubbation), the patching should help the model identify the basis and improve performance. Other- wise, it should hinder correct basis inference and therefore degrade performance. We perform this analysis different steps of training (b) and (c) from Figure 2, when the latent task representations are semi and fully separable.
Observation 1: Different Loss Dynamics per Basis. We interestingly observe that each basis, despite having identi- cal task complexities, exhibits different loss descents during training. Figure 2 shows the test MSE averaged over the sequence over training. B1 displays distinct loss descent dynamic, undergoing an abrupt drop at epoch 10. In contrast, the other three bases, B2, B3, and B₁. exhibit correlated loss descent dynamies, with two smaller descents 10 and 40 epochs. This suggests that the model learns to infer B differently and applies selective decoding algorithm.
Observation 2: Emergence of Separable Representations and Coupled Algorithmic Phase Transitions. We also an- alyze the geometry of the intermediate representations at layer 5 to question how the model may be encoding the latent bases. Surprisingly, at the three points of descent (a. b, c) marked in Figure 2, the model gradually builds separate representations for the different bases shown in the UMAP visualizations. At point (a), the three bases are clustered together and the model's algorithm resembles a 16- dimensional weighted LASSO regression. As B1 separates out at point (b), the model starts to leverage the inferred basis to switch to 4-dimensional regression. At point (c), when all four classes are separable, the model converges
In Figure 7 of Appendix D.3, we present the perturbation analysis at point (b) on the left. In this case, B2,,,4 forms one cluster and B₁ another. We observe that all the self- perturbations along the diagonal and intracluster (B2.3,4) slightly decrease the loss show effect. However, when we apply perturbations across different clusters, the loss spikes, indicating that we trigger different decoding algo- rithms unsuitable for the input sequence. This analysis shows that, because the model was only able to encode two different latent concepts the intermediate representations,

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
only learns two classes of algorithms, one for B1 and another for B2,3,4-
each attention head and measure the resulting change in MSE. Figure 17 shows that in the mixture of regression tasks, there is consistent sharing of attention heads across linear regression, leaky ReLU, and sparse linear regression tasks, excluding quadratic regression. This contrasts with the head pruning result of the sparse linear regression in Figure 18, where each basis seemingly operates with a dif- ferent set of attention heads and algorithm. This suggests structural similarity in the algorithms in this new mixture setting and provides direct evidence that they are struc- turally indistinguishable within the model, supporting our hypothesis.
On the right of Figure 7, we conduct the same study at convergence, when the model learns to encode all of the latent concepts as distinct representations. Surprisingly, we observe that the model undergoes an algorithmic phase tran- sition of implementing individual task-specific algorithms. Not only does all the self pelftpetturbaiio along the diagonal improve performance more noticeably, but also any per- turbation to different basis results in significantly higher losses.
These results altogether draw the picture that a transformer, when trained to perform ICL, gradually learns to encode the latent tasks into separable representation spaces and learns to conditionally apply decoding algorithms simultaneously These observations suggest that task encoding and decoding are mutually dependent.
3.4. Task Encoding-Decoding
Based on the observations above. we define task encoding and decoding that serves as the core framework throughout the paper. The formal definition is in Appendix B.
Definition 3.1 (Task Encoding and Decoding). Over training, transformers learn separable representa- tions by latent tasks task encoding. Simultane- ously, the model learns task-specific algorithms by leveraging the separable representation spaces - task decoding. We illustrate that these two processes combined manifest as task vectors, and that they emerge concurrently during training.
Generalizability to Increasing Dimensions and Basis Overlap We investigate this coupled emergence of task encoding and decoding under more complex settings. To this end, we increase the number of tasks and introduce non non-orthogonal bases with overlaps (i.c.. some bases are correlated). We describe the specific details in Appendix D.4. First, we, once again, observe the coupled emergence of task encoding and decoding in these more complex tasks in Figure 13 of Appendix D.4. Second, as shown in Fig- ure 15 of Appendix D.4, more intriguing pattern emerges: bases that share overlap and are correlated, even at conver- gence, are not fully separated in the representation space and share the same loss over training. This suggests that, in natural text, where tasks are often correlated and exhibit se- mantic overlaps, the model may similarly fail to disentangle their representations and distinguish these tasks.
4. Towards Natural Experiments
In this section, we empirically study the task encoding- decoding phenomenon in pretrained L.Mss. Specifically, we test several hypotheses driven by the the proposed task prodding-eccddng framework and demonstrate that our geometric measure- called Task Decodability can serve as proxy of the quality of task vector and ICL performance.
Generalizability to Mixture of Various Regression Tasks To further validate the robustness of the task encoding- decoding framework, we trained transformer using a mix- ture of various regression tasks (linear, sparse linear, leaky ReLU, and quadratic regression) from Li et al. (2023) de- tailed in Appendix F. In Figure 16, the model successfully learns linear, sparse linear, and leaky ReLU regression but fails at quadratic regression. Despite strong ICL. perfor- mance, intermediate representations across the regression tasks in intermediate layers have high overlap. We hypothe- size that task vectors do not naturally emerge here because these tasks could be solved with the same core linear regres- sion algorithm (Li et al., 2023).
Tasks, We construct two classes of algorithmic tasks natural language processing and arithmetic - comprising a total of 12 tasks. Within each class, the tasks designed to be semantically similar, ensuring that the input distributions are alike across tasks. While the underlying tasks differ (c.g., different arithmetic operations or linguistic patterns), the surface features of inputs remain consistent. By keeping the input distributions similar, we can effectively assess the model's ability to infer and encode latent concepts based solely on subtle differences in the data, rather than simply relying on the input variations. Refer to Appendix G for more details.
To verify the conjecture, we perform attention head pruning experiments detailed in Appendix F and report that the dif- ferent regression families, excluding quadratic regression, share the same decoding algorithm. We sequentially prune
Sprechf-Speech (POS) tagging. We construct a POS tagging (Blevins et al., 2022; Banko & Moore, 2004) dataset from Marcus et al. (1994), consisting of POS tags, such as Noun, Adjective, Verb, Adverb, Preposition, and Pronoun. Given
5

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
an input text and hidden POS tag =; (e.g., Noun), one needs to output the first word that is of the specified POS tag.
S ()), and find its k nearest neighbors (with k 10) in the remaining set. We then use majority voting on these neighbors' task labels to produce predicted label 2. If 2 = 2, classification for the task label is correct. The TD score at this layer is the fraction of query points classified correctly. TD = [P] E(h = 8].
Bitwise arithmetic. We construct a bitwise arithmetic dataset consisting of 6 different operators, AND, NAND, OR, NOR, XOR, and XNOR. Given a pair of 5-digit binary numbers and the hidden operator 2; (e.g., AND), one needs to output the resulting binary number after the operation.
(a) POS Tagging
(b) Bitwise Arithmetic
For both of these tasks, create additional Null class, for which there is no latent concept. In bitwise arithmetic, the Null operator outputs random binary digits, and in POS tagging, the Null class pairs the input sentences with a randomly selected word. This task helps us identify the cases in which the model is confused about the concept.
T-anot
10atet
-Nuu
+Adjective
Verb
NOR
80
TD Score
Model. We use Llama-3. 1.1-a-3.1-8B 1-8B as the default model for the experiments. We test our hypothesis across different families and scales (Gemma-2 2 2B/9B/27B and Llama-3. I- 8B/70B) in Section 4.1 and Appendix G. For analyzing the models over the course of pretraining,
Adverb
Proposition
Pronoun
NAND
XOR
XNOR
65
66
60
60
of demonstrations
demonatrations
use the check- points of OLMo 7B (Groeneveld et al., 2024). We do not train any model, except when we study the causal effect of task decodability by finetuning in Section 4.4. We further detail the experimental setup in Appendix G.
Figure 3. Task encoding in Llama-3.1-8B. (Top) UMAP of the
intermediate representations and (Bottom) TD scores for each tasks at layers 15 and 13, respectively, for POS tagging and bitwise arithmetic with varying numbers of in-context examples.
Evaluation. We evaluate the performance of the model on different tasks by computing the exact-match accuracy between the generated output under greedy decoding and the ground truth. All of the ovaluations assume 4-shots of examples, unless specified otherwise.
4.1. Task Encoding and Decoding in Pretrained LLMs
Hypothesis 1: In pretrained LLMs, task decodabil- ity varies across tasks and determines the effective- ness of the task vector.
Task Decodability (TD). To quantify how the task encod- ing progresses, we employ a simple k-Nearest Neighbor (k-NN) classification metric. Inspired by prior studies us- ing linear probes (Rimanic et al., 2020; Alain & Bengio, 2018), 8), we assess whether the latent task can be extracted in simple manner from their representations. Specifically, we use the representations of the token immediately before y. at a chosen layer and predict the latent task by majority voting among its k nearest neighbors (k: 10, N 100). For choosing which layer measure TD, we compute TD across all layers and choose the layer that best encodes the task. We empirically validate with UMAP visualizations that the representations become separable precisely when the TD score peaks (See Figures 10, 11 and 19).
Task decodability varies across tasks. We begin by quali- tatively analyzing the intermediate representations of LLMs across different tasks using UMAP. As shown in Figure 3, the degree of localization of representations depends on the task. For instance, tasks like AND, OR, Noun, and Pronoun exhibit clear and distinct clusters when sufficient in-context examples (c.g., 10-shots) are provided, while tasks like XNOR and XOR in bitwise arithmetic or Ad- jective and Preposition in POS tagging remain overlapped with the Null class. This suggests that the model inherently learns certain tasks better than others during pretraining, likely due to differences in their predictability or frequency in the pretraining corpus. Importantly, the emergence of sep- arability with increasing examples highlights that in-context examples act as a signal for materializing task-specific rep- resentations.
We now formally define TD. Let T be the set of latent tasks. For each task 2 E T, sample N 100 datapoints U.) and collect the intermediate representations (ha),2 from chosen layer (e.g., the token embedding immediately after s.;). Label each representation h; with the corresponding task 2. This yields a set S = U.ET (he, i= 1,..,,, ... over all tasks 2.
To better quantify such variability of representational sep- arations between different tasks in Figure 3, we use our proposed Task Decodability (TD) scores. Analogous to the UMAP visualizations, we observe that some tasks, such as Noun and Pronoun in POS tagging and AND and OR in bitwise arithmetic, are much more decodable from their rep-
Given query point (he, 2), we exclude (h,, 2) from S (i.e.,
6

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
resentations than the others. Also, we confirm in Figure 19a of Appendix G.1, TD score peaks in the middle, suggesting earlier layers encode the task and latter layers execute the decoding algorithm.
(a) POS Tagging
(b) Bitwise Arithmetic
houe lel. 10-dh
thi
AND on xvoon
1ohd 12atsi
ICL Accurrary 0%
Adocii
ICL Accuracty
39
24
Task decodability as an indicator of task vector effec- tiveness. Through mechanistic interventions, we further examine how TD represents the quality of the task vector formation how effective injecting the task vector is at trig- gering the desired ICL behavior. By patching layer outputs with either mean activations from correctly inferred latent concepts (positive intervention) or the Null class (negative intervention), we measure their impact on task performance. As shown in Figure 20, tasks with highly separable rep- resentations, such as Noun and Verb in POS tagging and AND and OR in bitwise arithmetic, exhibit significant im- provements from positive interventions (up to 14%) and noticeable degradation from negative interventions (up to 15%)
14
12
0.781
R) 0.525
940
072
03 02
05
TD Soore
TD Score
Figure 4. TD score vs ICL performance in Llama-3.1 1 8B. We observe positively correlated trend across most tasks. The grey dashed lines are linear lines of best fit. These results suggest that the accuracy of task encoding is closely coupled with downstream ICL performance.
the respective tasks. Notably, referring back to Figure 3, we again remark that the representations of some classes (Adjective and Preposition in POS tagging and XOR and XNOR in bitwise arithmetic) are mapped those of the Null class. We notice that this set of classes whose repre- sentations overlap with those of Null generally have low task performance and do not improve as much as the others given more demonstrations. We conjecture that the model does not accurately encode the latent tasks of those that are overlapped with the Null class representations.
In contrast, tasks with overlapping representations, such as XOR, XNOR, Adjective, and Preposition, show lim- ited sensitivity to interventions, with positive interventions improving performance by only 2% and negative inter- ventions causing a modest 6% drop. These differences suggest that when task encoding is done correctly and the representations are well-separated, the task vectors are more effective at guiding decoding processes, as they can eas- ily map distinct concepts to their respective downstream algorithms. Conversely, overlapping representations hin- der the ability of the task vectors reliably trigger distinct decoding algorithms, reducing their overall effectiveness.
We also test the generality of our hypothesis that TD predicts ICL performance across different model families and scales. We perform the same analysis on Gemma-2 2B, 9B and 27B (Google, 2024) and 1lama-3.1 70B and present the results in Figure 21 in the Appendix G.3. These results demonstrate that the correlation between TD and ICL performance robust across models and tasks. Interestingly, in all of the Gemma-2 family and Llama-3.1 70B models, Noun, Pro- noun, and Verb show the clearest signs of task encoding and decoding behavior, as we saw in the 1lama-3.1 8B model. In the bitwise arithmetic task, AND, NAND, OR, and NOR (classes that showed the strongest havior in 1lama-3.1 8B), also show the strongest signs of task encoding- -dceoiiig-g behavior across all of these mod- els. Given that many LLMs are trained on similar sources of pretraining data (Soldaini et al., 2024; Gao et al., 2020) (CommonCraw!! Wikipedia, etc.), we conjecture that the models may have learned similar task vector mechanisms for these concepts (Huh et al., 2024).
Overall, these findings demonstrate that task decodability is direct reflection of task vector quality. This implies that, when the model becomes more adept at a task, the task becomes correspondingly easier to decode from their representations. We further assess this hypothesis in the next section.
4.2. Task Decodability Predicts ICL Task Performance
-eecoddngg be-
Hypothesis 2: Quality of task encoding-decoding is predictive of ICL performance.
We now investigate the second hypothesis of whether the quality of task encoding-decee -eeoodigdeeooiigg is predictive of down- stream ICL performance. If the model is conditionally ap- plying a decoding algorithm by first inferring the latent task, the quality of the latent task encoding (measured by TD score) and ICL task performance should be closely corre- lated. To this end, we analyze the relationship between TD and test accuracy in Figure 4. In both datasets, we see that, higher TD scores correspond to better performance on
4.3. Studying the Emergence of Task Encoding and Decoding in LLM Pretraining
Another natural question is whether the simultaneous emer- gence of ICL ability and task encoding observed in synthetic experiments in Section 3.3 also occurs during L.MM pretrain-

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
(a) POS Tagging
(b) Bitwise Arithmetic
POS Tagging
Bitwise Arithmetic
Avg. TD
Avg.
Ace.
Avg. TD
Avg. Acc.
10.
-0.30
Apouraay
Spore
Apcuracy
Pretrained
0.68 i o. 25.5 6.7 0.43 0.13 26.2 +: 15.5
9.7
0.4
0:15
FT (first 10) 0.95 = 0.02 69.4 + 142 0.85 10.11 90.2 + 15.0
TD
FT (last 10) 0.68 = 0.11 33,9 # 12.3 0.43 0.13
66.3 # 9.01
04-
0.
0.00
Prompting 0.97 +: 0.02 34.4 + 10.3 0.90 +: 0.10 57.2 4 30.1
10
10'
16
Step
Step
Meen TD Soore tasks
Accurasy tasks
Tahle 1. Average TD scores and task accuracies with finetuning and prompting for POS Tagging and Bitwise Arithmetic.
Figure 5. TD scores and ICL accuracy during OLMo-7B pre- training (Groeneveld al., 2024), averaged across POS tagging and bitwise arithmetic tasks, evaluated using 4-shot prompts on 1000 test examples per task. The shaded regions represent the standard deviation across the tasks.
Given our observation that the earlier layers are responsible for task encoding and the latter layers for the decoding al- gorithms, we hypothesize that finetuning the earlier layers will enhance task representation and improve ICL task per- formance more effectively than finetuning the latter layers. This approach challenges the common assumption in model finetuning (Wu et al., 2024; Kumar et al., 2022), where the focus typically lies on adjusting the latter layers or the linear head, under the belief that the final layers primarily gov- em task-specific adaptation. However, our findings reveal that targeting the earlier layers provides more effective pathway for improving ICL performance.
ing. Since large-scale pretraining studies are computation- ally infeasible, we leverage different training checkpoints of OLMo-7B (Groeneveld et al., 2024) to investigate the relationship between TD and ICL task performance during pretraining. As shown in Figures 5 and 6, increases in TD scores closely align with gains in ICL accuracy. Interest- ingly, unlike the synthetic setup, this progression is more gradual. This suggests that task encoding and decoding for natural ICL tasks emerge more gradually during pretraining, likely because LLMS are simultaneously learning a diverse range of tasks, making the training dynamics more intricate to disentangle and comprehend. Thus, further investiga- tion into how task-specific representations evolve during pretraining is warranted.
Finetuning only the last 10 layers results in minimal im- provements to task encoding, as these layers do not change the upstream representations. As illustrated in Table 1, the TD scores of the last 10 layers remain unchanged compared to the pretrained model. In contrast, finetuning the first 10 layers significantly improves TD scores, aligning the repre- sentation subspaces with the latent concepts and enhancing task abstractions.
Generalization to Recurrent Language Models Recent work shows that recurrent language models, such as LSTMs (Xie et al., 2021) or Mamba (Gu & Dao, 2023), exhibit ICL behavior (Grazzi et al., 2024). To verify whether our encoding-decoding view explains the emergence and ef- fectiveness of task vectors in such models, we performed the same analysis on Mamba 8B (see Appendix G.4). As shown in Figure 22, strong positive correlation between TD scores and task performance confirms our hypothesis that TD scores predict ICL performance for state-space mod- els as well. This implies that the proposed task encoding- decoding perspective could potentially be generalized across various architectures beyond transformers.
This improvement in representation encoding directly trans- lates to better downstream ICL performance. With 4-shot examples, finetuning the first 10 layers outperforms fine- tuning the last 10 layers by 37% in the POS task and 24% in bitwise arithmetic. Moreover, finetuning the first 10 lay- ers achieves near-perfect accuracy across bitwise arithmetic tasks, except for XNOR, where overlapping representations with Null limit further improvement.
4.5. Investigating the Effect of Prompting on Task Encoding and Decoding
4.4. Improving Task Encoding by Finetuning the Early Layers
Hypothesis 4: Prompting enhances TD by providing stronger learning signal for task inference, and thus improves ICL performance correspondingly.
Hypothesis& 3: Building on the encoder-decoder framework, finetuning the early layers enhances task encoding and should yield greater improvements in ICL performance compared to finetuning the later layers.
In previous sections, we showed that in-context examples and finetuning both improve task encoding and hence ICL task performance. Prompting (Liu et al., 2023) is also a sim- ple and common method to improve a model's 's performance. In this section, we experiment with prompting to see how
8

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
changes TD along with performance. As shown in Table prompting in fact improves the task encoding and perfor- mance simultaneously. Together with the previous results, it suggests that enhancing task encoding may be the unifying principle through which the learning signal materializes in the model's representations across different strategies (e.g., in-context examples, finetuning, and prompting). We detail the setup and results in Appendix I.
of complexity in Appendix D.4, further studies are essential to apply our findings and insights the real-world. Another limitation stems from proposed TD metric. Since we measure the separability from one task to the others, for the measure to be meaningful, the distribution of tasks on which TD is computed needs careful design to consist of semantically similar, confusing tasks.
Impact Statement
5. Discussion
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
Our insights on the origins and mechanism of task vec- tors have several implications in light of recent works on understanding the mechanies of ICL (Mittal et al., 2024) and activation-steering methods (Bürger al., 2024; Pan- ickssery et al., 2024; Marshall al., 2024).
Why do models succeed at some ICL tasks, but not oth- ers? It is yet puzzling how to categorize the types of ICL tasks LLMS et al., 2023). An intuitive explanation is that the model can effectively encode the tasks frequently seen during pretrain- ing (Razeghi et al., 2022; Li et al., 2024). In our experiments, we observe patterns consistent with this conjecture, where AND and OR, the more common logical operators in lan- guage, were encoded more accurately. However, under our proposed two-stage mechanism, show the bottleneck in ICL tasks can exist in both levels of task inference and sub- sequent decoding algorithms. Therefore, even if the model already learned the algorithm for task, if the model cannot clearly distinguish the latent concept from the inputs, will fail, and vice versa.
and cannot solve (Qiu et al., 2023; Dziri
Does learning the right latent variables help? Mittal et al. (2024) investigate whether explicitly modeling the la- tent variables ICL outperforms implicit learning through ordinary autoregressive training with transformer. They draw the counterintuitive conclusion that explicit modeling does not enhance performance, albeit not worse; the under- lying reasons for which remain unclear. In our work, we explain this specific observation by analyzing the extent to which implicit modeling (standard transformers) captures the true latent variables. Our findings show that transformers can inherently encode these latent variables without explicit regularization. Therefore, we propose that the comparable performance between explicit and implicit models arises not because modeling the latent variables is unhelpful, but because both types of models effectively leam them.
Limitations. A limitation of our work is that the experi- mental setup used in this study does not encompass tasks that require multi-step reasoning (Clusmann et al., 2023; Zu- pan et al., 1999; Hosseini et al., 2024). Although we analyze the task srooding-decodiing mechanism with varying levels
9

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
References
Clusmann, J., Kolbinger, F. R., Muti, H. S., al. The future landscape of large language models in medicine. Communications Medicine, 3:141, 2023, doi: 10.1038/ 023-56-033-0033300 1. URL https::///// /ttp:::///ooooe/// org/0.. 0.38/ (055//43856--- 020-0-0002000000001-1000
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Trans- formers learn to implement preconditioned gradient de- scent for in in-context learning. Advances in Neural Infor- mation Processing Systems, 36, 2024.
Cunningham, H.. Ewart, A., Riggs. L.. Huben, R., and Sharkey, L.. Sparse autoencoders find highly interpretable features in language models, 2023. URL https:// arxiv. wwriii.rrg///oo htggabb/////aa abs 2209 08600.
Ajay, A., Han, S., Du, Y., Li, S., Gupta, A., Jaakkola, T., Tenenbaum, J., Kaelbling, L.., Srivastava, A., and Agrawal, P. Compositional foundation models for hierar- chical planning, 2023. URL https:://///.. https://arxxx: abs/ 2339 08587.
Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can gpt learn in-context? language models implic- itly perform gradient descent as meta-optimizers, 2023. URL https: /aw.ivivvvv.//////// http://wrggrbbb//......
Akyürek, E., Wang, B., Kim, Y., and Andreas, J. In-context language learning: Arhitectures and algorithms. arXiv preprint ar arXiv:2401.12977 2024.
Dalvi, F., Khan, A. R., Alam, F., Durrani, N., Xu, J., and Sajjad, H. Discovering latent concepts learned in bert. arXiv preprint arXiv:2205.072331 2022.
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes, 2018. URL https:// arxiv. org/ abs / 016.4114....
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., West, P., Bhagavatula, C.. Bras, R. L., Hwang, J. D., Sanyal, S., Welleck, S., Ren, X., Ettinger, A., Harchaoui, Z. and Choi, Y. Faith and fate: Limits transformers on compositionality, 2023. URL https: http///xxvvvv..r// abs / 2200 1865.
Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Trans- formers statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural infor- mation processing systems, 36, 2024.
Banko, M. and Moore, R. C. ofrt-hf-speech tagging in con- text. In COLING 2004: Proceedings of the 20th Interna- tional Conference on Computational Linguistics, pp. 556- 561, Geneva, Switzerland, aug 23 au-aug2 27 2004. COLING. URL https::t/// / ht///aaaanntttoooooooooo org/ C 1080.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. 2020. URL https: / //arxiv. org/ abs 00027.07....
Blevins, T., Gonen, H., and Zettlemoyer, Prompting language models for linguistic structure. arXiv preprint arXiv:2211.078300 2022.
Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What transformers learn in-context? case study of sim- ple function classes. Advances in Neural Information Processing Systems, 35:30583-30598 2022.
Bricken, T., Schaeffer, R., Olshausen, B., and Kreiman, G. Emergence of sparse representations from noise. 2023.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
Geiger, A., Richardson, K., and Potts, C. Neural ural language inference models partially embed theo- ries of lexical entailment and negation. arXiv preprint arXiv:2004. 1atr200044.4623,, 2020.
J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., HosberttVosss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler. D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. wrg/abs///....oo.o....o......... 2005 14165.
Google. Gemma 2: Improving open language models a practical size, 2024. URL htt.t://///// http::///axxv.oopp//p abs 00../248.0000 00118.
Grazzi, R., Siems, J., Schrodi, S., Brox, T., and Hutter, F, Is mamba capable of in-context learning?, 2024.
Bubeck, S., Chandrasckaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. Li, Y., Lund- berg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early experi- ments with gpt-4, 2023. URL https: http://xxvvvvo//// org/ 0bs/2303..m .1202....22...
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N.,
Bürger, L., Hamprecht, F. A., and Nadler, B. Truth is uni- versal: Robust detection of lies in llms. arXiv preprint arXiv:2407. 12831, 2024.
10

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Richardson, K., Zettlemoyer L.,, Dodge, J., Lo, K., Sol- daini, L.., Smith, N. A., and Hajishirzi, H. Olmo: Ac- celerating the science of language models, 2024. URL btttps://arae
Kumar, S., Marjieh, R., Zhang, B., Campbell, D., Hu, M. Y., Bhatt, U., Lake, B., and Griffiths, T. L.. Com- paring abstraction in humans and large language mod- els using multimodal serial reproduction, 2024. URL https: / a www.ii.wr....ooo.. www.///////////eo//////////eeeee 2402 03618. 8.
org/ abs/2402 00838.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
Li, X., Tang, H., Chen, S., Wang. 7.,, Chen, R., and Abram, M. Why does in-context learning fail sometimes? eval- uating in-context learning on open and closed questions. arXiv preprint arXiv:2407.02000 2024.
Gurnee, W. and Tegmark, M. Language models represent space and time, 2024. URL https: orxiv.org/ abs/23100 02207.
Li, Y., Ildiz, M. E., Papailiopoulos, D., and Oymak, S. Transformers algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pp. 19565-19594, PMLR, 2023.
Han, S., Shenfeld, I., Srivastava, A., Kim, Y., and Agrawal, Value augmented sampling for language model alignment and personalization, 2024. URL https: //axxiv...vvv org abg/abs//e 2405 06639.
He, T., Doshi, D., Das, A., and Gromov, A. Learning grok: Emergence of in-context learning and skill com- position in modular arithmetic tasks. arXiv preprini arXiv:2406.02550, 2024.
Liu, X., Wang, J., Sun, J., Yuan, X., Dong, G., Di, P., Wang, W., and Wang, D. Prompting frameworks for large language models: A survey, 2023. URL https: / wwwxiv..rrg/ae abs /ttp//21...2211
Heimersheim, S. and Nanda, N. How to use and interpret ac- tivation patching, 2024. URL httpt:tpp//// wtl.https:///aaee https//aaxvvvorrree abs/2404 15255.
Marcus, M., Kim, G., Mareinkiewicz, M. A., MacIntyre. R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The Penn Treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL https: haaaaaaaannoooooooo org/H944 #rg/H94-100
Hendel, R., Geva, M., and Gloherson, A. In-context learning creates task vectors. arXiv preprint urXiv:2310.15916 2023.
Hong, R., Zhang, H., Pan, X., Yu, D., and Zhang, C. Abstraction-ofthhuuhh makes language models better reasoners. arXiv preprint arXiv:2406.124: 2024.
Marks, S. and Tegmark, M. The geometry of truth: Emer- gent linear structure in large language model representa- tions of true/false datasets. arxiv, 2023.
Hosseini, A., Sordoni, A., Toyama, D., Courville, A., and Agarwal, R. Not all Ilm reasoners are created equal. arXiv preprint arXiv:2410.01745 2024.
Marshall, T., Scherlis, A., and Belrose, N. Refusal in llms is an affine function, 2024. URL https://arxiv: org abs abs/2411........ 09003.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL https: : //axiiv. org/abs/ 2106. 09685.
Merullo, J., Eickhoff, C., and Pavlick, E. Language mod- els implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305. 16130. 2023.
Merullo, J., Eickhoff, C., and Pavlick, Cireuit component reuse across tasks in transformer language models, 2024. URL https: / www.i.vvvooo orgg abs / 231 0. 08744.
Huh, M., Cheung, B., Wang, T. and Isola, The pla- tonic representation hypothesis, 2024. URL https: //arxiv. httgg//bb/24455. 07987.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learn- ing work?, 2022. URL https: htapps://ariiv..e /awaaaxxvvvororoe orr//bs//////oooo////////o/ abs/ 2202. 12837.
org/ htg/abs// 2006 16236.
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL https://arxiv. org/ abs/ 1412. 6980.
Mittal, S., Elmozzino, E., Gagnon, I.., Bhardwaj. S., Sridhar, D., and Lajoie, G. Does learning the right latent variables necessarily improve in-context learning? arXiv preprint 1rXi::245519162. 2024.
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and un- derperform out-of-distribution, 2022. URL https: //aaxii..rgaaas////// wtgg/bbs//200 10054.
Panickssery, N., Gabrieli, N., Schulz, J., Tong, M., Hub- inger, E., and Turner, M. Steering llama 2 via
//arxiv.

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
contrastive activation addition, 2024. URL https: //arxiv.org abs 0662..
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtow- icz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. Transform- ers: State-of-the-ar natural language processing. Liu, Q. and Schlangen, (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. doi: 10. (06653//1/2220mmm//pp/oooo //aclanthology' hwwp:aatthhooog..oo//2222222 wwt.ooogg//ooooooo
Park, C. F., Okawa, M., Lee, A., Lubana, E. S., and Tanaka, H. Emergence of hidden capabilities: Explor- ing learning dynamics in concept space. arXiv preprint arXiv:2406.194 19370, 2024.
Qiu, L., Jiang, L.., Lu, X., Sclar, M., Pyatkin, V., Bhaga- vatula, C., Wang, B., Kim, Y., Choi, Y., Dziri, N., et al. Phenomenal yet puzzling: Testing inductive reasoning ca- pabilities of language models with hypothesis refinement. arXiv preprint arXiv:2310.cm 0.050.9,, 2023.
URL. https: emnip demos. 6.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Wu, Z., Arora, A., Wang, Z.., Geiger, A., Jurafsky, D., Manning, C. D., and Potts, C. Reft: Representation
finetuning for language models, 2024. URL https: www.a///rggaaba////e/ea/e
Raventós, A., Paul, M., Chen, F., and Ganguli, S. Pretrain- ing task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning implicit bayesian inference. arXiv preprint arXiv:2111.02088 2021.
Razeghi, Y., au2, R. I.. L.. Gardner, M., and Singh, S. Im- pact of pretraining term frequencies on few-shot reason- ing, 2022. URL htwps://///// http://aaaavvvv... http:///wwwwwwww/////ooos/es/// 2202. 07206.
Ye, N., Yang, H., Siah, A., and Namkoong, Pre-training and in-context learning is bayesian inference a de finetti. arXiv preprint arXiv:2408.03300
2024.
Zheng, H. S., Mishra, S., Chen, X., Cheng, H.-T., Chi, H., Le, Q. V., and Zhou, D. Take a step back: Evoking reasoning via abstraction in large language models, 2024. URL https: //axiiv wwwpeiiiiorrroooooooe..eoeeooeee wwt.or////ooo.... 26110.017.
Rimanic, L., Renggli, C., Li, B., and Zhang, C. On conver- gence of nearest neighbor classifiers over feature trans- formations. Advances in Neural Information Processing Systems, 33: 2020.
Zupan, B., Bohanec, M., Demšar, J., and Bratko, I. Learning by discovering concept hierarchies. Artificial Intelligence. 109(1):211-242, 1999. ISSN 0004-3702. doi: http://do.ngg//0000003004-070000 URL cccene//attaoee//////00470000000
Soldaini, L., Kinney, R., Bhagia, A.. Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159) 2024.
Todd, E., Li, M. L.,, Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. arXiv preprint arXiv:2310.15213, 2023.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., and Shieber, S. Causal mediation analysis for interpreting neural nip: The case of gender bias, 2020. URL https::///// http/aaxvvvvrr// abs/2004 12265.
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento,
J., Mordvintsew, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent, 2023. URL httphtttp//ccmc / wrwaa.avvvoooo orr. abtpg//ab//22...... 2212. 07677.
Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable mod- els: Explaining and finding good demonstrations for in- context learning. Advances in Neural Information Pro- cessing Systems, 36, 2024.
12

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
A. Investigating Predictability of ICL Task Performance in Large-Scale Pretraining
Since is computationally infeasible to conduct large-scale pretraining studies, we leverage the different training check- points for OLMo-77 (Groeneveld al., 2024) to investigate the relationship between concept decodability and ICL task performance on POS tagging. Interestingly, as shown in Figure 5, we observe a correlated emergence of the variables. This analysis shows that the coupled emergence of concept encoding and decoding algorithms may also hold in large-scat pretraining. However, this warrants further investigation, since we do not fully understand the training dynamics of a LLM.
(a) POS Tagging
Noun
Adjective
Verb
0.2
0.8
TD
0.4
0.0
Adverb
Preposition
Pronoun
0.2
0.8
TD
0.4
10"
10"
10"
10
10"
10"
10"
10"
10"
Step
Step
Step
Accuracy
(b) Bitwise Arithematic
AND
OR
NOR
0.50
0.4
0.25
Accur
TD
NAND
XOR
XNOR
0.7
0.50
0.25
TD
0,00
10"
10
10
10
10"
10"
10"
Step
Step
Step
TD Score
Accuracy
Figure 6. Test accuracy and TD scores for POS tagging and bitwise arithmetic tasks OLMo-7B (Groeneveld et al., 2024) checkpoints (1000-500000 steps), evaluated using 4-shot prompts on 100 test examples per task.
B. Task Encoding
In this section, we formally define the Task Encoding and Decoding.
Definition B.1 (Task Encoding). Let M be a transformer model, Z = 21, 22....,
... 2n be set of latent tasks, and D be in-context examples with arbitrary length K. task encoding is internal mapping F 'D - Bddmm where Redems is the intermediate representation over the model's de mbddimmension embedding space.
Definition B.2 (Task Decoding). Given a transformer model M with concept encoding E, a task decoding transformer's behavior that there exists simple function G that recover the original latent concept and condition the algorithm:
G: R demb Z
ICL performance of given = related to how well the decoder G infer the original latent variable z. To quantify this, introduce the notion of decodability. For any given decoder, define decodability as follows:
Definition B.3 (Decodability). For a given decoder dedm - and specific latent variable 2, the decodability
13

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
measures how accurately the correct latent variable is inferred from representations. Representation is distributed E(2, D), and the inferred latent variable is G(F(z,, D)).
1. One-hot Accuracy: A A1.bae (z) E[1 = =]]
2. f-divergence: Af(z) = Dy(2 || z), where f is some f-divergence metric
In our study, we employ the one-hot accuracy metric with a kNN classifier to report task decodability.
C. Activation Patching
We perform the activation patching procedure in the synthetic and natural experiments as follows. We collect representations { at selected position and layer for set of in-context examples {z 34) y1) by passing the prompt through the model and recording the activations at a specified layer after v,. We then average these activations to obtain a single "task" representation hansk = : E, hg. For new query Fiueer, we add hask into the same layer's activations cetivatioss-efeectiveey replacing the representation after Equery- and let the model continue forward.
D. Synthetic ICL Experiment
D.I. Theoretical Error Bounds in sparse linear regressions
is known that transformers can achieve Bayes-optimal solutions for linear regression problems by implementing least- squares solutions the prior of weight sampling (Garg al., 2022; Raventós et al., 2024). The least-squares estimation of linear regression with Gaussian prior for task weights can be performed using ridge regression. In the presence of sparsity, the least-squares solution can be obtained through lasso regression with optimal weight searching. The error bounds of our task depend on whether the underlying basis is discovered by the model. We consider two extreme cases:
1. If the model is incapable of inferring any basis in it would perform a D-dimensional regression with r-sparsity, where D is the total dimension and is the number of non-zero elements.
2. If the model is capable of inferring the basis in B, it can perform an r-dimensional regression adjusted for the corresponding non-zero elements of the inferred basis. In this case, the model could benefit from the tighter dimensional regression bound.
The possibility of diverse algorithms and corresponding error changes enables us to track the Bayesian inference behavior of the model in a more detailed way. the following results, we indeed observe a transition from D-dimensional regression to r-dimensional regression, accompanied by changes in the representations of tasks for each basis.
D.2. Experimental Details
Mixture of Sparse Linear Regression. We adapt the conventional linear regression setup from (Garg et al., 2022; von Oswald al., 2023) create latent bases B that we can interpret far more easily than W. We study this setting with D = 16 dimensional with up K 20 in-context examples. Each B, has rank of 4 and is orthogonal with each other. We independently sample W and for each new input sequence from (0,, Ip) the noise ~ (0, 0.01). We add the sparsity constraints to the linear regression task to introduce the latent concept of sparsity basis B that is easily interpretable and analyzable in their representations. With the sparsity constraints, we construct the graphical model B - W - Y <- X. This construction allows us to visualize the representations each the bases (latent concepts in this graph) by aggregating the representations across a set of W and (X, Y) pairs.
Model. We a 12-layer GPT-2 (Radford et al., 2019) architecture transformer, implemented by HuggingFace (Wolf et al., 2020). This model is parameterized with an embedding dimension of 256 and 8 attention heads and hasa total of 9.5M parameters.
Training. We train the model with a batch size of 128 for 80K training steps. We use the Adam optimizer (Kingma Ba, 2017) with a learning rate of le-4 and betas of 0.9 and 0.9999. We use a MSE loss over the sequence and only compute the losses on the prediction it.
14

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Evaluation.
We construct test dataset of IK samples and evaluate the model MSE loss for the predictions yi along
the sequence.
Compute.
We A100 GPU with 80GB of VRAM. To train these models, it takes about 8 hours.
D.3. Additional results
(b) Epoch 20 8; B2
(c) Epoch 100 8: - B3
B; B
B: + 8; +B,
B: - B
B: - Bg
8: - Ba
B, -+ B
Bg +
Bg + B3
B, +
By - B
By - B3
B3 - B
8, - B1
Test MSE
Test Test MSE
Bg -,
B B,
B, Bg-BB
B, B,
By - B
B1 Bg
Bi - B3
81 - B1
B4 B
8, B,
84 B4-BB
B,
B, - Bg
Bi 1 Bg
8, - B3
8, - 8
# of demonstrations
# of demonstrations
Unperturbed
Intercluster (B, - Biz))
Intracluster (B2,3,4 32,,,4
Self Bi)
Figure Causal analysis by perturbation. the left are perturbation results epoch 20, when the latent concepts' representations are semi-separate (B, and B2.3.4). Intracluster refers to B2.3.4. At this stage of training when there are only two clusters of representations, there only exists two decoding algorithms as well. On right are results al convergence, when the latent concepts' representations are fully separable. In this case, each B; follows different algorithm and patching the activations of any other basis than itself increases the loss noticeably. On the other hand, self- pertuubrrurbtiion improves ICI. performance.
Perturbation analysis to study the causal relation between task encoding and performance over the course of training.
Replicate experiments Here, we run the different seeds of synthetic experiments in Figure 2, and we report the results in figure 8. We observe that a single basis produces distinct loss trajectories for Seeds I and 2 as in Figure 2, while Seed 3 demonstrates a consistent loss descent across basis.
Loss curve (seed 1)
Loss curve (seed 2)
Loss curve (seed 3)
Test MSE
MSE
Test
0.4
0.4
0.4
0.2
02
0.2
0.0
0.
0.0
t0
t0
10
Epoch
Epoch
Epoch
(a) Replicate
(b) Replicate 2
(c) Replicate
Figure 8. Results from three replicates of experiments corresponding to Figure 2. Each subfigure shows the loss trajectory by basis by different random seeds.
15

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
D.4. Additional Analysis Section 3.3
TD Over Training. We quantified the TD score for the synthetic experiments shown in Figure 2, with the results presented in Figure 9 and Figure 10. The TD scores for Basis effectively capture the separation of representations observed at (a). An increase in TD scores correlates with corresponding drop in MSE, seen in Figure supporting our hypothesis that the TD score can serve as a predictor for the predictability of TD.
CD score over Training
1
(c)
CD Score
0.6
(b)
0.4
(a)
0.0
10
Epoch
Figure TD score of synthetic experiments in Figure 2 training. (a), (b). (c) denote the same training points in Figure
Epoch
Epoch
Epoch 100
1.0
B1
1.0
1.0
Sogres
CD Scores
0.6
0.6
0.6
0.4
0.4
0.4
0.0
0.0
3
11
3
5
9
11
5
11
Layer
Layer
Layer
Figure 10. TD score across layers epoch 10, 20, 100 from the synthetic experiment in Figure 2.
UMAP Over Training. To analyze how the representations evolve over training across the different layers in the sparse linear regression task, we visualize the UMAP of the representations in Figure 11. We see that concept encoding, the separation of representations by concept, starts to appear at epoch 20 and is only clearly observed from layer 5, Note that the layer index in the figure starts at 0, so layer 4 in the plot equals to what we call layer 5, At convergence, each of the concepts' representations becomes separated from layer 5 and later.
E. Increasing complexity in synthetic experiments
F.1. Experiment More Orthogonal Bases
We conduct an experiment with orthogonal bases, each spanning 4 dimensions out of total bases. Similar to Figure 1, we observe distinct loss curves over the bases, coupled with clear separation in the representations (see Figure X). Importantly, we observe that basis is learned first (after around 100 epochs), and basis 2 learned second (after around 200 epochs),
16

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Figure UMAP visualization of representations across the layers over training in synthetic sparse linear regression We visualize the UMAP at epochs 5, 20, and 100 across all the layers. Note that the plot uses zero-based indexing, but we one-hased indexing refer to layers all the text.
1.0
0.8
Test MSE
0.6
Basis
Basis 2
Basis 3
0.2
Basis 4
Basis 5
Basis 6
0.0
10² 2
Epoch
Figure 12, Loss curve over training 300 epochs
17

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
epoch=100
epoch=160
epoch=230
epoch-300
UMAP
bade
bade
bace
(a) UMAP of representations (layer 10) epoch 5,10,00,50
Epach = 100
Epoch 160
Epach = 243
Epoch = 300
1.0
1.2
0.8
0.5
0.8
Scaros
0.8
022
02
0.2
042
0.0
0.
5.0
7.5
10.0
0.0
2.5
5.0
7.
10.5
0.
25
5.0
7.5
10.0
0.
2.5
5.0
7.5
10.0
Layer
Layer
Layer
Layer
(b) TD score across layers at epoch = 5,10,20,50
Figure 13. Experiment More orthogonal bases analysis
while the other four bases are not distinguished by the model until around 300 epochs. Notably, it requires significantly more epochs for the model to learn each concept compared to the scenario in Figure (which uses bases on 16 input dimensions). Following our intuition, it suggests that learning concepts becomes more challenging as the number of concepts increases. Overall, these results support the idea that our proposed concept incoding-decodlnng mechanism also holds under more complex settings.
E.2. Experiment Overlapping Bases
We conduct an experiment with 8 overlapping bases, where the first 4 bases (Bases 1, 2, 3, and 4) span 8 dimensions, and the remaining 4 bases span the other 8 dimensions (with total input dimension of 16). Thus, the first four bases have overlap with another and the second bases have overlap with another. this setup, we investigate the emergence of separation both within overlapping bases (e.g., within Bases 1, 2, 3, and 4) and between the groups (e.g., between Bases 1, 2, 3, 4 and Bases 5, 6, 7, 8), and examine their relation to subsequent ICL performance.
We observe that the loss curve for each base is identical and undergoes steep descent around epoch 5 (see Figure D-2 in the link). This loss descent coincides with the separation of the two groups of bases by their representations around epoch 5, while bases within the same group remain entangled and unsorted.
These observations suggest several key points. First, the models may not learn to fully separate overlapping concepts, as they can develop shared algorithms to predict overlapping portions. Second, non-overlapping concepts can be fully separated, which accounts for the significant ICI. improvement, as it allows the development of algorithms for orthogonal (non-overlapping) concepts. Third, transformers seemingly learn classify tasks based their similarity and associate algorithms at different levels of resolution over the course of training.
F. Mixture of Different Regression Families Experiment
To further validate the robustness of the concept mncoiing-dcooding framework under more diverse and complex conditions, we conduct experiments with a mixture of regression families-linear, polynomial (degree 3), and sinusoidal regressions. Each regression type represents a distinct latent concept with fundamentally different functional relationships between inputs and outputs.
18

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
1.0
Basis
Basis
Basis
Basis
Basis
Test MSE
Basis
0.6
Basis
Basis
0.2
0.0
10
Epoch
Figure 14. Loss curve over training 50 epochs
UMAP
UMAP of representations (layer 6) at epoch 5, 10,20,550
Layer
Layer
Layer
Layer
Layer
(b) TD score across layers at epoch = 5,10,20.50
CD Scores (Between Groups)
Epoch
Epoch
Epoch 20
Epoch 50
1.00
Cluster
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.5
0.00
0.00
0.00
0.00
10
5
10
10
0
10
Layer
Layer
Layer
Layer
(c) TD score between nonoverlapping bases sets(btw basis 1,2,3,4 and 5,6,7.8 at epoch 5,10,20,50
Figure 15. Experiment Overlapping bases analysis
19

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Experimental Setup We construct datasets with equal proportions of each regression type. Input dimensionality remains consistent with previous setups (T) 16), and training parameters follow the synthetic experiment protocol outlined in Appendix D.2.
Results Results shown in Figure 16 indicate clear separability of latent representations corresponding to each regression family after training. Furthermore, we observe significant performance improvements when latent concepts are clearly encoded, confirming that models effectively infer and leverage concept-specific algorithms. This suggests that trans- formers naturally learn to adaptively switch between qualitatively distinct prediction algorithms based on their internal representations.
Loss curve
TD score over Training (layer 6)
1.0
0.8
Test MSE
Linear
CD Score
0.8
Quadratic
Sparse Linear
0.6
Leaky Relu
0.4
0.2
0.0
0.0
10
10
10
10²
Epoch
Epoch
(a) curves for different regression families.
(b) UMAP visualization of representations convergence.
Epoch 10
Epoch 100
Epoch 290
1.0
1.0
1.0
Quadratic
Sparse Linear
0.8
0.8
TD Scores
TD Scores
Scores
0.8
0.8
0.6
0.2
0.2
0.2
0.0
0.0
3
9
11
9
Layer
Layer
Layer
(c) TD scores across layers for each regression family.
Figure Mixture Different Regression Families Analysis. (a) Loss curves highlight performance improvements when latent concepts are clearly encoded. (h) UMAP visualization demonstrates distinct clusters corresponding to each regression family, indicating successful latent concept encoding. (c) TD scores across layers validate the emergence of clear representations for each regression type.
Verification of Head-Prruning Results analyzing attention-heat pruning effects in synthetic and mixture regression tasks. Attention-head pruning was conducted individually based magnitude-based importance scores, removing each head one at time.
We validate our claim that distinct latent bases form separate representations by
Figure 18 shows that attention-head pruning in synthetic sparse linear regression tasks. Changes in mean squared error (MSE) were measured across 100 random sequences per basis (same setup as main manuscript Figure 2). Attention Importance Estimation (AIE) quantifies performance degradation. Layer distinctly maps attention heads to specific bases (e.g.. 14h5 for base and 14h3 for base 2), supporting distinct algorithm implementations.
20

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Figure 17 shows that attention-head pruning on mixture regression families. MSE changes assessed across 100 random sequences for each regression type (linear, leaky ReI.U, sparse linear, quadratic). Attention heads notably shared across linear, leaky ReLU, and sparse linear regressions (excluding quadratic), indicating structural similarity. This aligns with the previously observed "common structure" (Kim et al.), confirming structural indistinguishabiliyy among these regression variants.
Overall, This confirms that distinct algorithms require dedicated representational capacities, whereas shared algorithms benefit from structural overlap.
bases:
bases
4.50
+2.01
0.23 0.22 0.50
0.03 0.00 5.00
-0.00
M054
11
9AL MUL
0.29 0/02
MAM 0100
9AM
0.50 1.82 91M) 10
9100 9/00
Heed Indeet
0.52
6:00
0.03
0.00
5.09
DAL 0.23 0.28 9.00 0.00 0.00
6.02
0.01
0.33
062
0.00
4.00 0.01 0.00 0.00 4.00 -300 4.00
0.0
0.0
0.02
0.0
0.02 6.02
001 0.31 6.91 0.00 0.08 0.00
8:12
6.00
0.51
06) 0.02 0.61 3.00 0.00 3.00 6.00
1:44
0:00
920
0.23
0.82 0.00 0.20
..000
-0000
0:00
0.01 3.00
9:00 002 6:01
601
0.33 0.00
681
0.01 0.61
4.00
Laywr
0.5
0.5
0.22 0.8c
0.00
5.00
5.00 -3.00 40.00 0.20 0:00 0.20 0.00
6.02
+:.00
18:00
0,32 0.00 0.00 3.00 0.01 0.00 0.00 0.00
355
0.18
8.0 5.01 0.00 0.03 0.83 6:83 0.58 0.00
6:3G
4.01 (-0
6001
0.94 0.00
0.60 0.00 0.00 600 4.00
0.03
5.02 0.03 0.92
0.28
5.37
0:43
6:02
0.00 0.00
0.0 0.00
Head Incex
0.50
0.50
3.0
6/3 5.00 0.82 0.83 0.00
2.01
2.00 4.11
6.58 8:00 0.00
6.00
0.90
6:92
090
0.13
0:00
031
3.0
3.00
0.33 6.82 6.2 4.5
4001
0.00 8.0 4.00 600
6/11
-05
-0.5
Layer
Layer
Figure Head-Pruning: Mixture Tasks. Minimal impact individual attention-head pruning indicates robustness from shared algorithmic structures.
21

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
Mixture: Lineer
Mixture: : Quedrelic
0-4030
4:00
3.00
3:36 5.00 0.38 0:82
1-
0/93 0.0 0.00
0.00
0.00
5.00 12.0 -0.00 0.33
2-
5.00 64t 4.00 0.00
0.00 4.00 0.00 -201 6.00
Head Index
0.20 nucc 0..0
300 $10 5.00 0.00 0.09 4:00 0.50
0.00
00 5.00 4.00 6:00 0:90 10.00 0.00 0.00 9.00 0.00 (300 6.00
0.0
AIE
0.0
4- 0.20 <0.00 4..00 4.00
4 00 1.00 D.00 4001 4.00 4040 0.00
-.00
4-4 0.00 -2.00 -3,00 +600 0.00 40.00 0.00 3.00 0.00 COL 000 6.00
0.00
use <..0 0.00 9:10 1.0 D103 10.0 0333 0.00 40.3 0.10
00 6.00 6,00 009 0.91 1L.0 9.00 0.00 9.0 0.00 690 0/93
6- 0.00 0.02 <0.00 2.00 5.00 5.11 0.23
4.92
4082
0.00 4.00
0.00
53.00 4:00 0.00 0.03 4.00 3.00 0.0 0.00
0
9 10
2
10
11
Mixture: Sparse Linear
Mixture
Leaky
0.5
0.5
0.90
0.
3.00
G.0L -301 4:00 01:00 0.01
0.00
0-
00
5.00 DiL
16:03 0L0C
0.02
0.00 0:51 0.00 C.0L, D101,
6:11
0.50
0.00
2.00
3.00
0.00 -0.00 0.20 40:22 4.20 0.00
3--
0:10
-2.00 0:00
0.2
1.24
40.00
0.00 3.00 0.00 2.00
-2.00
.0.00
4:900
241 1:40
2MM 9: 6.M0 5101 0.03 0620 0182 0..3 0.00
2- 8AM
0:14
0G9 0.96 11:00 9.00 91.00 S.L
0991
Head Iidee
4.00
0.52
4.00
2.00
5.00
5.02 0.23 0.22 0.02 0.20
0.00
0.00 4800 0.02
0.01 3.00 0.00 0.03 0.00
0.0
0.00
4.00
0.33 0.80
0.90
0.00 9:00
0.00
0.00 0.00 0.00 802
1.00
4.00
0.32
0.01
0.00
7
-2.00
5.00 0.1 0.00
5:00
0.00
0.01
6:00
-.5
-0.5
11
Layer
Layer
Figure 18. Heaad-runnng: Synthetic Tasks, Individual attention-head pruning significantly affects synthetic task performance, demon- strating distinct basis sensitivity.
G. Natural ICL Experiments
Part-of-speech Tagging. We construct a Part-of-speech (POS) tagging dataset from the English Penn Treebank corpus (Marcus et al., 1994) from the articles of Wall Street Journal. Our POS tags are, Noun, Adjective, Verb, Adverb, Preposition, Pronoun, and Pronoun. We abide by the data-use regulations and, from total of 4K samples, we filter out sentences that have all 6 POS tags. Then, split the dataset into 80-20 train-test split. We evaluate all the models on the test split, and the train split is only reserved for the finetuning experiments.
Bitwise Arithmetic. We construct a bitwise arithmetic dataset consisting of different operators: AND, NAND, OR, NOR, XOR, and XNOR. We randomly sample pairs of input binary digits and generate the resulting binary. For training, we construct 10K samples, and, for evaluation, we construct 500 samples.
Model.
We use a pretrained Llama-3.1-80 model for all of the main natural ICL experiments, if not specified otherwise.
Training. For most of the experiments, we do not train the model and only evaluate its ICL performance on the different tasks. However, we only finetune the model in the causal experiments to study the causal relation between the accuracy of concept encoding and ICL task performance. We finetune model per task family (i.e. POS and bitwise arithmetic). For computationally efficient finetuning given compute constraints, we use LoRA (Hu et al., 2021), type of parameter efficient finetuning. We set the rank and alpha to be 16 and the dropout to be 0.1. We train the model on a total of 10K samples with the next-token prediction loss. We only backpropagate the losses on the yJ predictions.
To evaluate the model's ICL performance, we use greedy decoding to generate answers given different number of in-context examples and compute an exact-match accuracy score whether the generated sequence is exactly equal to the ground truth.
Evaluation.
Compute. We use an A100 GPU with 80GB of VRAM for training and inference. Training takes ~ 4 hours and evaluation takes ~ 30 minutes for each run.
22

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
G.1. TD score by Layers from Section 4.1
We present the TD scores across the 32 layers of LLaMA 3.1 8B for POS tagging and bitwise arithmetic tasks. We found that TD scores peak at layer 15 for POS tagging and layer 13 for bitwise arithmetic tasks, and used these layers for measuring TD throughout Section 4. The observation that TD scores peak in the middle layers is consistent with the findings of (Hendel et al., 2023) and (Todd et al., 2023).
TD By Layers
0.8
PT POS Tagging
Layer 15
Bilwise Arihmelic
POS Tegging Peak
0.6
TD Score
Bitvise Arihhmeicc Peak
Layer 13
0.4
0.2
0
10
15 Layer Index
20
25
30
layers for POS tagging and Bitwise arithmetic with 4-shot in-context examples, showing peak decodability in intermediate layers. (b) For POS tagging and (c) for Bitwise arithmetic, TD scores increase with number of demonstrations, the improvement noticeably varies by task.
Figure TD Scores by layers and number of demonstrations. (a) Mean TD
G.2. Mechanistic Intervention Study from Section 4.1
We present the results for the mechanistic intervention study probing whether helping hindering concept encoding improve or degrade activation of corresponding decoding algorithms and whether they are causally related in Figure 20.
Noun
Pronoun
Verb
AND
OR
NOR
60
60
Test Test Accuracy (%)
40
Test Aecuracy (%%
20
20
Adverb
Adjective
Preposition
NAND
XOR
XNOR
60
60
40
40
20
20
Unperturbed
Negative
Positive
Unperturbed
Negative
Positive
(a) POS Tagging
(b) Bitwise Arithmetic
Figure Causal analysis concept encoding by intervention. We patch activations of input with correct incorrect latent concept to demonstrate that the inferred concept embedded in the representation can causally improve degrade performance. We intervene at layers 15 and respectively the POS arithmetic tasks. The results show the performance causally dependent the latent concept representations. Error bars represent the standard deviation across five different replicates of experiments.
G.3. Generalization with Different Model Families and Scales
In both the POS and bitwise arithmetic tasks, we observe a positive correlation between CD and ICL test accuracy across different model families and scales. Interestingly, in all the Gemma-2 family and Llama-3.1 70B models, Noun, Pronoun, and Verb show the clearest signs of concept secoddingdecodding behavior, as we saw in the 3lama-3.1 8B model Figure
23

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
ADD OR AON
1-thet 1b-aher
A-D OR XNDR
1-dhet 15-
AND GR XON
1-0hd 10uttil
50
AND OR XIDe
T-td 10-lt!
Test Accussty (%)
Test Accuray
Test Accurasy
22
Teat
x
15
13
RI 0.776
RC 0.462
RI 0,000
R 0.824
04
0.5
0.5
03
TD Score
TD Score
TD Score
TD Score
(a) Bitwise: Gemma 2B
(b) Bitwise: German 9B
(c) Bitwise: Gemma 27B
(d) Bitwiser Llama-33 3.1 70B
51
I-anol
Niuu Vurk Prepaation
Advebb Adjectne
1hatal
Werk
10 shat
24
10-ahool
Adjiciio
Adeciia
Accunecy
24
Test Accuresy
Test Acaursty
Test Accurasy
x
24
Rf 0.422
R7. 0.531
R" 0.122
- 0.86
0.00 0.5 C.0 6.44 Ese 0.2 036
888 0.0
C.70 650 0.5
0.50 0.04
0.72
8.0
548 3M
0.51
104 0.72
0.00
0.0
TD Soore
TD Score
TD Score
TD Score
(e) POS: Gemma-2 2B
(1) POS: Gemma-2 9B
gg) POS: Gemma-2 27B
(h) POS: Llama-3,1 70B
Figure 21. TD score vs ICL performance across 2emma-2 models (2B/9B/27B) and 1lama-3.1-705 The positive correlation between TD and ICL performance seen in Llama-3,1-88 1-8B generalizes across different models and scales. The grey dashed lines are linear lines of best fit. These results suggest that the accuracy of concept encoding is closely coupled with downstream ICL performance.
In the bitwise arithmetic task, AND, NAND, OR, and NOR (classes that showed the strongest -eroddingdecodingg behavior in 1 8B), also show the strongest signs of concept mncoinng-ecodinng behavior across all of these models. Given that many LLMs are trained on similar sources of pretraining data (Soldaini et al., 2024: Gao al., 2020) (CommonCrawl,, Wikipedia, etc), we conjecture that the models may have learned similar encoding-decoding mechanisms for these concepts.
G.4. Extension to Recurrent Neural Network Architectures
To examine whether our task aucoding-deooding framework generalizes beyond transformer- based models, we conduct experiments using a recurrent neural network (RNN) architecture. Specifically, we utilize two-layer LSTM model with 512 hidden units per layer. The training and evaluation setups replicate those used for transformer experiments in Figure 4. Figure 22 shows that the L.TM- architecture also demonstrates a positive correlation between task decodability (TD score) and in-context learning performance. Although absolute performance metrics differ from transformer-basee models, the fundamental relationship between representation clarity and task performance remains consistent. These findings suggest that the task mecoding-ecoding mechanism is not exclusive to transformers, but also is applicable to sequential neural models more broadly.
24

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
NAND
Pronoun Prepostion
4-anct
Test Test Accuracy (%)
Test Test Accuracy (%)
XNOR
20
10
R² 0.468
R 0.596
022
9.5
0.8
0.9
1.0
0.2
0.5
0.8
0.8
0.9
1.0
TD Score
TD Score
Figure 22. TD scores vs ICL perofrmance for Mamba-2 8B model. The positive relationship between TD scores and accuracy in the Mamba-2 8B model suggests that the capacity of TD scores a proxy for task RNN-based language models.
processes might generalize
G.S. Pairwise Concept Decodability Comparison
POS Tagging
Bitwise Arithematic
Noun
0.9
0.9
0.97 0.83
0.82
0.8
0.9
0.9
Adjective
0.93
0
0.99
0.9
0.82
0.82
0.84
OR
0.93
0.99
0.9
0.82
0.82
0.84
0.8
0.8
Verb 0.84
0.96
0
0.7
0.69
0.7
0.71
NOR
0.84
0.96
0.7
0.69
0.7
0.71
Adverb-
0.98
0.91
0.9
0.72
0.68
0.73
NAND
0.98
0.91
0.9
0.72
0.68
0.73
0.7
Preposition
0.82
0.84
0.63
0.6
0.88
0.6
Pronoun
XNOR
0.9
0.86
0.89
0.68
0.58
0.56
0.5
0.5
Null
0.85 Noun
0.81
0.84
0.6
0.48
0.45
Null
0.85
0.81
0.84
0.6
0.48
0.45
Adjective
Verb
Adverb
Preposition
Pronoun
Null
-044
AND
QR
NOR
NAND
XOR
XNOR
Null
0.4
(b)
Figure 23, Pairwise TD scores for POS Tagging and arithmetic tasks at 4 shot. Pairwise TD scores identifies the clustered tasks
H. Finetuning Experiments
We visualize the ICL test accuracy changes before and after finetuning the first and last 10 layers of the model on each of the tasks in Figure 24. These results confirm the hypothesis that, contrary to the common practice of finetuning the last layers for classification tasks for instance, finetuning the earlier layers directly improves the task encoding and thus the ICL task performance more than finetuning the latter layers.
25

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
(a) POS Tagging
(b) Bitwise Arithmetic
Noun
Pronoun
Verb
AND
NAND
OR
100
100
Test Test Accuracy (%)
Test Accuracyy (%)
50
50
0
Adverb
Adjective
Preposition
NOR
XOR
XNOR
100
100
50
0
PT FT (last 10 layers)
FT (first 10 layers
Figure 24. ICL test accuracy at shots across tasks in POS and arithmetic after finetuning (FT) the first and last 10 layers. When restricting the model's ability to encode latent concepts in its intermediate representation (finetuning last 10 layers), the model fails fully align its representations for learning the latent concepts and falls behind the performance of finetuning the first 10 layers.
Prompting Experiments
Experimental Setup. To study whether concept encoding is unifying principle that underlies different mechanisms to improve ICL, we also experiment with prompting. Instead of hiding the concepts and letting the model infer, we include information about the true concept for the examples (e.g., including the true label of AND operator or the instruction of "Find the first noun in the sentence").
Results. As discussed in Section 5, we question how prompting may be affecting the concept encoding in increasing task performance. As expected, prompting improves the performance the model, especially in the bitwise arithmetic experiments. Simultaneously, we observe that the decodability score of the latent concepts also increases drastically, However, we interpret these results with caution because the model may be capturing spurious correlations from the differences in the input distribution. Specifically, the bitwise arithmetic experiments show high decodability even in the beginning layers of the model.
Part-of-Speech Tagging
Bitwise Arithmetic
TD Score
0.5
Prompting
0.0
10
20
30
0
10
20
30
Layers
Layers
Figure 25. TD score across layers for POS tagging and bitwise arithemetic Llama-3.1 3.B-BBBBBBBBBBBB for the prompting experiments. We include the true labels of the latent concept (i.c. "Find the first noun in the sentence."), We detail the experimental setup in Appendix I.
26

---

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
AND
NAND
OR
Noun
Pronoun
Verb
100
100
Test Test Accuracy (%)
50
Test Accuracy (%)
50
0
Adverb
Adjective
Preposition
NOR
XOR
XNOR
100
100
50
50
0
Pretrained
Prompted
Pretrained
Prompted
POS Tagging
(b) Bitwise Arithmetic
Figure 26. ICL test accuracy across 12 tasks in POS tagging and bitwise arithmetic with prompts containing the true concept AND, AND.
"Find the first noun the sentence") of task.
27


Reviews:
[{"review_id": "K7fmGkOi0n", "reviewer": "Reviewer_RjY9", "summary": "This paper studies how transformers learn task vectors for various tasks during pretraining. Task vectors refer to the intermediate representations from a middle layer of a transformer network, given an in-context learning (ICL) task. The paper investigates how these task encodings separate and cluster based on the task, and proposes that this separation can serve as a metric for encoding quality.\n\nTo measure this, the authors introduce Task Decodability (TD) scores, which quantify how separable and distinct the task encodings are. They show that TD scores are tightly correlated with ICL accuracy, and validate this relationship across multiple language models (e.g., Gemma, LLaMA) and tasks, including a synthetic mixture of sparse linear regression, part-of-speech tagging, and bitwise arithmetic. They also examine how task vector emergence progresses throughout pretraining using checkpointed models from OLMo-7B.\n\nFraming ICL through a Bayesian lens, where the model learns a latent task parameter $z$, the authors argue that transformers first encode the input into a task vector in earlier layers, and then decode it to solve the task in later layers. To go beyond correlation, they apply activation patching, showing that injecting high-quality encodings improves final task accuracy, while poor encodings degrade it—further supporting the encoder-decoder interpretation of task processing in transformers. This is also bolstered by the fact that finetuning earlier layers helped task accuracies more, though this is somewhat contrary to conventional practice.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and thoughtful review. We appreciate your recognition of Task Decodability as “a compelling new measure” and that our activation patching analysis “provides convincing evidence [of transformers’] encoder-decoder structure.” We also value your suggestion to generalize to Mamba, and address your comments and questions below.\n\n> What is the criterion for layer selection for task vector analysis?\n> \n\n**Common Point #1 Raised by Reviewer bnoD,  RjY9, UVog**\n\nWe select the layer with the highest TD score as the best encoding of a task. For the synthetic task (Section 3.3), we computed TD scores across all layers (Figure 10) and chose the fifth layer (layer index 4 in zero-based Figure 11). UMAP visualizations in Figure 11 confirm that representations become separable precisely when the TD score peaks. \n\nFor clarity, figures illustrating TD scores and UMAP visualizations are: \n\n- Synthetic tasks : Figure 10, Figure 11\n- Natural language tasks : Figure 16\n\n> Does the work generalize to various other architectures like Mamba?\n> \n\nWe **explored Mamba-2 8B and found our insights to generalize**. We conducted the same studies on natural tasks on the pretrained Mamba-2 8B model (results in https://sites.google.com/view/icl-encoder-decoder/home). The results show that the TD score correlates with ICL accuracy (Figure A), and separable representations emerges around layer 33 (Figure B). These results suggest that our encoder–decoder framework and the predictive power of TD score on ICL performance do indeed generalize beyond standard transformer-based models to recurrent models. \n\n> Selected natural tasks are simple, so can you try on MMLU per subject?\n> \n\nWe refer the reviewer to **Common Point #2** in rebuttal for reviewer EJ7T. In summary, in benchmark like MMLU the boundaries between tasks are inherently blurred, making it challenging to rigorously study the encoder-decoder framework compared to our controlled experiments.\n\n> What differentiates this work from previous studies on mechanistic interpretability of ICL, such as Core et al. (2024)?\n> \n\nWe concisely compare our contributions from prior work:\n\n1. **Different Task Setup**\n    - Core et al. (2024): Adopts a relatively homogeneous Markov chain task and examines how the system transitions between different algorithmic stages (unigram/bigram retrieval and inference).\n    - **Our Work**: Focuses on the *heterogeneous tasks*—a setting more reflective of how LLMs are trained and used in practice (where tasks vary substantially). We analyze emergence of distinct task vectors during pretraining.\n2. **Measured quality of task-specific vectors and its relation to downstream ICL peformance**\n    - Hendel et al. & Todd et al.: Demonstrate that large language models have *task vectors* but leave open questions about how these vectors emerge and why their effectiveness varies across tasks.\n    - **Our Work**: Addresses these open questions by analyzing *when and how* “task-specific” representations emerge, and how their “quality”(TD score) can be measured and used to predict downstream ICL performance.\n\n> Try Kim et al. experimental setup with different regression tasks\n> \n\nAs per the reviewer’s suggestion, we replicated the regression task mixture experiment from Kim et al., training a small transformer on linear, quadratic, sparse linear, and leaky ReLU regression tasks (results available at https://sites.google.com/view/icl-encoder-decoder/home). The model successfully learns the different tasks but fails quadratic regression (Figures A, C). TD scores and UMAP visualizations (Figures B, D, E) indicate no clear separation of representations, suggesting the absence of distinct task vectors. We interpret these findings as evidence that the transformer employs similar underlying algorithms across these tasks (e.g., implementing leaky ReLU via linear regression with sign adjustments). This is consistent with Kim et al.’s observation that a common structure between their regression tasks facilitates learning (Figure 6 and Section 4.2 in their paper).\n\n> Can you clarify the notation and definition in supplementary material for Section B?\n> \n\nWe will improve the decodability definition like the following in Appendix B.\n\nB.3 (Decodability). For a given decoder $G: \\mathbb{R}^{d_{emb}} \\rightarrow \\mathcal{Z}$ and a specific latent variable $z$, the decodability measures how accurately the correct latent variable is inferred from representations. Representation is distributed as $E(z,D)$, and the inferred latent variable is  $\\hat{z} \\equiv G(E(z,D))$ . \n\n1. One-hot Accuracy : $A_{\\text{1-hot}}(z) = E[\\mathbf{1}\\bigl[\\hat{z} = z\\bigr]]$\n2. f-divergence : $A_{f}(z) = D_{f}(\\hat{z} \\parallel z)$, where $f$ is some f-divergence metric\n\nOur Task Decodability is the one-hot accuracy with decoder of k-nearest neighbor majority-voting algorithm.\n\nFor  a definition of Task Decodability, we refer reviewer to **Common Point #3** in rebuttal to Reviewer EJ7T.", "claims_and_evidence": "The claims presented in the paper are generally well-supported. However, there are some comments that, if addressed, could further strengthen the claims. Notably, the paper selects intermediate representations from certain layers (typically mid-range within the overall transformer architecture) but it does not articulate a clear, prescriptive criterion for choosing these layers. Establishing such a criterion for layer selection in relation to task encoding would enhance the falsifiability and robustness of the paper’s hypotheses. As it stands, the implication is that any layer compatible with the encoder-decoder framework may suffice, which weakens the specificity and testability of the claims. Or should I assume the most representative TD score is the maximum TD score over all layers, as shown in Figure 16?", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are generally well-aligned with the problem at hand. The paper begins with synthetic in-context learning (ICL) experiments, building on standard regression-based ICL setups from prior work. By partitioning the base tasks and constructing a mixture of sparse linear regression problems, the authors effectively explore how task representations are encoded and separated. The naturalistic experiments are also reasonable and conceptually consistent with the research goals, though they tend to be relatively simple (bordering on toy examples), especially when contrasted with more complex, real-world benchmarks like MMLU. I would definitely like to see the same experiments interrogated on tasks like MMLU per subject.", "theoretical_claims": "N/A. The only issue I have here is the notation and definitions presented in the supplementary material for Section B. First, TD score is not formally defined in the main body as far as I can tell; plus, the connection to Section B is not clear as it does not define \"score\" there either. $D_f$ is ill-defined and probably represents some distribution metric, though $f$ is also unclear.", "experimental_designs_or_analyses": "The experiments seem sound.", "relation_to_broader_scientific_literature": "This work introduces Task Decodability (TD) as a quantitative measure of how well task-specific information is captured in intermediate representations, linking it tightly to ICL performance. This complements mechanistic interpretabiliy studies on ICL by providing concrete evidence that task-relevant encodings emerge early in the network and are critical for downstream accuracy. The paper’s encoder-decoder framing of task processing echoes Bayesian interpretations of ICL and deepens our understanding of how transformers internally represent and differentiate between tasks during pretraining.", "essential_references_not_discussed": "Not that I know of.", "other_strengths_and_weaknesses": "**Strength:**\n- Novel Quantitative Metric (Task Decodability): The paper introduces Task Decodability (TD) as a compelling new measure of how well intermediate representations capture task-specific structure. This metric is shown to correlate strongly with ICL performance, adding a valuable tool to interpret how transformers process in-context learning tasks.\n- Encoder-Decoder Interpretation of ICL: Through activation patching and partial fine-tuning, the authors provide convincing evidence that transformers operate in an “encoder-decoder” manner—where early layers encode the task into a latent vector, and later layers decode it to generate predictions. This mechanistic perspective adds clarity to how ICL unfolds.\n- Consistency Across Models and Tasks: The authors validate their findings on multiple transformer architectures (e.g., Gemma, LLaMA) and both synthetic and naturalistic tasks, which suggests a degree of generality and strengthens confidence in the interpretability of TD scores.\n\n**Weakness**\n- Investigation only on Transformer: While Transformers are the de facto model, synthetic ICL experiments have been conducted for various other architectures like LSTM and Mamba. The encoder-decoder framework needs further investigation on other architectures. Explained more in \"Questions For Authors\".\n- Relatively Simple Task Suite: The naturalistic and synthetic tasks used (e.g., sparse linear regression, part-of-speech tagging) are conceptually valid but still fairly simple. Testing TD on more complex tasks like MMLU or richer multi-task settings could better showcase the scope and robustness of the proposed method.\n- Unclear Layer Selection Criterion: While the paper observes that middle layers often provide more “task-relevant” representations, it does not prescribe a systematic way to choose these layers. A more principled method for identifying the “best” layer (or set of layers) would strengthen the paper’s claims.\n- Novelty: I am not entirely familiar with the mechanistic interpretability of ICL works, but what differentiates this paper from previous works that also examine loss plateaus and representations? e.g., https://arxiv.org/pdf/2412.01003 off the top of my head. A more detailed related work comparing and contrasting their work to prior work would help the reader situate this work.", "other_comments_or_suggestions": "None. See below for questions.\n\nEDIT: Score adjusted after rebuttal (as of April 8th).", "questions_for_authors": "I have listed some concerns and questions previously. I will only add some questions that I have not asked yet. In order of decreasing importance.\n\n1. **Other Architectures.**\nIt doesn't seem that this paper performs any unique analysis for transformers and this means that their method can be applied to other architectures as well. We can extract intermediate representations of simple architectures like LSTMs or even more modern ones like Mamba in similar fashion. Does this paper's hypotheses apply to other architectures and can the experiments yield any insight into the fundamental ICL capabilities of other models?\nIn the case that TD scores do or do not correlate well with ICL accuracies for these architectures, I think there's something interesting to be said or investigated here.\n\n2. **Synthetic ICL experiments.**\nI find the construction to the synthetic ICL experiment mathematically appealing. But empirically, I would also like to see training a network on a mixture of different tasks, rather than just a mixture of sparse linear regressions. For example, Kim et al. (https://arxiv.org/pdf/2410.05448) train multiple tasks together (linear+quadratic+ReLU regression) and I would like to see how the encoding separations evolve here. This would strengthen the results to more diverse tasks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t4jOOzSOrP", "reviewer": "Reviewer_EJ7t", "summary": "This paper investigates the learning mechanisms of autoregressive Transformers in In-Context Learning (ICL), with a particular focus on the emergence of task vectors and their impact on ICL task performance. Building on the Bayesian view of ICL, the authors propose an encoder-decoder analytical framework and empirically demonstrate its validity by examining the evolution of representations during training. The study further reveals that Transformer models develop task representations throughout pretraining, and the quality of task encoding strongly correlates with and effectively predicts ICL task performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and thoughtful review. We appreciate that you see our “theoretical logic as comprehensive, extending from experimental observations in regression tasks to natural language experiments,” and that you find our encoder-decoder framework provides “new theoretical foundations and empirical support for understanding the performance mechanisms and optimization methods of large language models in ICL.” \n\nBelow, we address your comments and questions.\n\n> The task complexity adopted in the paper is relatively low.\n> \n**Common Point #2 Raised by Reviewer RjY9, EJ7t**\n\nWe agree that our selected tasks are relatively simple compared to MMLU for example, but we intentionally focus on standard tasks (e.g., Oswald et al., Garg et al., Chen et al.) to rigorously study the emergence and effectiveness of task vectors using clearly distinguishable concepts (e.g., AND/OR in Bitwise Arithmetic, Pronoun/Verb in POS tagging). \n\nFor more complex settings in our synthetic setup, we experimented with higher numbers of bases and overlapping/shared bases between tasks (see Appendix D.1 and D.2). Interestingly, we observed that models tend to develop shared algorithms for overlapping concepts while distinctly separating non-overlapping ones—driving significant improvements in ICL performance.\n\n[1] Von Oswald, Johannes, et al. \"Transformers learn in-context by gradient descent.\" ICML. PMLR, 2023.\n\n[2] Garg, Shivam, et al. \"What can transformers learn in-context? a case study of simple function classes.\" NeurIPS 35 (2022): 30583-30598.\n\n[3] Chen, Xingwu, Lei Zhao, and Difan Zou. \"How transformers utilize multi-head attention in in-context learning? a case study on sparse linear regression.\" arXiv preprint arXiv:2408.04532 (2024).\n\n> What is the criterion for layer selection for task vector analysis and how is it related to Todd et al., 2023?\n> \n\nWe refer the reviewer to our **rebuttal to Reviewer RjY9 (under Common Points #1)** for how we select the layer based on the highest TD score. \n\nThis approach is aligned with Todd et al. 2024. Todd et al. observe that task vectors at different layers result in varying ICL performances (Figure 2c in their paper).  Building on this, we propose TD score to quantify the “quality” of task vectors. We demonstrate in Section 4 that **(1) TD score is *predictive* of downstream ICL performance directly and (2) both finetuning and prompting for ICL can be interpreted under this framework of an encoder-decoder**.\n\n> What data and procedure is used for UMAP in Figure 2?\n> \n\nTo generate the UMAP in Figure 2, we randomly draw 100 samples for each basis. Next, we collected $y$-token representations of 5th layer at 20th demonstrations and plotted the UMAP with the parameters of `n_neighbors=15` and `min_dist=0.1`. This specific layer was selected based on the maximum TD score (See Common Point #1 under Reviewer RjY9 and Figures 10 and 11 in the manuscript).\n\n> Minor improvements in supplementary presentation\n> \n\nWe appreciate the feedback and will refine the clarity of our supplementary results for the camera-ready version.\n\n> In Figure 7 (a), why do the solid and dashed lines coincide when the number of samples in the first column increases, but not in the first row?\n> \n\nWe hypothesize that the phenomenon occurs because the decoding algorithm for B2,B3,B4 may have developed after the 5th layer at epoch 20.  This allows perturbations to B1 introduced at the 5th layer to be corrected as there are additional encoding layers before decoding occurs, when sufficient demonstrations are provided.\n\nIn contrast, the decoding algorithm for B1 appears to be implemented directly at the 5th layer, as evidenced by the separated representations visible in the UMAP visualization in Figure 10. Consequently, the model cannot be recover from perturbations from B1 to B2, B3, or B4 at the 5th layer.\n\n> In Figure 7, why is the effect eliminated as the number of samples increases?\n> \n\nThis is an example of when **LLMs can recover the latent task against the adversarial task vector perturbation (patching wrong task vector)**. Previous works by Usman et al. [1]. and Chen et al. [2] show that higher number of in context examples can give stronger ICL signal and override the effect of adversarial demonstration attacks. In line with these findings, we hypothesize that a similar mechanism operates in our experiment (Figure 7), mitigating the impact of adversarial task vector perturbations. When there is uncertainty about the underlying task basis (# of demonstrations << task dimension), both self-perturbation and adversarial perturbation are more effective.\n\n[1] Anwar, Usman, et al. \"Adversarial robustness of in-context learning in transformers for linear regression.\" *arXiv preprint arXiv:2411.05189* (2024).\n\n[2] Cheng, Chen, et al. \"Exploring the robustness of in-context learning with noisy labels.\" *ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2025.", "claims_and_evidence": "The theoretical logic is comprehensive, extending from experimental observations in regression tasks to natural language experiments. However, the data sources for some figures are not clearly stated, leaving us puzzled about which data were used for UMAP in Figure 2.  \nThe article, based on a Bayesian view, uses UMAP in regression experiments to demonstrate the geometric structure of the representation space and measures loss using MSE, explaining the emergence of separable representations and coupled algorithmic phase transitions. However, it does not clarify why the fifth layer activation was chosen for substitution or the effects of substituting activations from other layers. Figure 7 effectively illustrates the encoder-decoder framework.  \n\nThe article proposes the TD score as a basis for selecting important layers, inspired by the background of task vector proposals. However, it is unclear how this differs from previous methods, such as those in (Todd et al., 2023), and whether the principles are consistent, which is a point of confusion for us.", "methods_and_evaluation_criteria": "The methods are reasonable, drawing conclusions from phenomena in regression tasks to inform understanding of natural language tasks. The loss settings during training and the calculation of the TD score are scientifically sound. The article has guiding significance for subsequent applications, such as using the encoder-decoder framework to characterize task vectors, which can guide the completion of ICL tasks.", "theoretical_claims": "The paper proposes that the formation of task vectors can be explained through an encoder-decoder framework, and that Task Decodability is a predictor of ICL performance. The theoretical logic is complete, and the description is clear.", "experimental_designs_or_analyses": "NULL (I did not reproduce the experiments, but the experimental logic is clear and complete.)", "relation_to_broader_scientific_literature": "The calculation of important layers using the TD score aligns conceptually with the identification of critical activations in (Todd et al., 2023). However, the theoretical computation process differs, and the specific connections between the two approaches warrant further exploration.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths**  \n\n1. The theoretical logic is comprehensive, extending from experimental observations in regression tasks to natural language experiments.  \n2. This paper systematically elucidates the formation mechanism of task vectors through an encoder-decoder analysis framework and verifies its predictive ability for ICL task performance. This research provides new theoretical foundations and empirical support for understanding the performance mechanisms and optimization methods of large language models in ICL.  \n\n**Weaknesses**  \n\n1. The specific data sources for some conclusion charts are not clearly stated.  \n2. The task complexity adopted in the paper is relatively low.", "other_comments_or_suggestions": "It is recommended to supplement the data sources used in the UMAP method and attempt to discuss whether there is consistency with the specific task vector extraction methods in related articles.", "questions_for_authors": "- Regarding Figure 2, what specific points were used when applying UMAP? Are the points used for the projections of the four different regression tasks consistent? Specifically, what data were utilized? Was the projection matrix consistent during the projection process?  \n- Additionally, in line 197 of the article, it is mentioned that the fifth layer activation was chosen. How was the fifth layer selected?  \n- In Figure 7 (a), why do the solid and dashed lines coincide when the number of samples in the first column increases, but not in the first row?  \n- In Figure 7, why is the effect eliminated as the number of samples increases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vMMCvHof7G", "reviewer": "Reviewer_bnoD", "summary": "This paper studies the task vector behavior in the trained-from-scratch and pretrained transformer, demonstrating the following conclusions:\n1. When the transformer develops the task encoding (that generates the task vector), it simultaneously generates the corresponding decoding functions.\n2. In pretrained LLMs, the well-separation of task vectors correlates with the ICL performance.\n3. Finetuning earlier layer helps for ICL performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful review. You highlight a key point that “the different encoding progress for different tasks is really interesting,” which resonates strongly with our primary objective of illustrating how task encodings develop and shape downstream ICL performance.\n\nBelow, we address your comments and questions.\n\n> Why does the model learn to encode B1 first in the synthetic experiment?\n> \n\nAlthough the four bases are theoretically equally difficult (same rank), random factors like data order and seed can cause the model to decode one basis first. As shown in Fig. 8, different seeds lead to different bases being prioritized.\n\n \n\n> How does our observation explain or contradict Yang et al. 2025?\n> \n\nWe agree that the formation of distinct task vectors depends heavily on task formalism (how distinct task are defined). When the tasks share a common structure (e.g., the variant of regression tasks in Yang et al. [1] and Von Oswald et al. [2]), the model has little incentive to learn distinct representations, and simple attention-based algorithm[2] may suffice. This aligns with the Kim et al. [3], who report the underlying common structure between the aforementioned regression tasks.\n\nOur synthetic sparse regression experiments are designed with algorithms that differ across tasks(basis), effectively enforcing a algorithmic diversifications and formation of distinct task vectors(albeit with varying quality). From these experiments, we report that the encoding and decoding algorithms emerge *simultaneously*, and that the quality of the learned encoding (measured by TD score) correlates with ICL performance.\n\nWe acknowledge that real-world tasks lie on a spectrum: some share overlapping structures, while others require truly orthogonal algorithms. This naturally places them between the two extremes of having fully separate versus completely shared task representations. Nonetheless, our key takeaway is that **when tasks are sufficiently distinct, the quality of the learned task encodings can serve as useful indicator of ICL performance**.\n\n[1] Yang, Liu, et al. (2025).\n\n[2] Von Oswald, Johannes, et al. (2023).\n\n[3] Kim, Jaeyeon, et al. (2024).\n\n> What is the criterion for layer selection for Fig. 2?\n> \n\nThank you for highlighting this point—it aligns with a common reviewer concern (Refer to **Common Point #1** in RjY9’s rebuttal). As shown in Fig. 11, concept separation is most distinct at layer 5, which is why we select it in Fig. 2.\n\n> In Table 1, prompting yields a high TD score but poor ICL performance on the Bitwise Arithmetic task. How can this observation be better explained?\n> \n\nWhile prompting (i.e., modifying only the input text) leads to strong TD scores, it does not always result in correspondingly high accuracy—particularly compared to finetuning cases. As discussed in Appendix G, we interpret this results with caution: high decodability observed in the early layers during prompting may not reflect meaningful task encoding-decoding. Instead, it could stem from **spurious correlations introduced by differences in the input token distribution**. The bitwise arithmetic experiments seem sensitive to these correlations, which can inflate decodability as shown in Figure 21 on the right. Nevertheless, prompting still consistently produces higher TD scores and accuracy than the pretrained model.\n\n> Why does learning the task encoding by finetuning only the early layers improve performance? Why is learning encoding more important than decoding?\n> \n\nThat’s an insightful question. Our hypothesis is that, since the LLM is pretrained on vast amounts of data, it often already has the “decoding” circuits (e.g., it can do XOR/AND) and mainly needs the right trigger(encoding signal) to activate them. Thus, directly changing the earlier layers (task encoding) can often be critical. However, for truly novel tasks that the model hasn’t seen (e.g., coding in an esoteric programming language [1]), it may need to learn both the task representation and the decoding algorithm from scratch.\n\n[1] https://en.wikipedia.org/wiki/Esoteric_programming_language\n\n> Two messages on emergence of task vectors and relationship between TD and performance.\n> \n\nOur paper indeed focuses on two key findings: (1) how task vectors emerge during pretraining and (2) the quantification of task vector quality. While these findings may initially appear distinct, they form a cohesive narrative as follows. \n\n**We first study the direct phenomenology of task vector emergence** with the synthetic experiments and establish the foundation of our task encoding-decoding framework. Then, for the second point, we demonstrate **how this framework is practically useful** by using task decodability to predict downstream ICL performance in natural setting—bridging mechanistic insights with applied relevance.\n\nTherefore, we believe the two claims offer a comprehensive encoding-decoding framework for understanding and evaluating ICL capabilities.", "claims_and_evidence": "Please see \"Experimental Designs Or Analyses\"", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "In the initial synthetic experiment, the author provides the evidence that one of the basis classes is decoded first compared to the other basis functions. I wonder if the 4 basis functions are equally difficult, why the model chooses to decode the basis function B1 first?\n\nIn observation 2, why choose the 5th layer, instead of any other arbitrary layer that is early or intermediate for the transformer? How do you validate that the representation at the 5th layer is strong enough for the later \"casual relation between task encoding and performance\" experiment?\n\nI think the different encoding progress for different tasks is really interesting; however, it is not clearly shown in any other synthetic tasks except the sparse linear regression. In the pretrained LLM experiment on OLMo-7B, only the correlation between \"clear separation between task vectors\" measured by TD and the ICL performance is compared, leaving me wondering whether the separations occurred also in the pretraining. As a first step, could you also conduct experiments like Figure 7 for Figure 6 to understand the learned decoding for different tasks?\n\nIn section 4.2, the author hypothesizes that the quality of task vector is predictive of ICL performance. However, in the recent preprint [Yang et al. 2025], it seems that the Transformer can solve the synthetic in-context learning task regardless of whether a clear task vector (and thus the accompanying encoding-decoding algorithm) is formed. Would this hypothesis only apply to the pre-trained LLM that is not yet finetuned on the downstream tasks? For instance, in table 1, even the prompting could result in good TD score, however, with worse ICL performance as in the Bitwise Arithmetic task. How to properly support this hypothesis then?\n\nI didn't quite understand the logic behind \"finetuning the earlier layer is more effective given the earlier layer is for task encoding\". Since the decoding algorithm needs to be developed as well given the task vector, and when the task encoding is not developed, the task decoding is not developed as well, why only learning the task vector by finetuning the earlier layer helps? Why learning task encoding is more important than learning the task decoding? Could you elaborate this further?\n\nFollowing the question above, I noticed for the bitwise arithmetic task, though layer 13 has the peak performance, it does not differ a lot with the remaining layers (after layer 13) as shown in Figure 16. How do you decide layer 13 is the layer that task vector resides?\n\n\n[Yang et al. 2025] Yang, Liu, et al. \"Task Vectors in In-Context Learning: Emergence, Formation, and Benefit.\" arXiv preprint arXiv:2501.09240 (2025).", "relation_to_broader_scientific_literature": "The key contribution of the paper extends the previous literature on task vectors by complementing the existence of task vectors with the hypothesis that the Transformer learns the task vector along with the encoding-decoding algorithm, and that the formation of task vectors helps in-context learning performance.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No.", "questions_for_authors": "I appreciate the author trying to develop and explain the behavior of task vector. However, the message of this paper seems to be a bit mixed. The first synthetic experiment demonstrates that the encoding - decoding algorithm emerges at the same time the task vector emerges for certain tasks. However, the later part of the paper focuses on the task vector quality correlating with the ICL performance. I think both are interesting findings, but the flow is a bit confusing to me.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CcPIfFtZjK", "reviewer": "Reviewer_Uvog", "summary": "This work studies the mechanisms underlying the emergence of in-context learning. \n\nThe authors show that, on a synthetic dataset composed of a mixture of sparse linear regression tasks, (a) the first layers of a transformer progressively learn to map each task into a separable latent space and (b) the following layers progressively learn task-specific ICL algorithms.\n\nThey extend this task encoding-decoding paradigm to real LLMs, showing that some (arithmetic and language) tasks are better encoded than others, which drives their ICL performance. They also propose a task decodability metric, and show that it predicts ICL performance. Finally, they show that finetuning earlier layers improves task encoding hence ICL performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thoughtful review. We greatly appreciate that you find “the authors make great work at demonstrating the emergence of the task encoding and task decoding mechanisms” and that you consider “the synthetic experiments … and the real LLM experiments to be well-designed and clearly executed.” We also appreciate your suggestion to add detailed clarifications and cite relevant works such as Olsson et al., which provided valuable feedback for improving our manuscript.\n\nWe now address the specific questions that the reviewer raised their review.\n\n\n> Provide precise definitions of the Task Decodability and patching procedure\n> \n\n**Common Point #3 Raised by Reviewer RjY9, UVog**\n\nWe define the patching procedure and task decodability more precisely below and will update the manuscript for the camera ready accordingly.\n\n1. **Task Decodabilty**\n\nLet $\\mathcal{T}$ be the set of latent tasks. For each task $z \\in \\mathcal{T}$, sample N=100 datapoints $\\{(x_i, y_i)\\} _ {i=1}^N$ and collect the intermediate representations $\\{(h_i)\\} _ {i=1}^N$ from a chosen layer (e.g., the token embedding immediately after $x_i$). Label each representation $h_i$ with the corresponding task $z$. This yields a set \n$\\mathcal{S} = \\bigcup_{z \\in \\mathcal{T}} \\{ (h_i, z) \\mid i = 1, \\dots, N \\}$\n, over all tasks $z$.\n\nGiven a query point $(h_i, z)$, we exclude $(h_i, z)$ from $\\mathcal{S}$ (i.e., $\\mathcal{S} \\setminus(h_i,z)$) and find its $k$ nearest neighbors (with $k=10$) in the remaining set. We then use majority voting on those neighbors’ task labels to produce a predicted label $\\hat{z}$. If $\\hat{z} = z$, classification for the task label is correct. The TD score at this layer is the fraction of query points classified correctly:\n\n$\\text{TD} \\coloneqq \\frac{1}{|\\mathcal{D}|}\\sum_{(h_i, z)\\in \\mathcal{D}} \\mathbf{1}\\bigl[\\hat{z} = z\\bigr].$\n\n2. **Patching Procedure**\n\nWe collect representations $\\{h_i\\} _ {i=1}^n$  at selected position and layer for set of in-context examples $\\{(x_i,y_i)\\} _ {i=1}^n$  by passing the prompt through the model and recording the activations at a specified layer after $x_i$. We then average these activations to obtain a single “task” representation $h_{\\text{task}} = \\frac{1}{n}\\sum_i h_i$. For a new query $x_{\\text{query}}$, we add $h_{\\text{task}}$ into the same layer’s activations—effectively replacing the representation after $x_{\\text{query}}$—and let the model continue forward.\n\n> Why does the model show maximal representation separation at layer 5 in the figure 2?\n> \n\nThank you for noting that detail, as it has been a common point of interest for the reviewers (Refer to **Common Point 1**). Indeed, Figure 11 in the appendix shows that the separation of representations by concept emerges most distinctly at layer 5, which is why we focus on that layer in the main Figure 2.\n\n> Suggestions for citing several papers(Olsson, Catherine, et al. (2022) and Sivaramakrishnan, et al.(2023))\n> \n\nThank you for these suggestions, as they seem very relevant. We will add these to our discussion of related works in the final camera ready.\n\n> Suggestions for adding a limitation about the discrepancy between the simple setup and LLMs for tasks with shared support.\n> \n\nWe agree this is a notable limitation of our simpler mixture-of-sparse-linear-regression framework. Perhaps, the large-scale pretraining in LLMs allows them to handle overlapping task supports more effectively, whereas our synthetic setup sometimes struggles to separate representations for tasks with shared support. We will add this in our limitations section for our camera ready version.\n\n> How many in-context example do you give to the TestMSE loss in Figure 2?\n> \n\nThe TestMSE in Figure 2[left] is mean squared error over sequence of 20 in-context examples. Specifically, TestMSE $\\epsilon$ is given as $\\epsilon = E[\\frac{1}{20} \\sum_{i=1}^{20} \\left( y_i - \\hat{y}_i \\right)^2]$. We will clarify this point in the followup. \n\n> Why does self-pertubation improve performance at convergence (Line 206, right)?\n> \n\nFigure 7 that the reviewer is referring to shows test MSE (y-axis) vs. number of demonstrations (x-axis). Because the task dimension is 4, having fewer than 4 demonstrations leaves uncertainty about the underlying basis. In this case, self-perturbation (patching the task vector) provides extra information about the basis and thus improves performance in that regime. That’s precisely why once the number of demonstrations matches the task dimension (around 4), the model identifies the task basis, and patching offers little to no additional benefit.\n\n> In the definition of the TD score, how do you define the set of representations over which you select the neighbors? Also what is N=100?\n> \n\nWe define the set of representations by first sampling 100 datapoints (N=100) from *each task* and collecting their representations. We then compute the TD score by performing kNN classification on these representations.", "claims_and_evidence": "The authors make great work at demonstrating the emergence of the task encoding and task decoding mechanisms during training. The synthetic experiments are well-designed and clearly executed\n  - Figure 2 clearly shows that task-specific latent separation emerges through training  \n  - The patching experiment shows that the decoding procedure is task-conditioned. \n\nThey also do a good job at showing, on real LLMs, that \n  - some tasks are better encoded than others (using UMAP visualization and task decodability)\n  - the best-encoded tasks have higher ICL performance\n  - since task encoding happens in the first layers, finetuning them boosts ICL performance", "methods_and_evaluation_criteria": "The methods and evaluation criteria all make sense. \n\nSome comments:\n(1) I would encourage the author to detail their patching procedure (line 216).\n(2) I understand the idea behind task decodability, but the definition is not precise. See my questions below.", "theoretical_claims": "The paper does not contain theoretical claims.", "experimental_designs_or_analyses": "I really enjoyed the synthetic and the real experiments.\n\nAs I was wondering why the main Figure 2 shows the representation of layer 5, I would encourage the author to mention their finding from Figure 11 in the main text: the separation of representations by concept is only clearly observed from layer 5.", "relation_to_broader_scientific_literature": "The emergence of task encoding (in the earlier layers) and task decoding (in the late layers) in LLMs, and the fact that task encoding drives ICL performance, is an important research problem. The fact that the authors show their findings on synthetic and real tasks makes their results very valuable to the community.", "essential_references_not_discussed": "The authors should cite this paper, which shows the existence of “induction heads” in transformers:\nOlsson, Catherine, et al. \"In-context learning and induction heads.\" arXiv preprint arXiv:2209.11895 (2022).\n\nThe paper below showed a similar finding to the paper: in a Bayesian sequence model, in-context learning is driven by the existence of task-specific representations\nSwaminathan, Sivaramakrishnan, et al. \"Schema-learning and rebinding as mechanisms of in-context learning and emergence.\" Advances in Neural Information Processing Systems 36 (2023): 28785-28804.", "other_strengths_and_weaknesses": "The paper is very well written.", "other_comments_or_suggestions": "The authors should add in the limitation section that, in their simple framework (with mixture of sparse linear regression tasks) bases with shared support are not separated in the representation space, which limits their ICL performance.\nIn contrast, LLMs seem to be able to perform ICL reasonably well on \"tasks with shared support\" (e.g. acting as famous person P1 vs. P2).", "questions_for_authors": "(1) How many in-context example do you give to the TestMSE loss in Figure 2[left]?\n\n(2) Why does self-pertubation improve performance at convergence (Line 206, right)? \nI would imagine that the model knows how to separate the basis, so it would not benefit from patching.\n\n(3) In the definition of the TD score, how do you define the set of representations over which you select the neighbors? Also what is N=100?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates how transformers develop task vector representations during pretraining and their impact on in-context learning (ICL) performance. The authors propose an encoder-decoder framework (not to be confused with the encoder-decoder Transformer architecture. The paper still focuses on decoder-only Transformer architecture.) where earlier layers encode inputs into task vectors while later layers decode these representations to solve specific tasks. They use kNN classification accuracy as Task Decodability (TD) scores to quantify the separability of task encodings and demonstrate a strong correlation between TD scores and ICL accuracy across multiple language models and tasks. Through careful experiments with synthetic tasks and real LLMs, they validate this relationship and show that finetuning earlier layers improves task accuracy.

The reviewers unanimously recommend acceptance based on several strengths. The paper provides a compelling mechanistic interpretation of ICL through the encoder-decoder lens, supported by robust evidence from activation patching and partial fine-tuning experiments. The consistency of findings across different model architectures and both synthetic and naturalistic tasks demonstrates generality. The systematic investigation of task vector formation over the course of pretraining through OLMo-7B checkpoints was also highlighted as particularly insightful, providing concrete evidence that task-relevant encodings emerge early in the network and are critical for downstream performance.
Decision: Accept (spotlight poster)