In-Context Learning Creates Task Vectors
Authors: Roee Hendel
Mor Geva
Amir Globerson
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
In-Context Learning
Interpretability

Abstract: In-context learning (ICL) in Large Language Models (LLMs) has emerged as a powerful new learning paradigm. However, its underlying mechanism is still not well understood.
In particular, it is challenging to map it to the "standard" machine learning framework, where one uses a training set $S$ to find a best-fitting function $f(x)$ in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query $x$ and a single "task vector" calculated from the training set. Thus, ICL can be seen as compressing $S$ into a single task vector $\boldsymbol{\theta}(S)$ and then using this task vector to modulate the transformer to produce the output.
We support the above claim via comprehensive experiments across a range of models and tasks.

In-Context Learning Creates Task Vectors
Roee Hendel
Mor Geva Google DeepMind
Amir Globerson
Tel Aviv University
Tel Aviv University, Google gami rêtauex. tau. il
roee. hendel@mail. tau. ac.
pipekegoogle.com com
Abstract
Large Language Model (T)
Yellow
In-context learning (ICL. in Large Language Models (LLMs) has emerged as powerful new learning paradigm. However, its under- lying mechanism still not well understood. particular, it challenging map the "standard" machine learning framework, where one uses training set S to find best-fitting function (=x) in some hypothesis class. Here we make progress on this problem by showing that the functions learned by ICL often have a very simple structure: they correspond to the transformer LLM whose only inputs are the query x and single "task vector" calcu- lated from the training set. Thus, ICL can be seen as compressing into a single task vector 0(S) and then using this task vector modu- late the transformer to produce the output. We support the above claim comprehensive ex- periments range of models and tasks.!
θ
A
layers
Apple
Red
Lime
Green
Corn
-
Demonstrations (S)
Query (x)
Figure ICL learning Hypothesis Class. ICL, provides an LLM with prompt including demonstrations of some task, and a query x. The model generates the output for x (here "Yellow"). We show that the underlying process can be broken down into two parts: A, a "learning algorithm" (marked in blue), computes a query-agnostic vector ((S), which we view as a parameter of function in a hypothesis class. The second part, denoted by f and marked yellow, is the application of the rule defined by 0 on the query without direct dependence on S.
1 Introduction
Large language models have improved dramatically over the last several years. One striking property of these models is that they can learn new rules from very few demonstrations. For instance, a model can be prompted with the input "Apple + Red, Lime + Green, Corn - and produce the output "Yellow". The model has thus learned mapping based on just two examples, which it can apply correctly to new examples. This capability, referred to In- Context Learning (ICL), has been used extensively, yielding impressive empirical results (Brown et al.. 2020; Liu et al., 2023; Dong et al., 2022).
concept of a hypothesis class from statistical learn- ing theory Shhalev-Shwarre and Ben-David, 2014). In the learning-theoretic formulation, one typically considers a hypothesis class H. where every ele- ment of 7 is function hx:; 0), operating on the input and specified by parameter vector 0. For example, if RRd then the class H could be the of linear classifiers. defined by a coefficient vec- tor 0 as h(#; 0) an element E H. that fits the training set well. This is known as Empirical Risk Minimization.
Given this success, it is natural to ask what is the underlying mechanism behind ICL. Namely, how does the model internally use the demonstrations S and the query z to produce the required output? Here we approach this question by utilizing the
r. Learning algorithms seek
It is unclear whether ICL operates in such a way because the prediction performed via T(SS, x]). where typically an ruto-regressive transformer
We release our code at httpss://githme com/ saahendel/iccl_tak_eeeee

---

and [S, ar] is concatenation of the tokens in S and z. Thus, in the general case, it can be an arbitrary function that operates on S and x to produce the output. This can include inoo-parammtrin methods such as nearest-neighhor. Recent work has begun to explore this question. For example, it was shown that when training a transformer from scratch to perform linear regression in context, the emerging learning algorithm is similar to Stochastic Gradient Descent (Akyürek et al., 2022; von Oswald al., 2022). However, for LLMs performing more com- plex natural language tasks, it is not at all clear what the hypothesis space may be.
2.1 Theoretical Framework
We use T to denote a decoder-only transformer LLM, S to denote the set of demonstrations (i.e. training examples) used as input to ICI., and to denote the query that ICL is asked to provide an output for. We use T((S, 2:]) to denote the output of ICL on the concatenation of S and x.
To demonstrate that ICL operates within a hy- pothesis space, we aim to show that its underlying mechanism can be broken down into two parts:
"Learning Algorithm" (denoted by A) that maps S into a "task vector" 6, independent of the query x. Given that attention layers can access both S and z, this independence is not trivial.
In this work, we show that on a wide range of tasks, ICL in LLMs can be viewed as working on a very natural hypothesis space. We argue that, given training set S, the transformer maps it into a "task vector" (SS) that essentially represents the mapping/rulc described in S.² Namely, given the transformer T and vector 0, we can construct a new function (x; 0) that implements the task. The function f is very similar to the original trans- former applied to x without demonstrations but instead modulated by O (see Fig. 2).
A "Rule Application" (denoted by f) which maps the query x to the output, based on 0 = A(S), without direct dependence on S. Again, this independence is not trivial.
Thus, we consider the following mapping from a set of demonstrations and a query to the predicted output: T[SS, f(x; A(S))
If we break down the forward pass of the LLM into the above two components, we can view ICL as operating on the following hypothesis class: H = {- 0) 0). In the next section we propose an implementation of such a class.
Our view is also related to soft prompts (Lester al., 2021), since both approaches modulate the function of the transformer towards particular task. However, in ICL, task vectors are calculated in the forward pass rather than being fine-tuned. Our contributions include proposing a hypothesis-class based mechanistic view of ICL, and conducting experiments to validate our view on a range of publicly available LLMs and diverse set of tasks. Our results further the understanding of ICL and may have practical implications for the efficient adaptation of LLMs to perform specific tasks.
2.2 A Proposed Hypothesis Class
There are many possible realizations of the above framework, that correspond to different choices of A and f. We next describe the realization we focus on, which naturally follows from the trans- former architecture. We consider an ICL setting as in Fig. I, where the input ends with a query x (i.e., Corn) followed by an " symbol. As mentioned above, we view learning as composed of two steps: calculating parameter vector 0 based on the train- ing sample S, and applying the rule defined by this parameter vector to the query x. A presumably simple way for a transformer to do this is for the first L layers of the -> representations to calculate and then for the remaining layers to take and x as input and produce an output. See Fig. 1. Recall that S and x are accessible to the transformer at any layer, presenting a challenge with our view. In the following sections, we address this chal- lenge and present experiments validating our view. Namely, we show that we can isolate our proposed A and f in the forward pass of L.LMs performing ICL. We also show that the vectors are inter- pretable and correspond to learned tasks.
2 A Hypothesis Class View of ICL
Motivated by the hypothesis class view of learning theory, our goal is to understand if ICL maps the set of demonstrations S to function the query and how this mapping occurs. Specifically, we seek see if ICL converts S into the "parameters" of a function within a certain hypothesis space. Our empirical findings suggest this view is applicable, shedding light on the structure of the hypothesis space on which ICL can be viewed to operate.
'The term "task vector" was coined by Ilharco et al. (2023) for directions in weight spuce that correspond to particular task. Although our vectors are in "activations space" they share a similar motivation and thus we overload the term.

---

(a)
(b)
Category
Task
Example
Sec- Large Language Model (T)
Yellow
Next letter
- b a.b,c +
List first
Algorithmic
List last
a.b,c +
To uppercase
-- A
French to Finlishh Spanish to English
bonjour + hello hola + hello
Translation
Present to gerund
go + going cat - cats
θ
Linguistic
Singular to plural Antonyms
happy + sad
A
"Color"
Knowledge
Country to Capital Person Language
France + Paris
Macron - French
Table 1: A representative subset of the tasks used in the study with input + output examples.
Apple
Red
Lime
Green
Plum
Com
Average Accuracy by Layer
LLaMA
Demonstrations (S)
Query (x)
LLaMA 130
LLaMA 308
Figure 2: Separating A and f. To make 0 independent of the query z, we use dummy query (x' Plum) and use the representation of at the 1. layer as 0. The vector is then patched at the same layer during forward pass of transformer that only takes and as input, to prevent the direct dependence of on S.
Accurec
0.2
0.0
30
3 Validity of the Hypothesis Class View
Layer
Figure Accuracy for each choice of the intermediate layer L, averaged across all tasks. Solid lines show average values, and shaded areas standard deviations.
We first show that separating the forward pass into the two distinct components A and f, defined in $2.2,
maintains the high accuracy of ICL.
3.2 Tasks and Models
3.1 Separating A andd
Tasks We consider a diverse set of 18 tasks across categories: algorithmic, translation, linguistic, and factual knowledge. For simplicity, we limit ourselves to single-token outputs. A representative subset of the tasks is described in Tab. 1. A com- plete detailed table, as well more information regarding the data, are provided in A.1.
We face some challenges in a regular forward pass: first, the initial L layers that correspond to A. up- dating the representations of + to create 0, can attend to the query Thus, they may depend on creating unwanted dependence of 0 ond, the remaining layers that correspond f, may directly access S, instead of using only x and
We propose the following procedure to tackle these challenges: to solve the first problem, we introduce "dummy query" x' and calculate the representations of - using that query. We use the representation of + after the first L layers, calcu- lated using x'. the vector 0 (as demonstrated on the left side of Fig. 2). An alternative was to block attention to x, but it led to poor performance. To solve the second problem of calculating f (ax 0) without allowing direct dependence on S. we per- form a forward pass of the transformer only on and + and "patch" the 0 we previously extracted at the Lth layer of the (right side of Fig. 2).4
Models We use multiple open LLMs: LLaMA 7B, 13B, and 30B (Touvron et al., 2023), GPT-J 6B (Wang and Komatsuzaki, 2021), and Pythia 2.8B, 6.9B, and 12B (Biderman et al., 2023).
3.3 Finding L
The mechanism we described in $2.2 has free parameter the layer where A ends and begins. We use the proposed (A, implementation for different choices of L and evaluate the accuracy on development set to find the best layer.
Fig. 3 shows the accuracy the development set, for different choices of L. We focus here the LLaMA models and include the rest in $ A.2. In- terestingly, all models exhibit performance peak at similar intermediate layer, irrespective of their parameters and layer count differences.
"Ignoring positional embeddings, this equivalent to blocking the atlention in these layers. "Note that the second token can actually he anything, he- cause is overridden by patching. We use + for simplicity.

---

Pythia 128
40
Pythia 6.98
Pythia 2.88
LLaMA 30B
0-
LLaMA 138
LLaMA 7B
-20
GPT-j 6B
0.0
0.2
0.4
0.6
0.8
1.0
-40
Accuracy
Baseline
BB
Regular
Hypothesis
-40 -20
20
40
Figure Average accuracy across all tasks for each model, using each of the three procedures: Baseline, Regular and Hypothesis.
Figure 5: A t-SNE plot of task vectors. A 2D t-SNE plot visualizing 50 task vectors for each task, each gen- erated from different choice of S and z' using LLaMA Points are color-coded according to task. Each task be to form its own distinct cluster.
3.4 Accuracy of Hypothesis Based Prediction
We next compare the accuracy of the (A,, f') mech- anism to that of a regular forward pass performing ICL. For each model and task, we evaluate the following three procedures:
To test this, we use LLaMA 7B to generate 50 task vectors per task with varied S and x' and con- duct two analyses.
Regular An application of the LLM to the Namely
Geometry of o A t-SNE dimensionality reduc- tion (Fig. 5) reveals that the task vectors form dis- tinct clusters, each containing task vectors of a sin- gle task. Fig. 9 further shows proximity between tasks of the same category, strengthening the idea that they encapsulate task understanding.
demonstrations S and query T((S, as in regular ICI.
Hypothesis Our proposed procedure from $ 3.1 where A generates 0 using dummy z', and f(: 0) is applied to x by running the transformer [x. + with 0 patched layer L of -+.
Variability of 0 Fig. shows histograms of dis- tances within and across tasks. can be seen that vectors within the same task are closer than those between different tasks, indicating that 0 is stable within tasks and not highly influenced by x' or S.
Baseline A forward pass of the LLM only on without demonstrations S. That is, T(xx +))
This is the same as the application of f from separated procedure, but without patching
Fig. 4 shows the average accuracy across all tasks of these 3 procedures, for each model. Full results are reported in Tab. 6 in $ A.2. Across all models, our procedure maintains around 80-90% the accuracy of regular ICL, while the baseline reaches only 10-20%. This shows that our proposed separation to AA and f provides good empirical approximation of the process underlying ICL.
Dominance of Patching
In $3 we prevented f from directly accessing S. However, in regular forward pass during ICL, the last token can attend to Here we verify that even in this case, f mainly uses the task vector 0, without directly accessing the demonstrations S. To this end, we use a pair of tasks. A and B, sharing the input space but differing on the output. We first use "Regular" forward pass, where we provide the model with demonstrations S for task A (denoted SA). to verify the model can perform this task using ICL. Then, we do "Conflicting" forward pass, still providing SA. while injecting Øg. For more details, refer to Fig. in $A.1.
4 Robustness of Task Vectors
In our setting, 0 is derived from S and dummy query x'. It is natural examine the robustness of 0 to variations these inputs. Intuitively, if represents the task, should remain stable across different and x' values.

---

Task A (S)
Task B (0)
Regular Task A
Conflicting Task B
Task
Top tokens in the task vector projection
Previous
e, unknown, alphabet, preceding, Cad, ZA, dit, bill
Next Letter
To Upper
0.92
0.77
Letter
List Last
List First
0.95
0.78
FR-EN utter, Span, Cone, English, equivalent
Mason, gram, immer, Santi, latin,
Present to Past
Gerund
0.96
0.95
Table 2: Conflicting tasks experiment results. The model's accuracy on the relevant task (A in "Regular" and in "Conflicting") is displayed for both scenarios.
Present
cin, thats, gram, Lorenzo, cian, Isabel, uld, berto, partici, Sah
Simple to Gerund
Country Capital
Paris, its, capital, central, Conc, cities, administrative, Los, Madrid, London
In Tab.2, the "Regular" forward pass shows high accuracy on task A (90%+), as anticipated. How- ever, the "Conflicting" forward pass yields high accuracy on task B, corresponding to the injected task vector O. This implics that the model mainly relies on 0, largely disregarding the demonstrations S for task A. We note that the accuracy on task B is slightly low, likely consistent with the perfor- mance dip seen in Fig. 6, and potentially further affected by the presence of S.
Table 3: The top tokens distribution induced by the task vector, for one task per category.
Meta-Learning in Transformers Studies by Akyürek et al. (2022); von Oswald et al. (2022); Garg et al. focus on the meta-learning capabilities of transformers. They typically train models from scratch on elementary tasks such as linear regres- sion, drawing theoretical parallels with algorithms like Gradient Descent and demonstrating how trans- formers could implement them. A key assumption of these works is known parameter space within which gradient descent operates. Our work focuses on identifying such parameter space for LLMs.
6 Interpreting 0
The learned vector 0 intuitively captures informa- tion about the task demonstrated by S. Here we pro- vide evidence supporting this interpretation. Since 0 is an intermediate hidden state of the transformer, we can employ a vocabulary projection method (nostalgebraist, 2020; Dar et al., 2022). Namely, we examine the top tokens in the distribution over the vocabulary induced by the hidden state.
ICL in LLMs Olsson et al. (2022) identify "in- duction heads" in transformers as likely main mechanism of ICL. Dai et (2022) provide empir- ical evidence for the connection of ICL. Gradient Descent in LLMs, focusing on classification tasks. Concurrent work by Merullo et al. (2023) also ex- plores a phenomenon similar to the task vectors we study here, where single vector can encode learned functions. Our findings are complemen- tary to theirs, and future work could explore the relationship between the two more closely.
Tab. 3 shows the top tokens for three tasks for LLaMA 13B (more models and tasks are provided in Tab. 7 in SA). In multiple cases, we observe to- kens that directly describe the task. Importantly, these terms never explicitly appeared in the context. For example in the task of translation from French English, we observe tokens such as "English" and "translate". This supports our view that O car- ries significant, non-trivial semantic information about the task.
8 Conclusions
Through this exploration of ICL in LLMs, we have shed light on new perspective of ICL learning mechanisms. We have revealed a simple and el- egant structure: ICL functions by compressing a given training set into single task vector, which then guides the transformer to generate appropri- ate outputs given queries. Our work provides a stepping stone towards understanding how LLMs perform ICL. In light of our findings, future work could focus on understanding how the task vector is constructed as well as how it is used to calculate the output.
7 Related Work
Emergence of ICL A key question with ICL is how it emerges as capability from pre-training the LLMs. Levine et al. (2022) provides results in this direction that highlight the importance of training data structure. Xie et al. use probabilistic analysis and model pre-training data using Hidden Markov Models theoretically explain the emergence of ICI, while Chan al. (2022) empirically explore the effect of several distributional properties of the pre-training data.

---

Limitations
Qingxiu Dong, Lei Li. Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.002
We study relatively simple tasks, whereas ICL can learn to perform more complex tasks, such as solv- ing arithmetic reasoning problems. It remains to be seen if and how the mechanisms we observe here will translate to these cases. E.g., our approach fo- cuses on cases where a single task vector suffices, while more complex ICL cases may require more elaborate parameterization. We also focus tasks where the oulput is a single token, while some other tasks require multi-token outputs.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gre- gory Valiant. What can transformers learn in- context? a case study of simple function classes. In Advances in Neural Information Processing Systems.
Gabriel Ilaarco, Marco Tulio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learn- ing Representations.
Finally, as noted above, we do not provide a mechanistic explanation for how the task vector is formed or how it is used. Namely, we do not explain how the transformer performs these calcu- lations using its parameters.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficieet prompt tuning. arXiv preprint
100::2004.861...
Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. 2022. The in- ductive bias of in-context learning: Rethinking pre- training example design. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
A Acknowledgements
This project is funded by the European Research Council (ERC) under the European Unions Hori- zon 2020 research and innovation program (grant ERC HOLI 819080).
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cexsing Systems, 35:17359-17377
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:230..0137..
Jack Merullo, Carsten Eickhoff, and Ellic Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic. arXiv preprint arXiv:2305. 16.3. 16130.
nostalgebraist. 2020. interpreting gpt: the logit lens. Less LessWrong.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint 220i::2209.11999.
Shai Shalev-Shwarzz and Shai Ben-David. 2014. Un- derstanding machine learning: From theory to algo- rithms. Cambridge university press.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. Data distributional properties drive emergent io-context learning trans- formers. Advances in Neural Informaiion Processing Systems, 1811888888899
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Ann Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.130
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in- in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10S:
Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsev, An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent. arXiv preprint 22X00:2212.07677.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022. Analyzing transformers in embedding space. arXiv preprini arXiv:2209.02533,

---

Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: 6 Billion Parameter Autoregressive Lan- guage Model, https::/ggttubb.com//iiing cot.:/ino//oglll// mesh-traanformmr-jjax
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learn- ing implicit bayesian inference. In International Conference on Learning Representations.

---

A Appendix
Dependence of A on Fig. and Fig. provide more results on the geometry of the vectors (see main text for discussion).
Here we provide additional details and results.
A.1 Additional Details
Inspecting Task Vectors Tab. 7 is an expanded version of Tab. 3. providing more vocabulary pro- jections of 0 for additional tasks and on multiple LLMs.
Full Task Descriptions Our study covers 18 tasks in 4 categories: Algorithmic, Translation, Lin- guistic and Knowledge. A detailed description of all tasks is provided Tab.
Model
Parameters
Dimension
Layers
Heads
Model Details More details on the models used in the study are provided in Tab. 4.
4096
32
32
LLaMA
13B
5120
40
6656
60
Task Data Here we detail the sources of the data for each task. The accompanying GitHub reposi- tory contains the data itself as well as the code used to create it.
GPT-J
6B
4096
28
16
2.8B
2560
32
32
Pythia
6.9B
4096
32
32
12B
5120
36
40
Algorithmic: Generated programatically.
Table 4: The models used in the study, with architectural information.
Translation: For each language pair, the most frequent words in the source language are first retrieved from https://githhu: http:ettthh....ooooeeeeeeeeee frekwencja/ most-common-uee wwwdswwultillnneeeeeeeeee and are then translated to the destination language using the open-source package nltk.
Linguistic: The data for the tenses tasks is parsed from https://gi thub. com/Drulac/ Engllsh--Verbs--oonjjgteee
The data for the plural-singular task is taken from https://gtt thub. www.cooo/ssioomoo com/ssinreeorhus/
irregular-plurals.
Finally. the data for the antonyms task is taken from https:///itthut thub. com/ Suzanak/ Baraanaa//nllann ls synonyms_antonyms.. list.
Knowledge Data for the knowledge tasks is taken from the counterfactual dataset intro- duced in (Meng et al., 2022).
Conflicting Tasks Experiment In Fig. 6, we pro- vide more details and a visualization of the experi- ment described in $5.
A.2 Additional Results
Finding A and f Fig. 7 shows results similar to Fig. 3, but for different models. It is interesting to observe that the curves are similar across different- sized models.
Detailed results for Fig. 4. Fig. presented re- sults for our (A, f) hypothesis-basee approach, av- eraged across tasks. Table. provides these results for all the specific tasks considered.

---

Category
Task
Description
Example
List first List last
Given list of letters, output the first letter
a.b,c
Given list of letters, output the last letter
a,b,c
Algorithmic
Next letter
Given letter in the English alphahet, outpur the next letter
h
Previous letter
Given letter in the English alphabet, outpur the previous letter
b
To lowercase
Given uppercase letter, output the correspond- ing lowercase letter
A
To uppercase
Given lowercase letter, output the correspond- ing uppercase letter
French to English Spanish to English
Given word in French, translate to English
bonjour - hello
Translation
Given word in Spanish, translate to English
hola + hello
English to Spunish
Given word in English, translate to Spanish Given word in English, translate to French
hola + hello
Finglish Spanish
hola + hello
Present to gerund
given an English verb in present simple tense, output the corresponding gerund form given an English verb in present simple tense, output the corresponding verb in past simple
+ going
Linguistic
Present to past
+ went
Singular to plural
Given an English noun in singular form, output the plural form Given an English adjective, output antonym
cat cats
Antonyms
happy + sad
Country to Capital
Given a name of country, output the name the capital city
France - Paris
Knowledge
Person to Language
Given a name of person, output their native
Macron + French
language
Location to Continent
Given a name of person, output their native
Paris + Europe
language
Religion
Given name of location or person, output the associated religion
Muhammad + Islam
Table 5: The tasks used the study with input - output examples.

---

Regular
b
b
q
Demonstrations (SA)
Query (x)
"Previous Letter" (Task A)
Conflicting
bkd
(SSp)
"Nēxt Letter" (Task B)
q
Demonstrations (Sa)
Query (x)
"Previous Letter" (Task A)
Figure 6: Conflicting tasks experiment. In the "Regular" scenario (top), the model is simply provided with demonstrations SA for Task A (c.g. outputting the previous letter in the alphabet). In the "Confficting" scenario (bottom), the model is still provided with demonstrations for Task A, but inject task vector 0(SB) from conflicting Task (c.g. outputting the next letter in the alphabet).

---

Average Accuracy by Layer
1.0
Pythia 2.8B
Pythia 6.9B
0.8
Pythia 12B
GPT-J 6B
Accuracy
0.6
0.4
0.2
0.0
0
5
10
15
20
25
30
35
Layer
Figure 7: Accuracy for each choice of L (the intermediate layer where the task vector is injected), averaged across all tasks. The solid line represents the average value, and the shaded area depicts the standard deviation.

---

Table 6: Complete results for Figure 4, reported for all tasks and models.
method
Baseline
Hypothesis
Regular
Model
Task type
Task name
GPT-J 6B
Algorithmic
List first
0.30
0.74
0.98
List last
0.24
0.64
1.000
Next letter
0.16
1.00
0.86
Prev letter
0.10
0.36
0.42
lower
0.00
0,46
1.00
upper
0.00
0.94
1.00
Knowledge
Country capital
0.19
0.72
0.80
Location continent
0.03
0.70
Location religion
0.09
0.78
Person language
0.02
0.82
0.82
Linguistic
Antonyms
0.43
0.68
0.78
Plural singular
0.08
0,90
Present simple gerund
0.00
0.88
0.98
Present simple past simple
0.02
0.76
0.96
Translation
Ein es
0.14
0.34
0.56
En fr
0.16
0.36
0.54
Es en
0.06
0.70
0.74
Fr en
0.13
0.66
0.76
LLaMA 13B
Algorithmic
List first
0.77
1.00
1.00
List last
0.07
0.70
0.92
Next letter
0.31
1.00
0.94
Prev letter
0.05
0.34
0.50
To lower
0.00
0.94
1.00
To upper
0.00
0.94
1.00
Knowledge
Country capital
0.17
0.84
0.86
Location continent
0.01
0.70
0.80
Location religion
0.10
0.74
0.84
Person language
0.02
0.76
0.88
Linguistic
Antonyms
0.19
0.74
0.80
Plural singular
0.24
0.84
0.88
Present simple gerund
0.00
0.96
0.96
Present simple past simple
0.01
1.00
0.98
Translation
En es
0.05
0.78
0.82
En fr
0.15
0.70
0.84
Es en
0.29
0.76
0.88
Fr
0.25
0.54
0.72
LLaMA 30B
Algoritmmic
List lirst
0.96
0.98
1.00
List last
0.02
0.64
0.96
Next letter
0.30
0.98
0.96
Prev letter
0.02
0.56
0.80
lower
0.00
1.00
1.00
upper
0.00
0.90
1.00
Knowledge
Country capital
0.27
0.72
0.88
Location continent
0.01
0.70
0.86
Location religion
0.05
0.70
0.88
Person language
0.72
0.90
Linguistic
Antonyms
0.37
0.76
0.82
Plural singular
0.21
0.84
0.90
Present simple gerund
0.00
0.76
0.98
Present simple past simple
0.02
0.98
1.00
Translation
En es
0.07
0.74
0.78
En fr
0.10
0.80
0.86
Es en
0.24
0.70
0.88
Fr en
0.20
0.62
0.78
LLaMA 7B
Algorithmic
List lirst
0.87
0.98
1.00
List last
0.03
1.00
1.00
Next letter
0.03
0.94
0.88
Prev letter
0.04
0.52
0.58
To lower
0.00
0.74
1.00
To upper
0.00
0.60
1.00
Knowledge
Country capital
0.28
0.82
0.80
Location continent
0.02
0.68
0.72
Location religion
0.12
0.84
0.94
Person language
0.02
0.68
0.78
Linguistic
Antonyms
0.33
0.74
0.76
Plural singular
0.15
0.84
0.88

---

Table 6 continued from previous page
method
Baseline
Hypothesis
Regular
Model
Task type
Task name
Present simple gerund
0.00
0.74
0.90
Present simple past simple
0.02
0.94
0.92
Translation
En
0.07
0.78
0.76
Fn fr
0.04
0.78
0.88
en
0.21
0.68
0.92
0.15
0.66
0.70
Pythia 12B
Algorithmic
0.53
0.96
last
0.09
0,98
1.00
Next letter
0.15
0,96
0.76
Prev letter
0.00
0.24
0.42
To lower
0.02
1.00
1.00
upper
0.00
0,98
1.00
Knowledge
Country capital
0.19
0.58
0.82
Location continent
0.01
0.68
0.80
Location religion
0.07
0.64
0.78
Person language
0.01
0.72
0.86
Linguistic
Antonyms
0.34
0.72
0.74
Plural singular
0.18
0,80
0.84
Present simple gerund
0.00
0.86
0.96
Present simple past simple
0.01
0.76
0.94
Translation
En cs
0.10
0.44
0.72
En fr
0.16
0.48
0.54
Es en
0.05
0.68
0.80
Fr en
0.14
0.68
0.80
Pythia 2.8B
Algorithmic
List first
0.69
0.96
1.00
List last
0.06
0.98
1.00
Next letter
0.42
0.86
0,90
Prev letter
0.01
0.22
0.48
To lower
0.00
1.00
1.00
To upper
0.00
1.00
1.00
Knowledge
Country capital
0.18
0.70
0.76
Location continent
0.01
0.62
0.72
Location religion
0.08
0.76
0.82
Person language
0.00
0.82
0.82
Linguistic
Antonyms
0.37
0.68
0.76
Plural singular
0.13
0.70
0.78
Present simple gerund
0.00
0.86
0.96
Present simple past simple
0.03
0.80
0.92
Translation
En
0.10
0.26
0.76
En fr
0.16
0.28
0.60
0.08
0.76
0.82
0.10
0.64
0.82
Pythia 6.98
Algorithmic
0.43
1,000
0.98
List last
0.08
0.60
0.98
Next letter
0.86
Prev letter
0.04
0.28
0.32
lower
0.00
1.00
1.00
To upper
0.00
0.94
1.00
Knowledge
Country capital
0.21
0.76
0.82
Location continent
0.01
0.62
0.78
Location religion
0.10
0.80
0.80
Person language
0.01
0.76
0.80
Linguistic
Antonyms
0.33
0.72
0.74
Plural singular
0.14
0.78
0.88
Present simple gerund
0.00
0.82
0.94
Present simple past simple
0.02
0.88
0.96
Translation
En cs
0.11
0.46
0.70
En fr
0.21
0.36
0.60
Ess on
0.06
0.72
0.82
Fr en
0.14
0.66
0.74

---

translation fr
Innuuisiic present simple gerundnowleedge country capital
algorithmic next letter
15.0
Within task
25
Within task Other tasks
Within task
Within task
12.5
Other tasks
20
Other tasks
15.0
Other tasks
Dersity
15
6-
10.0
10
3.0
2-
2.5
0.00
0:25
0.50
0.75
0.00
0.25
0.50
0.75
0.0
0.2
0.4
0.6
0.00
0.25
0,50
0.75
translation es, en
transiation en
translation en_ linguistic, present simple past simple
Within Other tasks
15
Other tasks
Other tasks
Other tasks
30.0
Denstty
7.5
5.0
2.5
0.0
linguistic plural singular
0.25
linguistic antonyms
0.50
0.75
0.50 knowledge person languagdknowledge location continent
0.25
0.50
0.75
0.00
0.25
20
Within task
Within task
15.0
Within task
Within task
00.0
7.5
5.0
2.5
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.00 algorithmic list last
0.50
knowledge location religion
algorithiic prev letter
algorithmic list first
20
Within
Other tasks
Other tasks
Other tasks
Other tasks
0.00
0.25
0.50
0.75
0.00
0.25
030
0.75
0.00
0.25
0.50
0.75
0.0
0.2
0.4
0.6
algorithmic upper
algorithmic, to
Within task
Within task
12.5
Other tasks
20
Denslty
10.0
15
10
2.5
0.0
Cosine Distance
Casine Distance
Figure 8: Task Vector Variability. For each task, two histograms are shown: (blue) the distribution of distances between different task vectors of this task, created from different æ'; (orange) the distribution distances between task vectors of the task and of other tasks.

---

40
fflection_presen present, simple_pass past_ simple
inlectionn present_simplees.ee..
translation_en en
poauctionsingular_pliee ppural
20
translatio en_fr
kg owledge location religoon
knolledge Becteedgeeeeessn____ perssonJangguage
translation fr en
atonyms
translation__s_ _e__on______
in miledge_counttycaplll knwwedde wewledge_location..mm continent
_ocaation_coniieent
engorithmic to_ttt___ppp upper
elgorithmic to Jower
a orithmic last
algonthmic_ first
algorithmic_neett http_eeetteeeeeeeee Jetter
-40
gorithmic_pree_lette letter
Figure 9: A 2D t-SNE plot, visualizing 50 task vectors for each task, each generated from a different choice of S and ø using LLaMA Points are color-coded according task category, such as algorithmic or translation. Each task can be seen form its distinct cluster. The labels provide the full name of the task in the cluster.

---

Model
Task
Tokens
Prev Letter
e, y, unknown, alphabet, preceding, c, Cad, ZA, dit, bill, closer, etc, Stuart, aa, null, cin, ads, g, ulo, Ku
FR-EN
Mason, gram, immer, Santi, latin, utter, Span, Conc, English, equivalent, engl, Usage, none, pron, ulo, translate, adu, Wiel, granmar, ML
LLaMA
Present Simple to Gerund
cin, thats, gram, Lorenzo, cian, Isabel, uld, berto, partici, Sah, reporting, eing, tc, Roberta, habit, Writing, etc, ientos, ores, Dutch
Country Capital
Paris, its, capital, central, Conc, cities, administrative, Los, Madrid, London, San, Isabel, exec, Ar, Bel, Wars, name, capit, Battle, History
Letter
r. b, a. d. n. e, P. u. h, f. c, in, g. the, ar, 1, in, and, m, d, or, out, the, all, to
FR-EN
c, have, act, e, is,
Pythia 12B
Present Simple to Gerund
in, t, m, a, and, the, ing, action, d, o, current, simple, te, not, have, out, what
Country Capital
the, in, a, C, N, B, M, T, P. s, R, G, and, F, I, K, U, D, b. c, v. 8. s, name, i, ro, n, j. d, t, A, ai, com, m, ust, test, active,
Prev Letter
FR-EN
other, name, the, true, is, social, 5, active, time, type, money, life
F, force, public, heart,
GPT-J 6B
Present Simple
getting, storing, working, moving, playing, doing, making, driving, shooting, picking, being, sending, putting, selling, watching, changing, taking, collecting, feeding, reading
to Gerund
Country Capital
London, Paris, New, West, Berlin, South, Tokyo, San, Chicago, City, Mascow, Jerusalem, Amsterdan, Philadelphia, East, Madrid, Vienna, Beijing, Mexico, Germany
Table 7: The top 20 tokens in the distribution induced by the task vector, for one task per category.


Reviews:
[{"review_id": "SqWI1cUHsU", "reviewer": "Reviewer_s9kR", "summary": "", "questions": "- The exact mathematical formulation of f(x, \\theta(S)) is unclear. Is it an inner product between two representations? \n- Coming to the usage of x’ instead of x to get the representation, the model has seen the query x’. Will this not tamper the Vector for S( i.e training data for the ICL) as x’ does not have a token succeeding “->” ?\n- Why do you think intermediate layers are more suitable as shown in the results in Figure 2? Also, is this pattern consistent across all subsets of tasks? \n- What’s the variance of \\thta(S) for different choices of x’? Is it very low? If that's not the case it will be hard to conclude that the transformer uses task vectors. \n- Are there any insights based on results for different types of tasks? \n- To elaborate, more experiments on validating whether tasks from a similar set  of tasks (Ex : En -> Fr, Fr -> En ) is nearer in n-d dimensional space rather than tasks that is dissimilar (Fruits -> Colour) would have made the analysis more stronger towards the positioning that ICL creates task vectors. This could have been validated with methods like t-sne etc.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoroughly reading our work and for the great questions!\n\nWe address the concerns under \"reasons to reject\":\n\n**(1) Regarding the variance of θ(S).**\n\nThanks for this great comment. We first explain why the variance between task vectors of the same task, obtained from different demonstrations S and query x', doesn't necessarily have to be low for our hypothesis-class view to hold. We then report new analysis results to show that, in practice, it is low.\n\n- The variance between task vectors of the same task does not necessarily have to be low: Note that for the hypothesis-class view to hold, we don't require that the thetas are close to each other in Euclidean space, but rather that they are \"functionally equivalent\" in the sense that they define the same function when used in f(x,θ). More formally, we can define the set of vectors Θ as all the θ that can be generated by different choices of x'. Each member of this set is a task vector, in the sense that it can be used with any query x to produce the correct output using f(x,θ). This phenomenon, where multiple parameter vectors (analogous to our task vectors) exist for any single dataset, can also happen in classical learning algorithms (e.g., there could be many hyperplanes that separate the same dataset). The above explanation shows that the variance does not necessarily have to be low in order for our hypothesis-class view to hold.\n\n- New results: Regardless of the above discussion, we agree with the reviewer that analyzing the variance of the task vector under different choices of x' (and S) is interesting, and we conducted an experiment following the reviewer's comment. We used LLaMA 7B and for each task, created a plot with two histograms, showing the cosine distances between (a) task vectors of the same task (with different S and x'), and (b) task vectors of different tasks. Across all the tasks, it is possible to see a clear separation between the two histograms, which provides evidence that the variance of task vectors of the same task is substantially smaller compared to between vectors of different tasks. Unfortunately, because of the restrictions on the rebuttal to include only text without links, we cannot share the plots. Therefore, we use the table below to report the mean and standard deviation of the aforementioned histograms, where the separation can be observed. We will include the full analysis in the revised manuscript.\n\nWe hope that this clarified formulation and further analysis answers your concerns, and are happy to answer any further questions. We are also happy to provide the plots if the program chairs allow.\n\nThe following table displays the new results. It includes the mean and standard deviation of the distances between task vectors within a task, as well as the distances to task vectors from other tasks. We can see that in all cases the in-task distances are much smaller compared to the distances to other tasks.\n\n| Task               |   Within Task Mean |   Within Task Std |   Between Tasks Mean |   Between Tasks Std |\n|:-------------------|---------------:|--------------:|--------------------:|-------------------:|\n| Next Letter        |          0.126 |         0.04  |               0.53  |              0.112 |\n| Prev Letter        |          0.128 |         0.048 |               0.557 |              0.111 |\n| List First         |          0.106 |         0.03  |               0.54  |              0.114 |\n| List Last          |          0.103 |         0.029 |               0.519 |              0.117 |\n| To Upper           |          0.084 |         0.029 |               0.464 |              0.102 |\n| To Lower           |          0.089 |         0.05  |               0.51  |              0.105 |\n| Fr En              |          0.159 |         0.038 |               0.5   |              0.12  |\n| Es En              |          0.115 |         0.034 |               0.524 |              0.086 |\n| En Fr              |          0.139 |         0.04  |               0.465 |              0.125 |\n| En Es              |          0.135 |         0.045 |               0.5   |              0.127 |\n| Gerund             |          0.087 |         0.023 |               0.48  |              0.145 |\n| Past Simple        |          0.094 |         0.026 |               0.465 |              0.138 |\n| Singular Plural    |          0.145 |         0.044 |               0.453 |              0.123 |\n| Antonyms           |          0.141 |         0.035 |               0.495 |              0.092 |\n| Country Capital    |          0.156 |         0.041 |               0.53  |              0.049 |\n| Person Language    |          0.13  |         0.037 |               0.57  |              0.085 |\n| Location Continent |          0.142 |         0.05  |               0.665 |              0.08  |\n| Location Religion  |          0.149 |         0.043 |               0.578 |              0.09  |\n\n\n**(2) Regarding the concern that \"... As even if the probe vectors show performance with a computation similar to the hypothesis class, it won't still be sufficient to imply that the hypothesis class is true of this particular type. Such empirical results in support of the claim are necessary. but not sufficient.\"**\n\nOur view of a hypothesis class is a set of functions that any learned function will be a member of. In that sense, we do show that _the functions learned by ICL are in the class of transformers with task vectors as input_. We agree that we do not show mechanistically how these vectors are computed by the transformer (e.g. is there some implicit gradient descent implemented). However, we think that characterizing such a simple class of functions as the hypothesis class is an important step forward. We will make sure to clarify this.\n\nWe further answer the reviewer's questions:\n\n**Q1: The exact mathematical formulation of f(x, θ(S)).**\n\nThe notation f(x, θ(S)) does not refer to an inner product or specific mathematical operation. It simply symbolizes a computation f that relies on a query x and a task vector θ(S), without direct dependence on S.\n\n**Q2: The usage of x' instead of x to get the representation.**\n\nWe appreciate the opportunity to clarify the role of x' in our framework. Note that x' is not part of the demonstrations set S, and its label is intentionally not provided. Rather, it functions as a query, keeping the structure of the prompt and signifying the type of input for which an output is expected. To elaborate, recall that the task vector is essentially an intermediate representation of the -\\> token. To keep the structure of the prompt, the -\\> needs to be preceded by an input, and we therefore put x' there. While in principle the task vector could be influenced by x', our empirical data, further supported by the above new histogram data (from the answer to concern (1)), demonstrate that its effect is weak.\n\nEssentially, x' helps in keeping the structure of the prompt for extracting the task vector from the demonstrations S. This task vector is then patched to the representation of x, which, until that point, was computed independently of both x' and S.\n\nWe hope this explanation clears up any confusion. Please feel free to ask if you have further questions or require additional details.\n\n**Q3: The higher suitability of intermediate layers and the consistency of this pattern.**\n\nThe pattern of intermediate layers being more suitable, as depicted in Figure 2, appears to be relatively consistent across the tasks we evaluated. Note that this is observable from the relatively low standard deviations shown in the figure, which were calculated across different tasks.\n\nThe exact reason for this pattern is not fully clear. It might be that the model requires initial layers to process the input (i.e. represent the examples and identify the rules), and then utilize intermediate layers to aggregate this understanding to create the task vector at an intermediate layer. Further layers could apply the identified rule and prepare the output. This possible interpretation is in line with prior work such as [3] and [4]. However, this is a hypothesis, and additional work is needed to analyze the entire mechanism.\n\n**Q4: A suggestion for strengthening the analysis by validating whether similar tasks have similar task vectors, using methods like t-SNE.**\n\nWe appreciate your recommendation to compare tasks in n-dimensional space using techniques like t-SNE. Following your suggestion, we conducted this analysis with t-SNE and found two reassuring phenomena: first, task vectors from the same task are closely clustered together, and second, similar tasks (from the same category) are indeed grouped together. Specifically, the inter-cluster standard deviation is 20 while the intra-cluster is ~1. These results are also consistent with the analysis of the histogram above (see our response regarding the variance of theta). Again, we are sincerely sorry that we cannot share the plot. Overall, this additional evidence supports our findings, and we will include it in the final version of the paper.\n\n**Regarding the missing reference:** Thanks, we will add this!\n\n**Regarding Reproducibility:** We wish to clarify that, upon acceptance, we plan to publicly release our code for generating the data for all tasks and for reproducing the experiments reported in the paper. Our code is in Python and relies on standard libraries and openly accessible models.\n\n[1] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. Wang, et al., ICLR 2023.\n\n[2] The Hydra Effect: Emergent Self-repair in Language Model Computations. McGrath et al., 2023.\n\n[3] The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives. Voita et al., EMNLP 2019.\n\n[4] Dissecting Recall of Factual Associations in Auto-Regressive Language Models. Geva et al., 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The empirical insights gained from this study could be valuable to other researchers in the field who are working to understand the phenomenon of in-context learning.", "reasons_to_reject": "- What’s the variance of \\theta(S) for different choices of x’? Is it very low? If that's not the case it will be hard to conclude that the transformer uses task vectors. This is something where the empirical evidence provided by this work is not satisfactory. \n- Also, the overall reasoning is somewhat weak. As even if the probe vectors show performance with a computation similar to the hypothesis class, it won't still be sufficient to imply that the hypothesis class is true of this particular type. Such empirical results in support of the claim are necessary. but not sufficient.", "questions_to_authors": "- The exact mathematical formulation of f(x, \\theta(S)) is unclear. Is it an inner product between two representations? \n- Coming to the usage of x’ instead of x to get the representation, the model has seen the query x’. Will this not tamper the Vector for S( i.e training data for the ICL) as x’ does not have a token succeeding “->” ?\n- Why do you think intermediate layers are more suitable as shown in the results in Figure 2? Also, is this pattern consistent across all subsets of tasks? \n- What’s the variance of \\thta(S) for different choices of x’? Is it very low? If that's not the case it will be hard to conclude that the transformer uses task vectors. \n- Are there any insights based on results for different types of tasks? \n- To elaborate, more experiments on validating whether tasks from a similar set  of tasks (Ex : En -> Fr, Fr -> En ) is nearer in n-d dimensional space rather than tasks that is dissimilar (Fruits -> Colour) would have made the analysis more stronger towards the positioning that ICL creates task vectors. This could have been validated with methods like t-sne etc.", "paper_topic_and_main_contributions": "This paper attempts to formalize the notion of the hypothesis class of functions for ICL. The authors propose a novel approach to build task vectors from the training set, which is then used to answer queries. The authors conducted experiments using large language models such as GPT-J, Llama, and Pythia to support their findings. The results of these experiments demonstrate that the proposed approach is practical for a wide range of tasks. This work provides one possible way to interpret how LLMs operate internally for performing ICL.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Potential missing reference: https://arxiv.org/abs/2205.05055."}, "extra_text": {}}, {"review_id": "tIxTmDuOif", "reviewer": "Reviewer_PY9i", "summary": "", "questions": "1. Regarding Figure 2, is there any reasoning on the behavior that layers of 20-25 led to much worse accuracy for LLaMA 7B or 13B?\n2. The recently work by Merullo et al. (2023) is closed related to this paper. Could the authors provide more insights on how these two views are under unified experimental scenarios (i.e. the same model and task)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful examination of our work and the positive response! We are encouraged by the recognition of our contribution, clarity, and solid experimentation.\n\nWe address the concerns under \"reasons to reject\":\n\n**(1) Regarding theoretical analyses underlying such interpretation of ICL.**\n\nWhile we agree that theoretical insights can provide a foundational understanding, we believe that empirical observations play a crucial role, particularly when exploring complex phenomena where theoretical models may be intractable or incomplete. Our empirical evidence not only sheds light on specific mechanisms but also paves the way for new theoretical directions. Specifically, we believe that the notion of a hypothesis class is key to any theoretical analysis of learning and in particular ICL. For example, previous theoretical approaches to ICL considered gradient descent as a possible mechanism for ICL (e.g. [1] and [2]). However, these works considered a synthetic setting where the hypothesis space was simple and clear. Here we provide an important step towards what could be a realistic hypothesis class. This can result in many exciting follow-ups that study both the algorithmic structure of ICL (i.e., how ICL finds a good element in the hypothesis class), as well as generalization bounds (that could, for example, rely on the low-dimensional representation of the task vectors).\n\n**(2) A request for more discussion on the connections between the proposed viewpoint and existing work (e.g. Merullo et al. (2023)).**\n\nThank you for this comment. Note that specifically Merullo et al., 2023 was published within less than 3 months from the EMNLP 2023 submission deadline, so it is considered a concurrent work. Nonetheless, we agree that an extended discussion will be helpful.\n\nWe propose a possible reconciliation of our work and Merullo et al., 2023. Considering the residual stream at the last position, the process might look as follows: first, the task vector is created from the demonstrations (a part which Merullo et al. don't discuss), then the query is integrated to the last position from which the prediction is obtained (as shown in their work), and finally, the mapping is applied (perhaps as they describe). Their discussion about the application of the mapping using an FFN mechanism might be seen as complementary to our work, as it provides an explanation for what we term f(x, θ). It's important to note, however, that they demonstrate this result only for GPT-2, which is a special case due to the weight tying between input and output embeddings. This may be why they observe a simple word2vec-like mechanism. Our results are more general as they hold across a wide range of models, some without such weight-tying. Their distinction between abstractive and extractive tasks is interesting but might not provide the complete picture, as we see our method works in the list first/last tasks, which are extractive tasks.\n\nWe will include this discussion in the camera-ready version.\n\nWe further answer the reviewer's questions:\n\n**Q1: Regarding the seeming qualitative difference between different models in Figure 2.**\n\nThanks for this question. Please note that this decline in accuracy in upper layers also occurs in LLaMA 30B (and other models, please see Figure 4 in the appendix). In Figure 2 we just cut the x-axis earlier, for readability, which prevents seeing this decrease for LLaMA 30B (which has 60 layers, compared to only 30-40 in the smaller models). But given your question we realize that this might be confusing. To address this, we will replace Figure 2 with a version where the x-axis is not cut, in which it is possible to see that the accuracy for LLaMA 30B shows a similar decrease in upper layers. E.g. at around layer 30, the accuracy of LLaMA 30B drops to a value of around 0.1.\n\nAlso, we will include another plot where the x-axis is normalized with respect to the total number of layers for each model – this plot (that we cannot share in this restricted format, we apologize again) shows that trends across models align more closely.\n\n**Q2: Regarding concurrent related work.**\n\nPlease refer to our response above to the \"reasons to reject\" (point 2).\n\n**Regarding typos:** We will fix this, thanks!\n\n[1] Transformers learn in-context by gradient descent. von Oswald et al., ICML 2023.\n\n[2] Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers. Dai et al., ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a novel viewpoint to understand the mechanism of ICL for LLMs. The two key components, compressing S into a single task vector, and using this task vector to modulate a transformer, are well presented and validated. The paper was clearly written; explained the intuition well and provided solid experiments to support the claims.", "reasons_to_reject": "I am looking for some theoretical analyses underlying such interpretation of ICL. Also, connections between the proposed viewpoint and other existing work (e.g. Merullo et al. (2023)) shall be more discussed.", "questions_to_authors": "1. Regarding Figure 2, is there any reasoning on the behavior that layers of 20-25 led to much worse accuracy for LLaMA 7B or 13B?\n2. The recently work by Merullo et al. (2023) is closed related to this paper. Could the authors provide more insights on how these two views are under unified experimental scenarios (i.e. the same model and task)?", "paper_topic_and_main_contributions": "The paper provides a new perspective of In-Context Learning (ICL) mechanisms for LLMs. Particularly, the authors claim that ICL is able to compress a given training set into a single task vector, and it guides the transformer to generate outputs given some query. This work propose a view of hypothesis space that ICLs learn with and show the validity on LLMs. The paper supports this claim using adequate experiments across several LLM and multiple tasks. Their work might contribute to the community with a better understanding of how LLMs perform ICL.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Minor: Line 45-46 \"then the class f would be linear classifiers...\" a linear classifier is just an example and apparently it is not necessarily the case. Perhaps change \"would\" to \"could\"?"}, "extra_text": {}}, {"review_id": "LB6wLWQXL3", "reviewer": "Reviewer_qoQT", "summary": "", "questions": "(1) How do interpret the performance degradation in the experiments?\n(2) attention blocks unavoidably access the inputs x in the real inference, how to interpret this with the hypothesis of task vector, which is claimed independent of x? ", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoroughly reading our work and for the great questions!\n\nWe address the concerns under \"reasons to reject\":\n\n**(1) Regarding the simplicity and apparently limited novelty of our experiments.**\n\nWhile the idea and experiments may appear simple, we believe that the simplicity can be seen as a strength, fostering accessibility and further exploration within the community. More importantly, our observation, though simple to understand in retrospect, has not been previously recognized, and we consider it to be far from trivial. The model's internal computation could have been implemented in many ways, and the emergence and utilization of these intermediate task vectors is just one possible (relatively sparse) solution among many. Our findings introduce new avenues for research that focus on both the efficiency and understanding of ICL. To highlight the broader implications and emphasize the novelty, we will include a detailed discussion in the camera-ready version.\n\n**(2) On the slightly lower performance of our method compared to ICL.**\n\nYou are correct in noticing the slightly degraded performance with our task vector method compared to ICL. While this discrepancy exists, it doesn't undermine our main claims. Rather, it suggests that there may be additional mechanisms at play or that our method might not be perfectly capturing every aspect. If enhancing performance is a priority, future work could build on our approach, possibly by averaging over more demonstrations. Our main contribution lies in the insights provided, not necessarily in achieving optimal performance.\n\n**(3) A request for more insights into the attention mechanism behavior in the ICL setting.**\n\nWhile we acknowledge the potential value of a deeper analysis of the attention mechanism behavior in the ICL setting, it's important to note that this investigation requires substantial effort and could constitute a separate study. We believe this line of inquiry, though intriguing, is outside the scope of our current work, and we plan to consider it for future research. Thank you for pointing out this interesting direction!\n\nWe further answer the reviewer's questions:\n\n**Q1: On the slightly lower performance of our method compared to ICL.**\n\nPlease refer to our response above to the \"reasons to reject\" (point 2).\n\n**Q2: Regarding the access to the query x during regular inference and how it relates to our claim that the task vector is independent of the query.**\n\nThank you for raising this important question, as this is a central point in our work. The idea is that when creating a task vector, the query can be any valid input for the task, and not necessarily the query that the model will provide the output for. This is demonstrated by using a different x' to create a task vector and then using this task vector to get an output for x. That is, while initial attention layers indeed access the query during real inference, the created task vector is still general to the task and can work with any query. This mechanism lets the model understand the \"type\" of the input query rather than the specific query.\n\n**Regarding Reproducibility:** We wish to clarify that, upon acceptance, we plan to publicly release our code for generating the data for all tasks and for reproducing the experiments reported in the paper. Our code is in Python and relies on standard libraries and openly accessible models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "it sheds some light on the working mechanism of in-context learning observed in LLMs. The numerical experiments gave a fairly well approximation.", "reasons_to_reject": "The overall idea and experiments are quite simple, and the total contribution seems insufficient with limited knowledge gain and novelty for the community.  The numerical approximation gives a bit degraded performance compared with the ICL. More deep and technical insights would be helpful such as the analysis of attention mechanism behavior in the ICL setting. \n", "questions_to_authors": "(1) How do interpret the performance degradation in the experiments?\n(2) attention blocks unavoidably access the inputs x in the real inference, how to interpret this with the hypothesis of task vector, which is claimed independent of x? ", "paper_topic_and_main_contributions": "The paper interpreted the so-called in-context learning(ICL) as a mechanism which first compresses the training samples to a task vector and then the transformer guided by the task vector would act in the input variables. It gave extensive experiments to illustrate the idea hypothesis. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper delves into the mechanism of in-context learning (ICL) within LLMs and proposes a novel approach that involves compressing training samples into a task vector and using it to modulate a transformer for handling queries. Extensive experiments using various LLMs support this concept.

The reviewers collectively acknowledge the empirical insights gained from the study and agree on the potential value of this work for researchers seeking to understand ICL. Given the average scores being above the acceptance borderline and considering the valuable insights provided by the paper, the decision should lean towards acceptance. However, the reviewers highlight the need for further theoretical analyses, deeper insights, missing references and some presentation issues. It's recommended to encourage the authors to address some of these concerns in a revision as promised in the discussion, to strengthen the overall impact and quality of the paper.
Decision: Accept-Findings