Function Vectors in Large Language Models
Authors: Eric Todd
Millicent Li
Arnab Sen Sharma
Aaron Mueller
Byron C Wallace
David Bau
Venue: ICLR 2024 poster (2024)
Keywords: In-Context Learning
Interpretability
Primary area: visualization or interpretation of learned representations

Abstract: We report the presence of a simple neural mechanism that represents an input-output function as a vector within autoregressive transformer language models (LMs). Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV).  FVs are robust to changes in context, i.e., they trigger execution of the task on inputs such as zero-shot and natural text settings that do not resemble the ICL contexts from which they are collected. We test FVs across a range of tasks, models, and layers and find strong causal effects across settings in middle layers. We investigate the internal structure of FVs and find while that they often contain information that encodes the output space of the function, this information alone is not sufficient to reconstruct an FV. Finally, we test semantic vector composition in FVs, and find that to some extent they can be summed to create vectors that trigger new complex tasks. Our findings show that compact, causal internal vector representations of function abstractions can be explicitly extracted from LLMs.

Published as a conference paper at ICLR 2024
FUNCTION VECTORS IN LARGE LANGUAGE MODELS
Eric Todd; Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau Khoury College of Computer Sciences, Northeastern University
ABSTRACT
We report the presence of simple neural mechanism that represents an input- output function as a vector within autoregressive transformer language models (LMs). Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which call function vector (FV). FVs are robust to changes in context, i.e., they trigger execution of the task on inputs such zero-shot and natural text settings that do resemble the ICL contexts from which they are collected. We test FVs across a range of tasks, models, and layers and find strong causal effects across settings middle layers. We investigate the internal structure of FVs and find while that they often contain information that encodes the output space of the function, this information alone is not sufficient reconstruct an FV. Finally, we test semantic vector composition in FVs, and find that to some extent they can be summed to create vectors that trigger new complex tasks. Our findings show that compact, causal internal vector representations of function abstractions can explicitly extracted from LLMs.
INTRODUCTION
Since the study of the lambda calculus (Church, 1936), computer scientists have understood that the ability for program to carry references to its own functions is a powerful idiom. Function references can be helpful in many settings, allowing expression of complex control flow through deferred invocations (Sussman, 1975), and enabling flexible mappings from inputs to target task. In this paper we report evidence that autoregressive transformers trained on large corpora of natural text develop a rudimentary form of function references.
Our results begin with an examination of in-context learning (ICL; Brown et al., 2020). ICL mechanisms have previously been studied from the perspective of making copies (Olsson al., 2022) and from a theoretical viewpoint (Von Oswald et al., 2023; Garg et al., 2022; Dai et al., 2023), but the computations done by large models generalize and execute complex ICL functions are not yet fully understood. We characterize a key mechanism of ICL execution: function vectors (FVs), which are compact vector representations of input-output tasks that can be found within the transformer hidden states during ICL. An FV does not directly perform a task, but rather it triggers the execution of specific procedure by the language model (Figure 1).
(a)
(c)
92 = Antonym FV
arrive:departs emall:big, common:
zaze
The word "fast" means slow
(b)
(d)
u. Eng-Spanish FV
O h Hidden-state intervention
aanunt:cantiddd, win:ganar,
word
"fast"
rápido
Demonstrations ((x.. (12.)))) Query Prediction Yie
Novel zero-shot prompt
Execution FV
Figure 1: An overview of function vectors (FVs). An FV is extracted from activations induced by in-context examples of (a) antonym generation (b) English to Spanish translation, and then inserted into an unrelated context to induce generation (c) a new antonym or (d) translation.
Correspondence to nad..cr@northesstem...tt Open-source code and data available at functions canctions.baulab... info..

---

Published as a conference paper at ICLR 2024
Function vectors arise naturally when applying causal mediation analysis (Pearl, 2001: Vig et al., 2020; Meng et al., 2022; 2023; Wang et al., 2022a) to identify the flow of information during ICL. We describe an activation patching procedure to determine the presence of handful of attention heads that mediate many ICI. tasks. These heads work together to transport a function vector that describes the task; the FV can be formed by summing outputs of the causal attention heads.
We test the hypothesis that function vectors are a general mechanism spanning many types of functions. To quantify the role and efficacy of function vectors, curate a data of over 40 diverse ICL tasks of varying complexity. We calculate FVs for these tasks and investigate impact of FVs in triggering those functions across a variety of LMs scaling up from 6B 70B parameters.
We further ask whether FVs are portable: are the effects of an FV limited to contexts very similar to those where it extracted, or can an FV apply in diverse settings? We compare the effects of FVs when inserted into diverse input contexts including differently-formattee and natural text contexts. We find that FVs are remarkably robust, typically triggering function execution even in contexts that bear no resemblance to the original ICL context.
forms, zero-shot formats,
key question is whether the action of FVss can be explained by word-embedding vector arith- metic (Mikolov et al., 2013; Levy & Goldberg, 2014; Merullo et al., 2023). We examine decodings of FVs (Nostalgebraist, 2020), and find that although FVs often encode a function's output vocabulary, those vocabularies do not fully identify FV. In other words, to invoke functions, FVs need to carry some additional information beyond their encoding of the top vocabulary words.
Finally, we investigate whether the space of FVs has its own vector algebra over functions rather than words. We construct a set of composable ICL tasks, and we test the ability of FVs to obey vector algebra compositions. Our findings reveal that, to some extent, vector compositions of FVs produce new FVs that can execute complex tasks that combine constituent tasks. We emphasize that FV vector algebra is distinet from semantic vector algebra over word embeddings: for example, composed FV vectors can specify nonlinear tasks such as calculating the antonym of a word, that cannot themselves be implemented as a simple embedding-vectt offset (Appendices A, L).
2 METHOD
2.1 A MOTIVATING OBSERVATION
When a transformer processes an ICL prompt with exemplars demonstrating task t, do any hidden states encode the task itself? We seek causal features rather than just correlations.
We can investigate this question with the following simple test: Gather a set of ICL, prompts P. for the task t and compute the average activation hy at the last token of each prompt at particular layer of the model (Figure 2a). Then perform an intervention where h is added to the representation after l when the transformer completes previously unseen zero-shot prompt (Figure 2b).
Surprisingly, we find that adding the average activations in this way at particular layers induces the model to perform the task in the new context. For example, if t antonym, the red line in Figure 2c shows that adding hí2 at layer 12 in GPT-J causes the model to produce antonyms in a zero-shot context, with 24.3% accuracy. That suggests that hf2 does encode the antonym task.
The effect of hy leads us to ask: Can we distill a more effective hidden-state representation of the task t? In the rest of Section 2 we describe analysis of the mechanisms of ICL that leads function vector representation whose stronger causal effects are shown as a green line in Figure 2c.
(a) Average Layer Activation
(b) Zero-Shot Intervention
(c)
Zero-Shot Intervention
0.35
old oldayoung, vanishrappear, dazkgj
8 simplef +
comples
Accuncy
0.5
FV h
awakerasleep, future:past, foym 5
top:bottom, tall:short, accept
encodeg + 6
decode
Layer
10
20
A motivating observation: (a) an average activation is computed over a set of antonym ICL prompts, and (b) added a zcro-shot context, which produces the opposite of unseen words. (c) Systematic effects (in red) adding hi middle layers of the network; even stronger effects are seen by the FV (in green).
Figure 2:
2

---

Published as a conference paper at ICLR 2024
2.2 FORMULATION
An autoregressive transformer language model f takes an input prompt p and outputs a next-token distribution f(p) over vocabulary V; we write [p)]yv] E [0, 1] for the predicted probability of output y E V in response to input Internally, / comprises I. layers, we examine their calculations at the last token position. Each layer & L has a vector representation of the last token, he E R that is computed from the previous layer he h + my + EjeJ agj. where my is the output of a multilayer perceptron, and aıj is the projection of the output of the jth attention head (out of J heads) into the hidden state at layer É. This definition of at, E R adopts the framing of Elhage et al. (2021) rather than that of Vaswani et al. (2017) (see Appendix B for details). Attention heads and hidden states can be viewed as functions of transformer input, so we shall write aez (p) or he(p) to denote their values when the transformer processes input p. The transformer's decoder T maps the last layer hidden state to the output distribution (hr (p)) = fp).
For each task T in our universe of ICL tasks T we have data set of in-oontext prompts pt E Pa. Each prompt P is a sequence of tokens with N input-output exemplar pairs (r, y) that demonstrate the same underlying task mapping between x and y, and one query input Zig corresponding to a target (correct) response Vig that is not part of the prompt, that should be predicted by the LM if it generalizes correctly. We focus our analysis on successful ICL by including P. only prompts pf where the prediction (p() ranks the correct answer Vig highest. We write one ICL. prompt as
pt = [(=.).)),. .. (JIN, ).
We also make use of uninformative ICL prompts 5 E P. for which the labels are shuffled; we use the tilde to indicate a shuffled prompt Pi = [ Is1), (TaN, DiN), Figg in which there is no systematic relationship between any of the Tigk and Vsk-
2.3 CAUSAL MEDIATION EXTRACT FUNCTION VECTORS FROM ATTENTION HEADS
To distill the information flow during ICL, we apply causal mediation analysis.
Given a transformer model f and an ICL prompt p E Pt from a dataset representing task t, we prompt the model with only input-output pairs (æ;, y:). Therefore, the LM must infer the implicit relationship between these (x, y) pairs to correctly predict the answer given a novel query Tig. We seek to identify model components with causal role in the prediction of Yig. We restrict to the attention heads since those are the components used by transformer LMs to move information between different token positions (Vaswani et al., 2017; Elhage et al., 2021). Formally, for each attention head az and task dataset Pt. we take the mean of task-conditioned activations at as
analysis
d's
Σ P.EP
azg (vi).
(2)
P
We then the model on an uninformative ICI. prompt ø P. where each is matched with random output / [x.,, 3.)] Now, the model is less likely to generate the correct output By as cannot infer the relationship from incorrect ICL examples (notwithstanding the observation from Min et al. (2022) that some tasks can be guessed from incorrect labels). While running the model on pt we replace an attention head activation aęj with mean task consicondt activation ai (Eq. 2) and measure its causal indirect effect (CIE) towards recovering the correct answer Dy
CLE(agy
p = f(pt agj := ( f(p))u))
The intuition here is to measure the degree to which using the "correct" mean attention head output fty-compted over the uncorrupted prompts for task i-increases the mass assigned to the target response Vig, relative to the likelihood of this token under the corrupted prompt P/. A larger value implies that the corresponding head is more influential in promoting the correct response.
Then each attention head's average indirect effect (AIE) is calculated by averaging this difference across all tasks E 'T and (corrupted) prompts:
AlE(ar)) = IT
CIE(ar (
(4)
IET
Men.
3

---

Published as a conference paper at ICLR 2024
Antonym
(a)
0.05 (b)
Attention Weight
Head Index
5
-0
English-Frened
15
15
AIE
eges
oeufs
sad
triste
suga
suerc
goods
bon
loan
prêt
read
Layer
Figure 3: (a) Average indirect effect across all tasks T for each attention head GPT-J, and (b) the top 10 heads' weights individual tokens for one example prompt p/. The most strongly implicated heads appear in middle layers. Attention weights are strongest on the output tokens each exemplar.
To identify the set of attention heads with the strongest causal effects, we repeat this process for each attention head azj in f. for all layers L, and all head indices j. We gather the attention heads with highest AIE over all layers as the set A.'
Figure 3a shows the AIE per attention head in GPT-J over many tasks (see Appendix G for larger models). The 10 attention heads with highest AIE (which make up A) are highlighted pink (square outlines) and are clustered primarily in early-middle layers of the network. The average attention pattern of these heads at the final token is shown for two tasks in Figure 3b. These heads primarily attend token positions corresponding to example outputs; this observation is consistent with the high salience of ICI. label tokens observed by Wang et al. (2023a) and while this resembles the same attention pattern "induction heads" (Elhage al., 2021; Olsson et al., 2022) not all heads in A reproduce this pattern on other contexts with repeated tokens (Appendix H).
Due to their high causal influence across many tasks (see Appendix G for breakouts by task), hypothesize that this small set of heads is responsible for transporting information identifying the demonstrated ICL task. We can represent the contribution of A as single vector by taking the sum of their average outputs, over task, which we call function vector (FV) for task t:
Σ
d
AGEA
We then test the causal effect of FV by adding it to hidden states any layer as the model resolves prompt and measuring its performance in executing the task (Appendix B).
EXPERIMENTS
Models. We deploy a series of decoder-only autoregressive language models; each is listed and described in Table 1. We use ouggingface ace implementations (Wolf al., 2020) of each model.
Tasks. We construct diverse array of over 40 relatively simple tasks to test whether function vectors can be extracted in diverse settings. To simplify the presentation of our analysis, we focus on representative sample of 6 tasks:
Antonym. Given an input word, generate the word with opposite meaning.
Capitalize. Given an input word, generate the same word with a capital first letter.
Country-Capital Given a country name, generate the capital city.
English-Frie French. Given an English word, generate the French translation of the word.
Present-Pase Given verb present tense, generate the verb's simple past inflection. Singular-Plural. Given singular noun, generate its plural inflection.
All other tasks are described in Appendix E.
For GPT-J, we use |A| 10 attention heads. For larger models, scale the number of attention heads we approximately proportionally to the number of attention heads in the model. (We use 20 heads for Ilama (7B), 50 for Ilama (13B) & GPT-NeoX, and 100 for Ilama 2 (70B).)
4

---

Published as a conference paper at ICLR 2024
Table 1: Models used in this study. We focus on decoder-only autoregressive language models that are capable of ICL. For each model, we present the number of parameters, the number of layers [ and number of attention heads per layer lael.
Model
Huggingface ID
Citation
Parameters
Training Tokens
[L]
|as|
GPT-
EleutherAT/got-J-e
(Wang & Komatsuzaki, 2021)
6B
4028
28
16
GPT-NeoX
FleutherAT/got-neen 20b
(Black et al., 2022)
20B
472B
44
64
Llama
meta-1LaamaLLLama222773
(Touvron et al., 2023)
7B
2T
32
32
Llama
meta-1lana/LLama-22133--
(Touvron al., 2023)
13B
2T
40
40
Lluma meta-1llame/LLaaa-22788-8-
(Touvron al., 2023)
70B
2T
80
64
Table 2: Average accuracy across tasks amacro-averaged across random seeds) for both shuffled-
label and zero-shot contexts- adding the FV increases performance of the task compared to the base model in both contexts. For GPT-J we compare to layer averages (Section 2.1) and find that our FV works best. We also report results for both settings on an additional 34 tasks for GPT-J+FV and Llama 2 (FOB)+FV. More details on additional tasks in Appendix E.3.
Shufeed-Label (2x1 Ba). (INN, MiN), 2ig]
Zero-Shot [Zig]
GPT-J (baseline on uninformative input) hij
39.1 + 1.2%
5.5 + 0.8%
Layer average (Section 2.1)
79.5 = 3.1%
9.5 + 1.8%
Vi
FV (Eq. 5)
90.8 0.9%
57.5 + 1.7%
GPT-NeoX (baseline on uninformative input)
32.5 + 1.3%
6.7 + 0.1%
Uu FV
90.7 0.6%
57.1 1 1.5%
Llama (70B) (baselinc on uninformative input)
52.3 2.2%
8.2 + 0.7%
01 FV
96.5 0.5%
83.8 1 0.7%
GPT-J FV on 34 additional tasks
80.4 = 0.6%
46.1 + 3.7% 74.2 + 3.1%
Llama (70B) u FV on 34 additional tasks
93.0 + 0.5%
3.1 PORTABILITY OF FUNCTION VECTORS
In this section, we investigate the portability of function vectors-i.e., the degree which adding an FV to particular layer at the final token position of the prompt can cause the language model to perform a task contexts that differ from the ICL contexts from which it was extracted. For simplicity of analysis, we only include test queries for which the LM answers correctly given 10-shot ICL prompt; all accuracies and standard deviations over random seeds are reported on this filtered subset, and can be thought of the proportion of model's task performance encoded by FVs. Results when incorrect ICL are included are similar (see Appendix D).
Evaluating FV's at I. Layer /3/33. In Table 2 we report results (averaged across the tasks mentioned above) for adding FVs to shuffled-label ICL prompts and zero-shot contexts across 3 models- GPT-J, GPT-NeoX and Llama 2 (70B), at layers 9, 15, and 26 respectively (approximately /L/3)). For GPT-J, we also compare the efficacy of FVs other approaches for extracting task-inducing vectors including simple state averaging ($2.1).
Our first observation is that the base model is substantially unable to perform the tasks in the uninformative shuffled-label ICL and zero-shot settings; however, adding the FV allows the model to recover task performance significantly both cases. We also observe the proposed approach for constructing FVs via causal mediation outperforms the layer-averaging h approach in both contexts.
Zero-Shot Results Across Layers. Figure 4 shows results across layers for the zero-shot case. The sharp reduction of causal effects in late layers suggests that FVs do not simply act linearly, but that they trigger late-layer nonlinear computations. This pattern of causality is seen across variety of tasks, autoregressive model architectures, and model sizes. Even in cases where performance is low, as in English-Fremn Frenchh with GPT-NeoX and Llama 2 (70B), adding the function vector in middle layers still results in large relative improvements to accuracy over the zero-shot baseline. Results are also consistent across model sizes: see Appendix J for results with all sizes of Llama 2.
5

---

Published as a conference paper at ICLR 2024
Antonym
Capitalize
Country-Capital
English-Prench
Present-Past
Singular-Plure
GPT
0.5
20
10
15
1015
2025
15
5 1013
5 10 20
Accuracy
GPIcNooX
162432
1624
1624
16243240
1624
16243220
Liana 2 (708)
0.5
16 32 48 16 32
16 32
48 Layer
16 48 Layer
16 32 48 64
Layer
Layer
Layer
Layer
Figure 4: Task accuracy across tasks and models, applying FVs in zero-shot settings. We show accuracies before adding the function vector (dotted lines) and after adding the FV to specific layer (solid lines). Adding the FV to early-middle layers pushes models to perform the target task without any exemplars, as demonstrated by accuracy increases over the zero-shot without FVs.
Table 3: Natural text portability of the Antonym FV. We provide a natural template and substitute in a query word for 'x' Then, measure accuracy based on whether the correct antonym is produced in this natural text setting within generated tokens.
Prompt
GPT-J
+Antonym FV
The word "x", means
1.5 1.1%
55.2 : 3.8%
When I think of the word "x", usually means
0.3 + 0.2%
67.7 3.0%
When think of x, I usually
+ 0.0%
61.1 2.4%
While reading a book, came across the word "x"". looked it up in dictionary and it turns out that means The word can be understood as a synonym for
2.7 + 1.9%
46.0 + 4.6%
2.4 1.7%
52.7 + 11.0%
FVs are Robust to Input Forms. To check whether the FV is dependent on the ICL template that it is extracted from, we also test the FV on 20 additional ICL templates (Appendix C) and in natural text settings, adding the FV at layer f. = 9 for GPT-J (approximately T.I/3).
We create 20 different ICL templates that vary the form of the ICL prompt across prefixes and delimiters of input-output pairs. We evaluate FVs on GPT-J for these 20 templates in both shuffled- label and zero-shot settings. Across our 6 tasks, adding the FV executes the task with an average accuracy of 76.2 + 13.8% with shuffled labels and 40.0 + 16.7% in the zero-shot setting, while GPT-J settings, respectively. Despite higher variance, this performance is similar to performance in the same settings with the original template.
only scores 32.3 + 12.8% and 6.2 + 4.3% on the
We also evaluate FVs on natural text completions. Given a natural text template, insert a test query word and have the model generate n tokens. We add the FV to the final token of the original prompt, and for all subsequent token predictions to guide generation. We use simple regex match to compute whether the generation includes the correct target for the inserted query word.
Table 3 shows natural text portability results for the antonym FV for GPT-J, generating new tokens. In each of the templates, the antonym is in the FV completion significantly more than the original completion. In fact, we find that the efficacy of the antonym FV eliciting the correct response in these natural text templates performs setting. This true for all tasks (Appendix F), suggesting that the task representation transported during ICL is similar one that is used during autoregressive prediction in natural text settings.
par with the results previously reported for the zero-shot
We include a few qualitative results for the English-Frent French and Country-Capital tasks (Table 4). We that the English- Frenchh FV will sometimes translate the whole sentence after giving the proper completion to the original one-word translation task, indicating that it has captured more than the original task it shown. Additional natural text portability results are included in Appendix F.

---

Published as a conference paper at ICLR 2024
Table 4: Qualitative examples of natural text completions for English-French. and Country-Capital
English-French
Prompt:
The word "daily" means
The word 'link' can he understood synonym "connection" 'relation'. The term 'link' is used in...
GPT-J
every day
English--eeh FV
tous les jours
'lien,, le mot 'lien' pout être compris comme synonyme...
Country-Cupina
Prompt:
When you think of Netherlands.
GPT-J
you probably think of tulips. windmills, and cheese. But the Netherlands is also home to... you think of Amsterdam. But there are many other cities in the Netherlands. Here are some...
Cappplaauntryy FV
Table 5:
A direct decoding of the function vector for each task.
Task E
Tokens in the distribution (vv)) in order of decreasing probability
Antonym Capitalize
lesser', counterpart', 'wrong', negate', destroy' Vanilla', Copy', Adapter', AAtor', Container' Moscow", Bangkok', Paris', 'Londn',, Madrid akī', . masc', cyl', embr', è' 'received', 'changed', 'killed', 'answered', Changed 'cards', 'stocks', helmets', items', phones'
Country-Capital
English-Fremn
Present-Past
Singular-Plun
3.2 THE DECODED VOCABULARY OF FUNCTION VECTORS
Several studies have gleaned insights about the states and parameters of transformers by viewing them in terms of their decoded vocabulary tokens (Nostalgebraist, 2020; Geva et al., 2021; 2022; Dar et al., 2023; Belrose et al., 2023). Therefore we ask: can we understand directly to a token probability distribution? Results are shown in Table 5, which lists the top five tokens in the decoded distribution D Dvv)) for each task (additional tasks in Appendix I).
FV by decoding ot
clear pattern emerges: for most tasks, the decoded tokens lie within the task's output space. The Singular- Plural function vector decodes to a distribution of plural nouns, and Present- passe decodes to past-tense verbs. However, that is not the case for all tasks: English-French decodes to nonsense tokens, and the Antonym task decodes to words that evoke the abstract idea of reversal.
Given these meaningful decodings, we then ask whether the token vocabulary is sufficient to recreate working function vector. That is, begin with the token distribution Q: D(v.), and determine whether a function vector can be reconstructed if we know the top words in Q.. Denote by Qιx the distribution that resamples Qe while restricting to only the top i words. We perform an optimization to reconstruct a tω that matches the distribution Quk when decoded (where CE is cross-entropy loss):
U = arg min CE(QuA: D(v))
(6)
In Table 6, the performance of He is evaluated when used function vector. We find that, while it is possible to partially recreate the functionality of an FV, good performance typically requires more
Table 6: Performance of FV 0g is compared to the reconstruction é; 100 that matches the top 100 tokens, and 0 that matches all 50k tokens in D(v)). The KL divergence between the D (0x) and Qtk are shown for each reconstruction as KI.,, Lowest performers for each task red.
Accuracy on zero-shot prompts
Task &
Úz 100
KL100
Grall
KL
Antonym
48.2 + 2.0%
4.8 4 2.0%
0.0033
39.6 + 2.6%
0.0137
Capitalize
70.5 + 2.1%
5.7 + 2.2%
0.0001
51.5 4 11.6%
0.0053
Country-Capita
83.2 + 2.7%
58.1 + 18.5%
0.0002
29.0 15.1%
0.0019
English-Frema
44.7 1.2%
4.8 : 1.7%
42.0 5.6%
0.0056
Present-Past
19.7
4.4 1.4%
0.0052
6.8 2.6%
0.0139
Singular- Plural
47.0 3.4%
23.3 1 6.1%
0.0
27.4
0.0145
7

---

Published as a conference paper at ICLR 2024
than 100 vocabulary tokens. In other words, knowledge of the top decoded tokens of ( is usually not enough on its own to construct a working function vector. That suggests that the FV contains some needed information beyond that expressed by its top decoded tokens.
3.3 VECTOR ALGEBBA ON FUNCTION VECTORS
(a) Input: "Italy, Russia, China, Japan, France"
(b)
Ya
FV
Task
Expected Output
Van
'Ac
First-Copy
Italy
First-Capital
Rome
Last-Copy
France
VB
Last-Capital
Paris
8D
Yac
Figure 5: (a) A set of three list-oriented tasks that can be composed to fourth task using FV vector algebra. (b) The parallelogram arrangement of the fourth vector VDD when it is composed out of the other three FVs.
Although Table 6 suggests that function vectors cannot be understood simple semantic vector offsets on word embeddings, we can ask whether function vectors obey semantic vector algebra over the more abstract space of functional behavior by testing the composition of simple functions into more complex ones. We begin with three conceptually decomposable ICL tasks; the list-oriented tasks First-Copy, First-Capital, and Last-Copy, as illustrated Figure 5a. Using ICL, we collect FVs for all three tasks and denote them VAC, UBC, and VAD-
Then we form a simple algebraic sum to create new vector that we will denote VBD-
VAC Last-Capital" Last-Copp + First-Capital First-Copy
BB WAD + UBC
(8)
In principle we could expect vBD to serve as a new function vector for new composed task (Last- Capital). We perform several similar task compositions on a variety of tasks. each case, we combine task with First-Copy and Last-Copy produce composed Last-* vector, then, we test the accuracy of enD as a function vector. We compare to the accuracy of the FV extracted from ICL, well accuracy of the same model performing the task using ICL. Results for GPT-J are reported in Table 7; see Appendix K for results for Llama 2 (13 and billion parameter models). We find that some FV's can be composed, with algebraic compositions outperforming FVs and even ICL on some tasks. Other tasks, including some for which ICL and FVs perform well, resist vector composition.
The ability compose the tasks that we have demonstrated may hinge on the fact that "word-selection" from context and wword-transformation are different components of language tasks that could involve FVs triggering complementary underlying mechanisms (e.g., one for locating and extracting input and another for transforming it). We therefore believe that FV composition may be useful tool for further understanding the mechanisms of LMs.
Table 7: The accuracy of ICL, calculated FV zero-shot interventions, and vector-composee vpp zero-shot interventions when performing several list-oriented tasks. Unlike previous evaluations, here we measure performance on all available samples of the task, without restriction to the subset where the L.M predicts correct. output. few cases, composed function vector intervention VED can perform a task hetter than ICI.
Task
ICL (ten-shot)
UBD (FV zero-shot)
vip (sum zero-shot)
Last-Antonym
0.25 + 0.02
0.02 + 0.01
0.07 + 0.02
Last-Capitalize
0.91 0.02
0.61 + 0.03
0.76 0.04
Last- Mast-Country-ee Capital
0.32 0.02
0.15 0.03
0.60 =-t 0.02
Last- Eaae--nngllheeeeee French
0.45 = 0.04
0.16 - 0.02 0.02
0.06 0.02
Bast-Present-De Past
0.89 = 0.02
0.18 - 0.02
0.29 + 0.03
I Bast-Singular-Plun
Plural
0.90 0.01
0.28 0.01
0.29 + 0.02
I Last- waw.ttaaaieeeiiss--tia.e
0.75 0.01
0.76 0.02
0.95 + 0.00
Last- Product- Product-Company
0.35 0.03
0.30 + 0.02
0.41 + 0.03
8

---

Published as a conference paper at ICLR 2024
4 RELATED WORK
cousin to function vectors has been independently observed in concurrent work by Hendel et al. (2023); they study causal effects of h (similar to Section 2.1) on a different set of models and tasks.
Task Representations. Our work shows that it is possible to extract FVs with strong causal effects from LLMs; this is advance over previous examinations that have added task representations to LLMs, e.g. Lampinen& & McClelland (2020); Shao et al. (2023); Mu et al. (2023); Panigrahi et al. (2023); Ilharco et al. (2023), who devised ways to create compositional task encodings for LLMs using metamappings, codebooks, soft poft-prompts or sets of model parameter perturbations that Ilharco et al. calls task vectors. Unlike these previous works that create function representations, compact FVs already exist within LLMs and show how to extract them. Likewise Lake & Baroni (2018); Hill al. (2018) show RNN hidden states cluster on similar tasks. Our work differs because FVs are causal, not just correlative, so they can be explicitly extracted and inserted.
find that
In-Context Learning. Since its observation in LLMs by Brown et al. (2020), ICL has been studied intensively from many perspectives. The role of ICL prompt forms has been studied by Reynolds & McDonell (2021); Min et al. (2022); Yoo et al. (2022). Models of inference-time metalearning that could explain ICL have been proposed by Akyürek et al. (2022); Dai et al. (2023); Von Oswald et al. (2023); Li et al. (2023b); Garg et al. (2022). Analyses of ICL as Bayesian task inference have been performed by Xie et al. (2021): 1): Wang et al. (2023c); Wies et al. (2023); Hahn & Goyal (2023); Zhang et al. (2023); Han et al. (2023). And ICL robustness under scaling has been studied by Wei al. (2023); Wang et al. (2023b); Pan et al. (2023). Our work differs from those studies of the externally observable behavior of ICL by instead focusing on mechanisms within transformers.
Mechanisms of task performance in LMs. Our work is related to Merullo et al. (2023); Halawi ct al. (2023) which analyze components during execution of ICL tasks and identify causes of false statements. Also related are several methods that modify activations at inference time to steer LM behavior (Li et al., 2023a; Hernandez et al., 2023a; Subramani et al., 2022; Turner et al., 2023; Rimsky al., 2023; Liu et al., 2023; Zou et al., 2023). Our work is consistent with Wang al. (2023a) which observes salience of label tokens during ICL, Wang et al. (2022b) which observes individual neurons that correlate with specific task performance, and Variengien & Winsor (2023) which task requests are processed in middle layers. We measure causal mediators across a distribution of different tasks mechanism that identifies and distinguishes between tasks.
to find generic
Mechanistic Interpretability. We also build upon the analyses of Elhage et al. (2021) and Olsson al. (2022), who observed prevalent in-context copying behavior related to jumps in performance during training. We isolate FVs using causal mediation analysis methods developed in Pearl (2001):
Vig et al. (2020); Meng et al. (2022); Wang al. (2022a); Geva et al. (2023). Our examination of FVs in vocabulary uses the logit lens of Nostalgebraist (2020); Geva et al. (2021); Dar et al. (2023).
Analyzing the Attention Mechanism. Our work is related to previous inenttion-weigh analy- ses (Voita et al., 2018; Clark et al., 2019; Voita et al., 2019; Kovaleva et al., 2019; Reif et al., 2019; Lin et al., 2019; Htut et al., 2019; Kobayashi et al., 2020), that have found attention weights that align with linguistic structures. Our work is motivated by the observation that attention weights alone do not fully explain model outputs (Jain& & Wallace, 2019; Wiegreffe & Pinter, 2019; Bibal understanding of attention by investigating the
et al., 2022). The focus of our paper is to extend content of the information transported by the attention heads in ICL to open a new window into the human-interpretable role that attention plays in language processing.
DISCUSSION
Function vectors are a surprising finding. The metalcarning capabilities of LLMs that have been studied since Brown et al. (2020) seem complex enough be inserutable. Yet in this paper we have found simple mechanism in a range of transformer LLMs that is common across tasks and robust to shifts in context: function vectors (FVs) that represent the task within a hidden state. FVs can be explicitly extracted from a small fixed set of attention heads that can be easily identified, and these FVs represent a range of tasks just as simply as word vectors (Mikolov et al., 2013) yet our findings also reveal FVs must be distinct phenomenon (Appendix A). Although FVs are not yet complete accounting of how ICL works, they do provide new clarity on one level of mediation within ICL, and they open up new path for future research to fully characterize function execution within L.LMs.

---

Published as a conference paper at ICLR 2024
ETHICS
While work clarifying the mechanisms of function representation and execution within large models is intended to help make large language models more transparent and easier to audit, under- stand, and control, we caution that such transparency may also enable bad actors abuse large neural language systems, for example by injecting amplifying functions that cause undesirable behavior.
ACKNOWLEDGMENTS
Special thanks to Evan Hernandez whose valuable advice and mentorship made this research possible. We are grateful for the generous support of Open Philanthropy (ET, AS, AM, DB) as well as National Science Foundation (NSF) grant 1901 117 (ET, ML, BW). ML is supported by an NSF Graduate Research Fellowship, and AM is recipient the Zuckerman Postdoctoral Fellowship. We thank the Center for Al Safety (CAIS) for making computing resources available for this research.
REFERENCES
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2022.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.06 01130.88122 12, 2023.
Adrien Bibal, Rémi Cardon, David Alfter, Rodrigo Wilkens, Xiaoou Wang. Thomas François, and Patrick Watrin. Is attention explanation? an introduction to the debate. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 1889-3900, 2022.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACI. Workshop Challenges & Perspectives in Creating Large Language Models, 2022. URL hnttps://arxttt /ttpa/aiii..oraab////eeeeee 06745.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020, URL https: / //ppr//eeooceeeeseeeeeeee ceessnnerippp rpurips. httplee//ppppeeee///e/// /atee///22///0// fitee//ffee////// 457c9d6bf 4ww.e444466771811111111111 8bfb8acc 42f Paefff4aa-aaper... pdf.
Alonzo Church. An unsolvable problem of elementary number theory. American journal of mathe- matics, 58:345-373, 1936.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019.
Alexis Conneau, Guillaume Lample. Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017.
Word translation without parallel data, arXiv preprint arXiv:
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in contentet?? language models secretly perform gradient descent as meta-optimizers, In Findings of the Association for Computational Linguistics: ACL 2023, pp. Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/2023 findings-acl. 247. URL https:://///..... ht////aaaaaanttgtooeee org/2033. findings-ace acl. 247.
Toronto,
10

---

Published as a conference paper at ICLR 2024
Guy Dar. Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Anna Rogers. Jordan
Graber, and Naoaki Okazaki (eds.), Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16124-16170, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.11865//1/2223.cc-ha
htt.ttts////aaaaataoooooo http:///aaaaaatoggg..oooo org/ 2023 acl-long. 893.
URL https:
N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer eircuits. Transformer Circuits Thread, 2021.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30599 2022.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Framcine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653//122021.emm 2ttp://w12220...mmmmaaaaa. mainlp-main.46.. URL https: //aclanthology.co orgee/orrr/////// wttpe//rrr//22...emmmmaaammm 444.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.1863311/0222.mmmlp //aclanthology org/2022. emn ewn..-manlllnle.e mmman.....
URL https:
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. /4767, 2023.
Michael Hahn and Navin Goyal. A theory of emergent in-context learning implicit structure induction. arXiv preprint S3XV::2303.07771, 2023.
Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint urXiv:2307.0944 2307.0947777 2023.
Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. In-context learning of large language models explained as kernel regression. arXiv preprint urXiv:2305. 12766, 2023.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguisties: EMNLP 2023, 2023.
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representations in language models. In Arxiv, 2023a. URL https://arxiv.orrg http://axxxii.or//bbaa// abs//304../2 abbp:////3////////33//////// 00740.
Evan Hernandez, Arna Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.091 09124, 2023b.
Felix Hill, Adam Santoro, David Barrett, Ari Morcos, and Timothy Lillierap. Learning to make analogies by contrasting abstract relational structure. In International Conference on Learning Representations, 2018.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In Anna Rogers, Jordan Boyd-Graber. Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pp. 1935-1952, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 1al2 1on. 108. URL /acas///olola org/2023.8 acl- acl-long. 108.
Phu Mon Htut, Jason Phang. Shikha Bordia, and Samuel R Bowman. Do attention heads in BERT track syntactic dependencies? arXiv preprint 1rXiv:1911. 121,,1246,, 2019.
11

---

Published as a conference paper at ICLR 2024
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023.
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3543-3556, 2019.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, 2020.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4365-4374, 2019.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-seque sequence recurrent networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2873-2888, PMLR, 10-15 Jul 2018. URL https: pttp///eoccee.....//// mlr. press/v80/ //s//k/0 8a html.
Andrew K Lampinen and James L McClelland. Transforming task representations to perform novel tasks. Proceedings of the National Academy of Sciences, 117(52)33990-3322) 2020.
Omer Levy and Yoav Goldberg. Dependency- -ased word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 302-308, 2014.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- intervention: Eliciting truthful answers from language model. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023a. URL https: httpss///ppeeeeeeenneee www.wwwwww..oooumiiiieee aLLuYpn83y.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transform- ers as algorithms: Generalization and stability in in-context learning. Proceedings of the 40th International Conference Machine Learning, 2023b.
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside BERT's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 241-253, 2019.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. arXiv preprint 2xXiv23311.06666 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35:17359- 15517359-17372, 2022.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec- style vector arithmetic. arXiv preprint arXiv:2305. 16130, 2023.
Tomas Mikolov, 1lya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1 11048-1 1064, 2022.
12

---

Published as a conference paper at ICLR 2024
Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning compress prompts with gist tokens. arXiv preprint XXXv:2330.008467, 2023.
Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Distinguishing antonyms and synonyms in a pattern-base neural network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume 1, Long Papers, pp. 76-85, Valencia, Spain, April 2017. Association for Computational Linguisties. URL https:
//acllatholoog.orggfEBt
Nostalgebraist. Interpreting GPT: The logit lens. URL https: (akrBoooppaaGGvrrrninnteeetttrtd
www.w/.ssswrogcoomm com/posts/
tot--th--oggit-tee tens,, 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
OpenAL. GPT-4 technical report, 2023.
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning" learns" in-context: Disentangling task recognition and task learning. arXiv preprint arXIv:2305.09731, 2023,
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06660 2023.
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty and Artificial Intelligence 2001, pp. 411-420. Morgan Kaufman, 2001.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of BERT, Advances in Neural Information Processing Systems, 32, 2019.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.0666 2023.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language- independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NACCL 2003, pp. 142-147, 2003.
Nan Shao, Zefan Cai, Chonghua Liao, Yanan Zheng, Zhilin Yang, et al. Compositional task representations for large language models. In The Eleventh International Conference on Learning Representations, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL http::ttp///// http://wwwwwwwlwl.. .tt..or/////aooo.eeee httgg/nthology/D111 11700
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 566-581, 2022.
Gerald Jay Sussman. Scheme: an interpreter for extended lambda calculus. MIT Al Memo, 1975.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQAA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp.
2019.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Alex Tumer, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Acti- vation addition: Steering language models without optimization. arXiv preprint wrXhv:2308.10244 2023.
Alexandre Variengien and Eric Winsor. Look before you leap: A universal emergent decomposition of retrieval tasks in language models. arXiv preprint arXiv:2312.1000 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:1 00:1128881222 -288011 2020.
Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. Context-aware neural machine translation learns anaphora resolution. Proceedings the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 1264-1274 2018
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-hear self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 5797-5808, 2019.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174 PMLR, 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https://githbbe cwt http:/ttww....oooooooo/o/////taa
May 2021.
Kevin Ro Wang. Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2022a
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv 2023a.
preprint ar arXiv:2305.14
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons in pre-trained hansformer-basee language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11132-11152 152, 2022b.
Xindi Wang, Yufei Wang, Can Xu, Xiubo Geng, Bowen Zhang, Chongyang Tao, Frank Rudzicz, Robert E Mercer, and Daxin Jiang. Investigating the learning behaviour of in-context learning: A comparison with supervised learning. arXiv preprint arXiv:2307.155: 154111, 2023b.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. In Workshop on Efficient Systems for Foundation Models at ICML2023, 2023c. URL https://openrewt
met/forumiid:HCkk
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Demny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint 13Xi:2233330384 2023.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNI.P-IJCNIP)) pp. 11-20, 2019.
14

---

Published as a conference paper at ICLR 2024
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. arXiv preprint arXiv:2303.07895 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the th-t--rrt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, Octoher 2020. Association for Computational Linguistics. URL https: //t::///wwwwww acl web org/ / momlp-demos. 6.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning implicit bayesian inference. In International Conference Learning Representations, 2021.
Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang- Woo Lee, Sang- goo Lee, and Taeuk Kim. Ground-trute labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2422-222222222 2437, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang. and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprini arXiv:2305. 2023.
Sumu Zhao, Damián Pascual, Gino Brunner, and Roger Wattenhofer. Of non-linearity and commuta- tivity in bert. In 202/ International Joint Conference on Neural Networks (IJCNN) pp. 1-8. IEEE, 2021.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Katie Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023.
15

---

Published as a conference paper at ICLR 2024
A DISCUSSION: FUNCTION VECTORS VS SEMANTIC VECTOR ARITHMETIC
In this appendix we discuss the experimental support for our characterization of function vectors in more detail, in particular the assertion that function vectors are acting in way that is distinct from semantic vector arithmetic on word embeddings.
The use of vector addition to induce a mapping is familiar within semantic embedding spaces: the vector algebra of semantic vector offistss has been observed many settings; for example, word embedding vector arithmetic was clearly described by Mikolov et al. (2013), and has been observed in other neural word representations including transformers (recently, Merullo et al., 2023). In recent examinations of internal transformer states, Geva et al. (2022) and Dar al. (2023) have suggested that many internal transformer calculations can be understood in terms of such word vector arithmetic. Therefore one of our main underlying research questions is whether our function vectors should be described as triggers for nontrivial function, or whether, more simply (as would be suggested by Dar al.), they could be thought of as just ordinary semantic vector offsets that induce a trivial mapping between related words by adding an offset to a rich word embedding.
Our investigation of the possibility of a distinct and localized representation for tasks is similar to the studies of task encodings in RNNs by Lake & Baroni (2018) and Hill et al. (2018), as well as the studies of linguistic attribute encodings by Clark et al. (2019), but our focus on causal effects rather than correlations allows us to explicitly extract and test the computational roles of vector representations of functions, which leads to several new lines of evidence.
The main paper contains three pieces of experimental evidence that support the conclusion that function vectors are different from semantic vector offsets of word embeddings, and that they trigger nontrivial functions:
1. Function vectors implement complex mappings, including cyclic mappings such as antonyms that cannot be semantic vector offsets.
2. Function vector causal effects cannot be recovered from the target output vocabulary alone; they carry some other information.
3. Function vector activity is mediated by mid-layer nonlinearities (i.c., they trigger nonlinear computations), since they have near-zero causal effect late layers.
We discuss each of these lines of evidence in more detail here.
Cyclic Tasks Cannot be Semantic Vector Offsets. The first task analyzed in the paper is the antonym task. Because the antonym task is cyclic, it is simple counterexample to the possibility that function vectors are just semantic vector offsets of language model word embeddings.
Suppose there were an 'antonym' vector offset v, such that adding to word embedding w produces w', the embedding of the antonym of w (i.e. w + Va anlonym(w) = w'). An example of this might be vec (big) + U vec (small). But then: if and w' are antonyms of each other, the relationship holds both ways. That means that the antonym offset vector Uo would have to satisfy both + 2 w' and w' + va w, which can only happen if v, = O, implying w w' and creating a contradiction to the assumption that w and w' are antonyms, distinct from each other. Thus there could be no constant vector offset that would properly model the antonym task. The same argument excludes semantic vector offsets for any cyclic function. We evaluate additional cyclic tasks in Appendix L.
Since we are able to find constant antonym function vector ey that, when added to the transformer, does cause cyclic behavior, we conclude that the action of 1/ is a new phenomenon. Function vectors act in a way different from simple semantic vector offsets.
Function Vectors Contain Information Beyond Output Vocabulary. Not every function is cyclic, but the vector offset hypothesis can be tested by examining word embeddings. Following the reasoning of Geva et al. (2022); Dar et al. (2023), one way to potentially implement a semantic vector offset is to promote certain subset of tokens that correspond to particular semantic concept (i.e. capital cities, past-tense verbs, etc.). Function vectors do show some evidence of acting in this way: when decoding function vectors directly to the model's vocabulary space often see that the tokens
16

---

Published as a conference paper at ICLR 2024
with the highest probabilities words that part of its task's output vocabulary (see Table 5 and Table 19).
To determine whether it is the vocabulary itself that contributes to the function vector's performance, in Section 3.2 we construct another vector (via optimization) that decodes to the same vocabulary distribution and measure its performance when used as "function vector". In that experiment create reconstructions 12100 and Ctall that encode the same decoder vocabulary as U, (the near-zero KL divergences show that the reconstructions of the decoder vocabulary are near-perfect). Yet the performance of the reconstructions when used as FV is poor when the top 100 words are matched, and still lower than 2y even when the distribution over the full vocabulary is reconstructed.
That experiment reveals that while function vectors do often encode words contained in the output space of the task they are extracted from, simply adding a vector that boosts those same words by the same amounts is not enough to recover its full performance-thoree some cases a fraction of performance is recovered. Our measurements suggest that while part of the role of function vector may act similarly to a semantic vector offset, the ability for function vectors to produce nontrivial task behavior arises from other essential information in the vector beyond just a simple word embedding tocabulary-basee offset. See Appendix N for a related experiment.
if enough words are included, in
Function Vectors' Causal Effects are Near-Zero at Late Layers. Across the set of tasks and models we evaluate, there is common pattern of causal effects that arises when adding a function vector to different layers. The highest causal effects are achieved when adding the function vector at early and middle layers of the network, with a sharp drop in performance to near-zero at the later layers of the network (Figure 4, Figure 26, Figure 14, Figure 16). Interestingly, FV causal effects are strongest in largest models, yet the cliff to near-zero causal effects also sharpest for the largest models (Figure 16).
If the action were linear and created by a word embedding vector offset, then the residual stream would transport the action equally well at any layer including the last layers. Thus the pattern of near-zero causal effects at later layers suggests that the action of the function vector is not acting directly on the word embedding. but rather that it is mediated by some nonlinear computations in the middle layers of the network that are essential to the performance of the task. This mediation is evidence that the function vector activates mid-layer components that execute the task, rather than fully executing the task itself.
This pattern is in contrast to the vector arithmetic described in Merullo et al. (2023), that are most effective at later layers of the network and have little to no causal effect at early layers of the network; those offsets more closely resemble semantic vector offsets of word embeddings.
Function Vectors Therefore Represent Functions summary, the three lines of evidence lead us conclude that the vectors U2 should not be seen as simple word embeddings, nor trivial offsets or differences of embeddings, nor simple averages of word embeddings over vocabularies of words to be boosted. These characteristics distinguish function vectors from from many linguistic concepts that can be viewed as probability distributions over words. Rather, the evidence suggests that the vectors we have identified act in way that is distinct from literal token embedding offsets. Instead they directly represent and trigger nonlinear execution of abstract functions.
This finding is surprising since the transformer is trained on word- prediction task; it would be expected that they should learn representations that can be expressed in terms of adjustments to word probabilities as observed by Geva et al. (2022); Dar et al. (2023). Our evidence indicates different kind of representation: we find that transformers learn a compact, concrete, and causal vector representation of higher-level functional concepts that cannot be reduced to probability vector over a set of words.
Thus we come to the conclusion that the vectors v. are references functions, and call them function vectors.
17

---

Note that a transformer repeats this process with different weights and data at each layer; we add the layer subscript to disambiguate their notation. In the Vaswani et al. formulation, each head at layer E resides in a low dimensional space Rd with dimension dy < d that differs from the main Published as a conference paper at ICLR 2024
B ATTENTION OUTPUTS ARE INDEPENDENT AND ADDITIVE
In this section define our formulation of attention notation øej in detail, relating our notation to the original formulation of Vaswani et al. (2017) via the framework of Elhage et al. (2021).
B. EXPRESSING ATTENTION Ozj IN THE RESIDUAL STREAM SPACE
The transformer architecture as introduced by Vaswani et al. (2017) has multihead attention mechanism. They describe it in terms of concatenation procedure for performing the attention function for several "heads" headj E R2d (i S J) at each layer, all in parallel. Equation 9 reproduces the relevant equation from page 5 of Vaswani et al.:
MultiHead(Q.. K, V) Concat(head,
Mead,)W"
where heady = Attention ( vw))
(10)
hidden state residual stream of the transformer, which we write as h2 Rd. All the heads the layer are concatenated and then transformed through transformation W to produce the full MultiHead, attention output in Rd
Elhage et al. (2021) observes that the Vaswani et al. formulation is equivalent to dividing the matrix WO into block form [WQ W2 ... W and then projecting each headg, into the residual stream space directly. In our formulation of agj in Section 2.2, we adopt this view. The attention head output agj can be defined in terms of the notation of Vaswani et al. and Elhage et al. as follows:
agj = heade, W E R
(11)
In this way the total attention contribution at layer the sum of the attention output of each individual head, and these all reside directly in the Id space of the residual stream:
MultiHcad, (Qe. Ke, Ve) = Σ
atj E R
(12)
JSJ
While the left and right-hand sides of equation 12 are computationally equivalent definitions of attention, the "independently additive" form of attention on the right-hand side allows us to see the contributions of individual attention heads more clearly.
B.2 ADDING FUNCTION VECTORS TO LAYER
Using this formulation of attention dej. return notation defined in section 2.2 to understand what we mean when we say we add a function vector to a layer
We focus the hidden state residual stream the final token of given prompt. Recall that the ordinary operation a transformer defines the hidden state
h h me +
Σ dE
(13)
JSJ
This recursive residual structure forms telescoping sum that creates common vector space for nearby layers; for example Zhao et al. (2021) has observed that adjacent layers of a transformer can be swapped with little change. Thus it is meaningful to collect, average, and sum attention head outputs dez E Rdd among nearby layers, and that observation inspires our definition of a function vector Vt average attention head values for selection of relevant heads āæÿ (equation 5). See Appendix M for an analysis of an alternative formulation that does not swap layers.
Since the function vector also resides in the residual stream space of h Rd. when we add function vector to the hidden state of layer f, we can therefore add it to the hidden state just as attention is added; we write the updated hidden state h' as
h h + me +
Σ agj + U2
(14)
55J
This could also be written as simply h/ h + Vt-
18

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL DETAILS
In this section, we provide details the function vector extraction process (section 2.3), and the evaluation of function vectors (section 3).
Function Vector Extraction. We compute function vector the sum over the average output of several attention heads, where the average is conditioned on prompts taken from a particular task. We write this as 2/ Eayea o How many attention heads should we use? To extract function vector (FV), we first compute the task-conditioned mean activation of each head dzy. using [P:| = 100 clean (uncorrupted) 10-shot prompts. We use this to identify a set causal attention heads A, which
ranked based on the average indirect effect (AIE) of each head. For GPT-J we found that the increase in performance for many tasks begins to plateau when using |.A| = 10 attention heads, though for some tasks using more heads increases performance even more (Figure 6).
The AIE is computed over subset of all abstractive tasks (Appendix E), using P: = 25 corrupted 10-shot prompts per task. Because we are interested in tasks the model can successfully do via ICL, task is only included if its 10-shot ICL performance is better than the baseline (majority-label) performance. For GPT-J, there are [7] 18 tasks satisfying this criteria which we use to compute the AIE (Figure 8).
Antonym
Capitalize
Capitalize First Letter
Country-Capital
Accuracy
0.5
0
Country-Currend
English-Prent
English- Germann
-panish
Accuracy
0.5
Landmark Landmarkk-Cuunry
Lowerease First Letter
National Parks
Park Park-Country
Accuracy
0.5
0
Person-Sport
Present-Past
CroducctCompany
Sentiment
Accuracy
0.5
0
0
10
20
30
0
10
20
30
Singular-Phi Plurall
Synonym
Number of Heads
Number of Heads
Accuracy
0.5
0
0
10
20
30
10
20
30
Number of Heads
Number of Heads
Figure 6: Zero-shot accuracy across 18 different tasks for adding a function vector to GPT-J. We vary the number of heads in A that are used to create the function vector and find that the change in performance begins to plateau around |A] 10 attention heads for majority of the tasks. For this reason, we use [A| 10 for GPT-J.
Evaluating Function Vectors. To evaluate a function vector's (FV) causal effect, we add the FV to the output of particular layer in the network (h₂) the last token of prompt p with query Zg, and then measure whether the predicted word matches the expected answer yn- This can be expressed simply f(p b he + vx) (see section B for more detailed explanation). We report this top-1
19

---

Published as a conference paper at ICLR 2024
accuracy score over the test set under this intervention. If By is tokenized as multiple tokens, the first token of Yy as the target token.
For all results in Section 3 and in the following appendix sections (unless stated otherwise c.g. Figure 4), we add the FV to the hidden state at layer z LL//3, which we found works well in practice. This corresponds to layer 9 for GPT-J, layer 15 for GPT-NeoX, layer 11 for Llama (7B), layer 14 for Llama 2 (13B) and layer 26 for Llama 2 (70B).
Prompt Templates. The default template we use to construct ICL prompts is: Q: (zak)) AnA: [vak J n,, where Fike and Yikk (or Vsk for comrupted prompts) substituted for the corresponding element in brackets, and each example is concatenated together. An example of full prompt template is:
Q: [ AnA: (Ba Pinin (=aN)) AnA: (ViN) nn (2ig) AnA:
(15)
To evaluate function vector we use a few different prompt contexts. The shuffled-label prompts are conrupted 10-shot prompts with the same form (15), while zero-shot prompts only contain query Ty. without prepended examples (c.g. Q: {zig (nAA))
In section 3.1 we use FVs extracted from prompts made with the template shown in (15), and test them across a variety of other templates (Table 8).
Table 8: We test variety of ICL prompt templates in $3.1. which are shown below. The function vectors (FVs) we collect are constructed from default template the form Q: {z AnA {yx M An)n. and tested on prompts created with the new prompt form.
Template Forms
question: (ara) \ answer: [vee) (nyn,
question: (Ir=a answer: (Ma+)l
A: A:{ra B: [ys Anin,
(ma))) Answer: (sra) inn,,
Input: [x:e] Output: (1i*]l
[.x=] - ++[xx))nnn
{zu.))n in,
input: (2ta) output:
( answer: {ya).
x: x: Fln y:
input: {as (outppt::: An.
(zua + (ya.)n
input: [mae] output: (via (n.
In: [na Out: (ya
text: [2t labbll: (vra) An)n,
x: [max]] ((x):<uu
x: far Hy: (yu
A: [aa B: (36a)
text: (æxe [label: (n.
x: [aax nn y: [ma) An)n
20

---

Published as a conference paper at ICLR 2024
D RESULTS INCLUDING INCORRECT ICL
For simplicity of presentation in Section 3, filter the test cases where the model correctly predicts By given 10-shot ICL prompt containing query Tg. In this section we compare those results the setting in which correct-answer filtering not applied. When filtering not applied, the causal effects of function vectors remain essentially unchanged (Figure 7).
Antonym
Capitalize
Capitalize First Letter
Country-Capital
Accurccy
0.5
0-
20 Layer
Layer
Layer
Layer
Country-Curenccy Currency
English-Fremn French
English German
English-Spunie
Accuscc
0.5
0
4
16 Layer
12
20
24
0
4
8
12
16 20
24
0
4
8
12 16 20 24
12 16
20 24
Layer
Layer
Layer
Landmark-Counti
Lowercase First Letter
National Parks
Park-Country
Accuracy
0.5
0-
24
Layer
Layer
Layer
Layer
Person-Sport
Present-Pass
Product-Company
Sentiment
0.5
0
4
8
12
16
20
24
0
4
12
16
20 24
0
4 8 12 16 20
24
8
12
16
20 24
Layer
Layer
Layer
Layer
Singular- Plural
Synonym
GPT-J 10-Shot Performance GPT-H+FV Unfiltered Test Set
Accuracy
0.5
GPT-J on Unfiltered Test
GPT-J+FV on Filtered Test Set
GPT-J on Filtered Test Set
0
Layer
Layer
Figure 7: Comparing layer-wise zero-shot results of adding function vector to GPT-J with and without filtering the task test set to cases where GPT-J correctly answers a 10-shot prompt. The results when filtering the test set in this manner are shown in blue, while the results without filtering the test set are shown in magenta. In both cases the performance is very similar, with performance dropping only slightly on few tasks when not filtering the dataset to correct answers. The black dashed line corresponds the oracle accuracy (GPT-J's s 10-shot performance) in the unfiltered setting, while colored dashed lines correspond to GPT-J's performance in the zero-shot setting.
21

---

Published as a conference paper at ICLR 2024
E DATASETS
Here, we describe the tasks we use for evaluating the existence of function vectors. A summary of each task can be found in Table 9.
Antonym and Synonym. Our antonym and synonym datasets are based on data taken from Nguyen et al. (2017). They contain pairs of words that are either antonyms or synonyms of each other (c.g. "good + bad", or "spirited and verb pairs from all data splits and then filter out duplicate entries. We then further filter to word pairs where both words can be tokenized single token. As result, keep 2,398 antonym word pairs and 2,881 synonym word pairs.
fiery"). We create an initial dataset by combining all adjective, noun,
We note that these datasets originally included multiple entries for single input word (e.g. both "simple + difficult" and "simple- - complex" are entries in the antonym dataset). In those cases we prompt a more powerful model (GPT-4; OpenAI, 2023) with 10 ICL examples and keep an answer as output after manually verifying it.
Translation. We construct our language translation datasets - English--French Einlish-German and English-Spanish using data from Conneau et al. (2017). which consists of word in English and its translation into a target language. For each language, we combine the provided train and test splits into a single dataset and then filter out cognates. What remains are 4, 4,705 pairs for English-French. 5,154 pairs for English- English-German, and 5,200 pairs for English-Spanise
These datasets originally included multiple entries for single input word (e.g. both "answer > respuesta" and "answer - contestar" are entries in the Anglish-Spanisee dataset), and so we filter those with GPT-4, in a similar manner as described for Antonym and Synonym.
Sentiment Analysis. Our sentiment analysis dataset is derived from the Stanford Sentiment Tree- bank (SST-2) Socher et al. (2013), a dataset of movie review sentences where each review has binary label of either "positive" or "negative". An example entry from this dataset looks like this: "An extremely unpleasant film. - negative". We use the same subset of SST-2 as curated in Honovich al. (2023), where incomplete sentences and sentences with more than 10 words are discarded, leaving 1167 entries in the dataset. See Honovich et al. (2023) for more details.
This is a question answering dataset where model is given a question and options, each labeled with letter. The model must generate the letter of the correct answer. For example, given "Where is business restaurant likely to be located?" and answer options "a: town, b: hotel, c: mall, d: business sector, e: yellow pages", a model must generate "d", (Talmor et al., 2019)
CommonsenseQA.
AG News. A text classification dataset where inputs are news headlines and the first few sen- tences of the article, and the labels are the category of the news article. Labels include Business, Science/Technoloy, Sports, and World. (Zhang et al., 2015)
We also construct set of simple tasks to broaden the types of tasks on which we evaluate FVs.
Capitalize First Letter. To generate list of words to use to capitalize, we utilize ChatGPT² by prompting it to give us a list of words. From here, we curate a dataset where the input is a single word, and the output is the same word with the first letter capitalized.
Lowercase First Letter. Similar to the Capitalize First Letter task, we use the same set of words but instead change the task instead lowercase a word. The input single word title cased, and the output is the same word, but lowercase instead.
We also generate a list of coaptay-cppitals with ChatGPT. Here, we ask ChatGPT come up with a list of countries and following that, ask it to name the capitals that are related to the countries. This dataset contains 197 country-capite city pairs.
"https://chat.openaa
22

---

Published as a conference paper at ICLR 2024
We also generate list of country-currency pairs with ChatGPT. Similar to country-capital, we ask ChatGPT to come up with a list of countries and following that, ask it to name the currencies that related to the countries.
National Parks. The National Parks dataset consists of names of official units of national parks in the United States, paired with the state the unit resides in (e.g. Zion National Park -> Utah). It was collected from the corresponding page Wikipedia', accessed July 2023.
Park-Country. The Park-Core Cark-Couttry dataset consists of names of national parks around the world, paired with the country that the park is in. The countries were first generated by ChatGPT, then the parks were also generated by ChatGPT. After, the dataset was hand-checked for factual accuracy since ChatGPT tended to hallucinate parks for this dataset. A subset of all national parks is used.
Present-Past. We generate a list of present-tense verbs with ChatGPT then ask ChatGPT to find the past-tense version. After generation, the dataset was hand-corrected for inaccuracies. The dataset inputs are simple present tense verbs and outputs are corresponding simple past tense verbs.
The Landmark-Count dataset consists of entries with the name of a land- mark, and the country that it is located in. The data pairs are taken from Hernandez al. (2023b).
Londmark-Country
dataset contains entries with the name of professional musician and the instrument they play. The data pairs are taken from Hernandez et al. (2023b).
The
The Person-Occupation dataset is taken from Hernandez et al. (2023b), and contains entries of names of well-known individuals and their occupations.
Perspp-Occupation.
Person-Sport. The Person-Sport dataset is taken from Hemaande al. (2023b), and each entry consists of the name of professional athlete and the sport that they play.
The Podduct-Compan dataset contains entries with the name of a commercial product, paired with the company that sells the product. It is curated from Hernandez et al. (2023b).
Product-Company.
Next-Item. The Next-Item dataset contains pairs of words which communicate the abstract idea of "next". Our pairs are made up of days of the week, months of the year, letters of the alphabet (which cyclic), and number pairs (both numeric and text form). Some examples entries in this dataset "Monday" + "Tuesday", "December" + "January", "a" - "b", and "seven" "eight".
Previous-Item.
The Previous-Iten dataset contains the reciprocal version of the pairs of words in the "Next-Item" dataset, communicating the idea of "previous". Example entries include: "Tuesday" + "Monday", "January" + "December", and "a" + "z".
E. EXTRACTIVE TASKS.
Many NLP tasks are abstractive: that is, they require the generation of information not present the prompt. We also wish to test whether function vectors recoverable from extractive tasks that is, tasks where the answer is present somewhere in the prompt, and the task is to retrieve it.
CONLL--0033 In our experiments, we use a subset of the CoNLL-2003 English named entity recognition (NER) dataset Sang & De Meulder (2003), which is common NLP benchmark for evaluating NER models. The NER task consists of extracting the correct entity from a given sentence, where the entity has some particular property. In our case, we create three different datasets: NER- person, NER-location and orgaingann, where the label of each task is the name of either person, location, or organization, respectively. Each dataset is constructed by first combining the CONIIL-2003 "train" and "validation" splits into single dataset, and then filtering the data points to only include sentences where single instance of the specified class (person, location, or organization) is present. This helps reduce the ambiguity of the task, as cases where multiple instances of the same class are present could potentially have multiple correct answers.
Phttps://en //en. wik / http://stttofootttlteeeeee
units
23

---

Published as a conference paper at ICLR 2024
As with abstractive tasks, also construct set of new extractive tasks.
Choose 7th Item from List. Here, the model is given list of comma-separated items, and the model tasked with selecting the item at specific index. We construct tasks where the list size is either 3 or 5. In our tasks, we have the model choose either the first element last element in the list.
Choose Category from List. These tasks are similar to our choose rihh element tasks, but instead, the model must select an item of particular type within list of or items. word with the correct type is included once in the list while the remaining words are drawn from another category. The categories we test include the following: fruit vs. animal, object vs. concept, verb vs. adjective, color animal, and animal vs. object.
Task Name
Task Source
Abstractive Tasks
Antonym
Nguyen et (2017)
Capitalize first letter Capitalize
Country-capital
Country-currency
English-Freech
Conneau et al. (2017)
English-German
Conneau et al. (2017)
English-Spanish
Conneau et al. (2017)
Landmark-Countr
Hernandez et al. (2023b)
Lowercase first letter
National parks
Next-item
Previous-item
Park-country
Person-instrument
Hernandez et al. (2023b)
Person-occupation
Hernandez et al. (2023b)
Person-sport
Hernandez et al. (2023b)
Present-past
Product-company
Hernandez et al. (2023b)
Singular-plural
Synonym
Nguyen et al. (2017)
CA (MC-QA) Sentiment analysis (SST-2) AG News
Talmor et al. (2019)
Socher et al. (2013)
Zhang et al. (2015)
Extractive Tasks
Adjective vs. verb
Animal vs. object
Choose first of list
Choose middle of list
Choose last of list
Color vs. animal
Concept object
Fruit vs. animal
Object concept
Verb vs. adjective
CoNLL-20033, NER-person
Sang & De Meulder (2003)
CoNLL-2003, NER-location
Sang De Meulder (2003)
CoNLL-2003, NER-organization
Sang & De Meulder (2003)
Table Summary tasks used in this study. Tasks without sources are tasks we construct.
24

---

Published as a conference paper at ICLR 2024
Antonym
Capitalize
Capitalize_ _irst Letter
Capitalize Last Letter
Accurey
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
Capitalize Seconddde__d Letter
Commonsense Qa
Country-Capital
Country-Curen Curreeyyy
Accunacy
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
English-French
English-Geerman
English-Spanish
Landmark-
Accuncyy
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
Lowercase First Letter
Lowerease Last Letter
National Parks
Next Capital Letter
Accuracy
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
Park-Country
Person- Instrmmnnttument
Person-Occupatie
Person-Sport
Accuracy
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
Present-Pass
Product-Company
Sentiment
Singular-
Singular-Plural Plural
Accunccy
0.5
Number of Shots
Number Shots
Number Shots
Number of Shots
Synonym
Accuracy
0.5
Number of Shots
Figure 8: Few-shot ICL performance (top-1 accuracy) for GPT-J on a set 25 abstractive-style tasks. In general, more shots improves performance. However, for many of these tasks the accuracy plateaus after a number of shots. The dotted baseline shows the accuracy of predicting only the majority label.
E.2 FEW-SHOT ICL PERFORMANCE
Figure 8 shows the few-shot performance (top-1 accuracy) for GPT-J on a larger subset of our task list. The dotted baseline is based on the majority label for the dataset, computed as (# majority label)/(# total instances).
Figure shows the few-shot performance (top-1 accuracy) for GPT-J on additional extractive tasks. For datasets with lists of words as input, the dotted baseline computed to be 1/size of the input list, (e.g. 1/3, 1/5), and for all other tasks it represents predicting the majority label of the dataset, computed as (# majority label)/(# total instances).
E.3 EVALUATING FUNCTION VECTORS ON ADDITIONAL TASKS
25

---

Published as a conference paper at ICLR 2024
Adjective V Verb
Adjective Verb
Alphabetically First 3
Alphabetically First
Accuracy
0.5
2
4
6
8
10
2
10
8
10
2
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Alphabetically Last
Alphabetically Last
Animal V Object
Animal Object
Accuracy
0.5
2
4
6
8
10
2
10
8
10
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Choose First
Choose First Of
Choose Last Of
Choose Last
Accuracy
0.5
2
4
6
8
10
2
10
8
10
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Choose Middle Of 3
Choose Middle Of
Color Animal
Color Animal 5
Accuracy
0.5
2
4
6
8
10
2
10
6
8
10
4
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Concept V Object
Concepl V Object
Conll2003 Location
Coull2003 Organization
Accuracy
0.5
2
4
8
10
2
10
8
10
2
4
6
8 10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Conll2003 Person
Fruit Animal 3
Fruit V Animal
Object Concept
Accuracy
0.5
2
4
6
8
10
2
10
8 10
8 Number of Shots
6
10
Number of Shots
Number of Shots
Number of Shots
Object V Concept
Verb V Adjective
Verb Adjective
Accuracy
0.5
2
4
8
10
0
2
10
8
10
Number of Shots
Number of Shots
Number of Shots
Figure 9: Few-shot ICL performance (top-1 accuracy) for GPT-J on set of 27 extractive-style tasks. In general, more shots improves model performance. dataset with 3 or 5 at the end denotes the size of the input list of words used. There are few cases where the model cannot perform the task any better than random (e.g. choose the alphabetically first word a list), which we do not analyze further. The dotted baseline shows the accuracy 1/ (list size), or the majority label in the case of the datasets derived from conl12003.
26

---

Published as a conference paper at ICLR 2024
Ag News
Antonym
Capitalize
Capitalize First Letter
Accuracy
0.5
2
4
10
0
2
10
8
10
2
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Capitalize Last Letter
Capitalize Second Letter
Commonsense Qa
Country-Capital
Accuracy
0.5
10
2
10
10
2
4
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Country-Currenge
English-French
English-Gemn German
Soglish-Spanish
Accuracy
0.5
2
4
10
0
2
10
8
10
2
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Landmark Landmark-Cunntry
Lowercase First Letter
Lowercase Last Letter
National Parks
Accuracy
0.5
4
6
10
2
10
8
10
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Next Capital Letter
Park-Country
Person- trument
Gerson-Occupation
Accuracy
0.5
2
4
10
0
2
10
8
10
2
4
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
-erson
Present-Pate
Product-Company
Sentiment
Accuracy
0.5
2
4
10
0
2
10
8
10
2
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Singulur-PP Pluurll
Synonym
Accuracy
0.5
2
4
6
10
0
2
10
Number of Shots
Number of Shots
Figure 10: Few-shot ICL performance (top-1 accuracy) for Llama 2 (13B) set of 26 abstractive- style tasks. Using more ICL examples (shots) improves performance for many of these tasks, though the performance does plateau after few shots for many of the tasks. There Llama 2 (13B) cannot perform the task any better than random (e.g. capitalize the second letter the word), which do not analyze further. The dotted baseline shows accuracy choosing the majority label.
few cases where
27

---

Published as a conference paper at ICLR 2024
Adjective V Verb
Adjective Verb
Alphabetically First 3
Alphabetically First
Accuracy
0.5
2
4
6
8
10
2
10
8
10
2
4
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Alphabetically Last
Alphabetically Last
Animal V Object
Animal Object
Accuracy
0.5
2
4
6
8
10
2
10
8
10
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Choose First
Choose Of
Choose Last Of
Choose Last
Accuracy
0.5
2
4
6
8
10
2
10
8
10
4
6
8
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Choose Middle Of 3
Choose Middle Of
Color Animal
Color Animal 5
Accuracy
0.5
2
4
6
8
10
2
10
8
10
6
10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Concept V Object
Concept V Object
Conll2003 Location
Coull2003 Organization
Accuracy
0.5
2
4
6
8
10
2
10
8
10
2
4
6
8 10
Number of Shots
Number of Shots
Number of Shots
Number of Shots
Conll2003 Person
Fruit Animal
Fruit V Animal
Object Concept
Accuracy
0.5
2
4
6
8
10
2
10
8 10
8 Number of Shots
6
10
Number of Shots
Number of Shots
Number of Shots
Object V Concept
Verb V Adjective
Verb Adjective
Accuracy
0.5
2
4
6
10
0
2
10
8
10
Number of Shots
Number of Shots
Number of Shots
Figure 11: Few-shot ICL performance (top-1 accuracy) for Llama 2 (13B) set of 27 extractive- style tasks. Using more ICL examples improves performance for most of the tasks. dataset name ending with 3 or 5 denotes the size of the input word list. There cannot perform the task any better than random (e.g. choose the alphabetically first word a list), which do analyze further. The dotted baseline shows the accuracy 1/(list size) (e.g. 1/3, 1/5. or the majority label in the case of the con112003 datasets.
few cases where the model
28

---

Published as a conference paper at ICLR 2024
Antonym
Capitalize
Capitalize First_ Letter
Country-Capital
Accuracy
0.5
16
24
16 Layer
20
12
20
Layer
Layer
Layer
Country- Country-Currency
English- Frenich
Baglish-Germann
Accuracy
0.5
16
24
16 24
16 20
20
Layer
Layer
Layer
Layer
Landmark-Countyy
Lowercase First Letter
National Parks
Park-Country
Accuracy
0.5
16 24
16 24
16 20
12
20
Layer
Layer
Layer
Layer
Person-Sport
Present-Past
Product-Company
Sentiment
Accuracy
0.5
8
16 24
16 20 24
16
20 24
4
12
16
20
24
Layer
Layer
Layer
Layer
Singular-Plural
Synonym
Accuracy
0.5
8
12 16 20
24
0
8 12 16 20
24
Layer
Layer
Figure 12: Zero-Shot Top-1 1 Accuracy Results of adding FVs to GPT-J across of our task datasets. In addition the 6 analyzed in the main paper, we see similar results and performance across variety of other tasks adding FVs in early-middle layers seems have the most effect. Notable exceptions include lowercase first letter, person-sport, synonym, and sentiment, where the FV doesn't it seem to have much effect. Note we did not evaluate the FV on tasks where the model's ICL performance was poor (compared to the majority label baseline) (Figure 8).
29

---

Published as a conference paper at ICLR 2024
Adjective V Verb 3
Adjective V Verb 5
Animal V Object 3
Animal V Object 5
Accuracy
0.5
8
16
24
12 20
12 20
12 24 Layer
Layer
Layer
Layer
Choose First 3
Choose First 5
Choose Last Of
Choose Last Of
Accuracy
0.5
8
16
24
12 16 20
12 20
0 12 20 24
Layer
Layer
Layer
Layer
Choose Middle Of 3
Color V Animal 3
Color V Animal 5
Concept V Object
Accuracy
0.5
8
12
16
20
24
4
12
16
20
24
4
12
16
20
24
0
8
12
16 20 24
Layer
Layer
Layer
Layer
Concept V Object 5
Conl12003 Location Conll2003 Organization
Conll2003 Person
Accuracy
0.5
8
12
16
20 24
4
12
16
20
24
4
12
16
20
24
8 12
16 20 24
Layer
Layer
Layer
Layer
Fruit Animal
Fruit Animal
Object Concept
Object V. Concept
Accunscy
0.5
16
12
20
24
Layer
Layer
Layer
Layer
Verb V Adjective 3
Verb V Adjective 5
Accuracy
0.5
0
8 12 16 20
24
12 16 20 24 Layer
Layer
Figure 13: Zero-Shot Top-1 Accuracy Results of adding FVs to GPT-J across 22 of our task datasets. Shown here are mainly extractive tasks, and we see similar trends across all tasks. The FV perfor- mance is fairly stagnant across layers, with many tasks having peak performance in middle layers of the network. The model's zero-shot performance without intervention is plotted as a dotted line. Adding the FV causes the model extract the correct entity much more often than the base model in this uninformative zero-shot case.
30

---

Published as a conference paper at ICLR 2024
Ag News
Anlonym
Capitalize
Capitalize First Letter
Accuracy
0.5
10
3035
20
10
20 30335
20 30
Layer
Layer
Layer
Layer
Commonsense Qa
Country-Capital
Country-Currenne
English-Frend
Accurney
0.5
10
20
Layer
Layer
Layer
Layer
English-Gemma
Eingish-Spanis
Handmark-Counte
Lowercase First Letter
Accurccy
0.5
0
10
3035
30
10
20 Layer
Layer
Layer
Layer
Lowercase Last Letter
National Parks
Park-Country
Person-Instrument
Accurcy
0.5
0
10 20
20
20
10
20 30
Layer
Layer
Layer
Layer
Person-Oecupattio
Person-Sport
Present-Pass
Product-Company
Accuracy
0.5
10 20 30.3
15.20 30
10
30
20
30
Layor
Layer
Layor
Layer
Sentiment
Singular-Plural
Synonym
Accuracy
0.5
0
10 15 3033
20
10 20
Layer
Layer
Layer
Figure 14: Zero-shot results for adding a function vector (FV) at different layers of Llama 2 (13B) across a 23 abstractive-style tasks. Adding the FV at early-middle layers gives good performance on the task, but there is drop in performance when adding the FV at later layers of the model (around layer 20-25 for Llama (13B)).
31

---

Published as a conference paper at ICLR 2024
Adjective V Verb
Adjective V Verb
Animal V_ Y_bjectt.
Animal V Y_Ojjcct_ 5
Accuncy
0.5
10
15
3035 Layer
25
10 20
15 20 30.5
25
Layer
Layer
Layer
Choose First 3
Choose First Of
Choose Last Of
Choose Last 0f
Accuracy
0.5
0
5
10
15
20
25
30
10 15 2025
10 15 20
25 30
0
5 10
15 20 25 30
Layer
Layer
Layer
Layer
Color V Animal 3
Color V Animal 5
Concept V Object 3
Concept V Object
Accuracy
0.5
0
10
15
20
25 3035
10 15 20
10
15
20
25
30
5 10 20 30
Layer
Layer
Layer
Layer
Conl12003 Location Lonation Conl12003 Oraannizztonnnnitiiinn
Conli2003
Fruit V Animal
Accunccy
0.5
10 15
20 30.5
10 20255333333
15 20 30
20
Layer
Layer
Layer
Layer
Fruit Animal 5
Object V Concept 3
Object
Concept 5
Verb Adjective 3
Accuracy
0.5
0
5 10 15 20 25 30
10 15 20.5 35
10 15 20
25 35
0 5 10 20155020 30
Layer
Layer
Layer
Layer
Verb V Adjective 5
Accuracy
0.5
0
5 10 15 20
25 3035
Layer
Figure 15: Zero-shot results for adding a function vector (FV) at different layers of Llama (13B) across 21 extractive-style tasks. Adding the FV at earlier layers has higher causal effect of performing the extracted task, while later layers see performance dip.
32

---

Published as a conference paper at ICLR 2024
Antonym
Capitalize
Capitalize First Letter
Country-Capital
Accuracy
0.5
0
16
32
48
64
16
32
48
64
16
32
48
64
16
32
48
64
Layer
Layer
Layer
Layer
Country- Country-Currency
English-Frenn
Dnglish-Germin
Spanish
Accuracy
0.5
0
16
32
48
64
0
16
32
48
64
16
32
48
64
16
32
48
64
Layer
Layer
Layer
Layer
Landmark-Country
Lowercase First Letter
National Parks
Park-Country
Wyll
Accuracy
0.5
0
16
32
48
64
0
16
32
48
64
16
32
48
64
16
32
48
64
Layer
Layer
Layer
Layer
Person-Instrument
Person-Occupation
Person-Sport
Present-Past
Accuracy
0.5
16
32
48
64
32
48
64
16
32
48
64
16
32
48
64
Layer
Layer
Layer
Layer
Product-Company
Sentiment
Singular-Plun
Synonym
Accuracy
0.5
48
64
48
64
32 48
64
32 48
Layer
Layer
Layer
Layer
Figure 16: Zero-shot results for adding function vector (FV) at different layers of Llama 2 (70B)
across a 20 abstractive-style tasks. Adding the FV at early-middle layers gives good performance on the task, and there is sharp drop in performance when adding the FV at later layers of the model (after layer 48 for Llama 2 (70B)).
33

---

Published as a conference paper at ICLR 2024
Adjective V Verb
Adjective V Verb_
Animal V Y_Ojject_ 3
Animal V Object
Accuracy
0.5
0
16
32
48
64
16
32
48
64
16
32
48
64
0
16
32
48
64
Layer
Layer
Layer
Layer
Choose First Of
Choose First of 5
Choose Last Of
Choose Last Of
Accuracy
0.5
0
16
32
48
64
16
32
48
64
16
32
48
64
0
16
32
48
64
Layer
Layer
Layer
Layer
Choose Middle Of
Color V Animal 3
Color Animal 5
Concept Object
Accuracy
0.5
64
48
32
16
Layer
Layer
Layer
Layer
Concept V V_Object55 5
Conl12003 Location Conl12003 Conll20033Organizzatte Organization
Conl12003 Person
Accuracy
0.5
0
16
32
48
64
16 32 48
64
16
32
48
64
0
16
32
48
64
Layer
Layer
Layer
Layer
Fruit V Animal 3
Fruit V Animal 5
Object V Concept 3
Object V Concept 5
Accuracy
0.5
16
32
48
64
16
32
48
64
16
32
48
64
0
16
32
48
64
Layer
Layer
Layer
Lay er
Verb V Adjective 3
Verb V Adjective
Accuracy
0.5
64
32
48
Layer
Layer
Figure 17: Zero-shot results for adding a function vector (FV) at different layers Llama (70B) across 22 extractive-style tasks. In general, adding the FV early-middle layers gives good performance on the task, and there often sharp drop in performance when adding the FV at later layers of the model (around layer 48 for Llama (70B)).
34

---

Published as a conference paper at ICLR 2024
F PORTABILITY
In addition the quantitative results provided in $3.1, we include additional qualitative examples for natural text completions for the Antonym FV (Table 10). We also include qualitative examples of completions on different natural text prompts using the English-French FV (Table 11), the English- Spanish FV (Table 13), and the Country-Capital FV (Table 14).
We also include additional quantitative results for testing FVs in natural text settings on various natural text templates, averaged over 5 seeds for each of the remaining representative tasks: Capitalize (Table 16), Present-Past (Table 17), Singular-Plural (Table 18), English-French (Table 12), and Country- Capital (Table 15).
Table 10: Additional Qualitative Examples of the Antonym FV in Naturalistic Text Settings
Antonym Task
Prompt: The word "(x)", means
(a) GPT-J The word "limitless", means "without (mmits".Ammnnnii word
(a) GPT-J + Antonym FV The word "limitless", means "finite". AnlnnTh word
(b) GPT-J The word "improvement". means "to make better". Anterr"AnnnnThe
(b) GPT-J Antonym FV The word "improvement" means "deterioration". AAaaa
Prompt: When think of [x], usually
(a) GPT-J When think of sit, usually think of chair. But, in"
(a) GPTJJ Antonym FV When I think of usually think of stand. \n),,
(b) GPT-J When I think of maximum, think of the maximum number of people that
(b) GPT-J Antonym FV When I think of maximum, I usually think of minimum. An think
Prompt: While reading a book, came across the word "[x}" x)"". looked up a dictionary and turns out that it means
(a) GPT-J While reading book, I came across the word "credible". I looked it up in dictionary and it out that means "believable" nn (a) GPT-J Antonym While reading a book, came across the word "credible". looked it up in dictionary and it turns out that it means "unbelievable". An
(b) GPT-J While reading book, came across the word "marriage". looked it up in dictionary and turns out that means legal union between a man and While reading a book, came across the word "marriage". looked it up in dictionary and it turns out that it means "divorce", nl came Prompt: The word } can be understood synonym for
(b) GPT-J Antonym
(a) GPT-J The word erroneous can be understood as a synonym for wrong, but it is also a synonym for
(a) GPT-J + Antonym FV The word erroneous can he understood as synonym for correct. n''Te
(b) GPT-J The word injure be understond a synonym for harm, but it also has meaning
specific
(b) GPT-J + Antonym FV The word injure can be understood as a synonym for heal.) n'The word
35

---

Published as a conference paper at ICLR 2024
Table 11: Evaluating the English-French FV in Naturalistic Text Settings
English-French Task
Prompt: The word "[x]". means
(a) GPT-J The word "readers", means "people who read". "nldd""Anmmmm
(a) GPT-J + English-French FV The word "readers", means "lecteurs""
(b) GPT-J The word "sky". means "the heavens" many languages.
(b) GPT-J English-French FV The word "sky", means "ciel" in français
(c) GPT-J The word "daily", means "every day".Am nn'eTh word
(c) GPT-J + English-French FV The word "daily", means "tous les jours",
Prompt: The word '(x)' can be understood as synonym for
(a) GPT-J The word 'link' can be understood as synonym for 'connection' or 'relation'. The term 'link' used the context of the Internet, (a) GPT-J English-Free Frennch FV The word 'link' can be understood as a synonym for 'lien', et le mot 'lien" peut être compris comme un synonyme de 'lien', ct
can used other contexts
(b) GPT-J: The word 'prohibited' can be understood as synonym for 'forbidden' 'forbidden do', (b) GPT-J + English-French FV "The word 'prohibited' can be understoud as synonym for 'interdit'
Prompt: While reading a book, came across the word "[x]". looked it up in a dictionary and it turns out that it means
(a) GPT-J While reading a book, came across word "knight". looked up dictionary turns out that means "a person who is a member of military order". Ai nl was wondering if there is similar word for person who is member
(a) GPTTJ English-Free FreiihhhFFeehhhhh FV While reading book, I came across the word "knight". looked it up in dictionary and turns out means "chevalier". An' nle lis livre, un chevalier.
Table 12:
Natural text portability of the English-Frent FV. Given natural template we substitute query word for 'x'. We measure accuracy based on whether the correct french translation is produced this setting within generated tokens.
Prompt
GPT-J
+English-Frende FV
The word "x", means
0.0 1 0.0%
51.2 1 1.2%
When think of the word "x", it usually means
0.1 + 0.1%
41.7 + 1.3%
When think of x, I usually
0.0 + 0.0%
19.0 + 1.5%
While reading book, came across the word "x".
0.1 = 0.1%
45.6 + 1.1%
looked up in dictionary and it turns out that it means The word can be understood as synonym for
0.3 + 0.2%
44.2 + 1.6%
36

---

Published as a conference paper at ICLR 2024
Table 13: Evaluating the English-Spanish FV in Naturalistic Text Settings
English-Spanish Task
Prompt: The word [[x]", means
(a) GPT-J The word "children", means "offispring of the human species". A\ \nThe word "children" is used in the Bible
(a) GPT-J English-Spanish FV The word "children", means "niñios" in español. \nknThe word "chil- dren", means
(b) GPTJJ The word "foods", means "food" in the sense of "food" and "drink" in the sense of (b) GPT-J + English-Spanish FV The word "foods", means "alimentos"" "nn nnThe word "alimentos", significa "alimentos
Prompt: While reading a book, came across the word [[x]" looked up a dictionary and turns out that it means
(a) GPT-J While reading book, came across the word "attacking" looked it up in dictionary and turns out that means "to attack". nl nI was wondering if there is difference between two words.
(a) GPT-J English-Spanish FV While reading book, I came across the word "attacking". I looked it up in a dictionary and it turns out that it means "atacar", An) A leido un libro encontré la palabra "atacar", nPPro
(b) GPT-J While reading book, came across the word "computer", looked it up in a dictionary and turns out that means "a machine that computes", \nn nI was surprised to learn that the word "computer" is derived from the word "computor".
(b) GPT-J English-Spaniie FV While reading book, came the word "computer". I looked up dictionary and turns out that means computador, (nn,nn looked up in dictionary y en un diccionario y que dice que es un computador. A Prompt: When I think of (x). usually
(a) GPTJ When think of find, I usually think the command line.
(a) GPT-J+ + English-Spanisl FV When think find, I usually think de encontrar, \ When
(b) GPT-J When I think of tomb, usually think of tombstone. But, in the case of the tomb of
(b) GPT-J + English-Spanish FV When I think of tomb, I usually pense cámara de sepul-
cro. cro.Ap),nCuande nCunndo
(c) GPT-J When think of shoes, I usually think of the ones I wear. I'm not a shoe person
(e) GPT-J English-Spanish FV When think of shoes, usually pense sobre zapatos. \ nhhen pense sobre
37

---

Published as a conference paper at ICLR 2024
Table 14: Evaluating the Country- FV in Naturalistic Text Settings
Country-Capital Task
Prompt: A couple of years ago visited (x), and
(a) GPT-J: couple years ago visited Grenada, and was struck by the beauty of the island. I was also struck by the fact that the island was a British colony until 1974. was also
(a) GPT-J + Country-Cap
FV A couple years ago I visited Grenada, and was in the capital, St. was there see the Grenada National Museum, and I was there to
George's I
(b) GPT-J: A couple years ago I visited Italy, and I was lucky enough to be able to visit the city of Florence. I was there for a few days, and was able to see lot (b) GPT-J + Country-Capital FV A couple years ago visited Italy, and was in Rome. I was in the Colosseum, and was in the Roman Forum. I was in the Pantheon, and
(c) GPT-J: couple years ago visited Belarus, and I was surprised to see that the country was still using the old Soviet-era typewriters. I was even more surprised to see that the typew
(c) GPT-J Country-Capita FV A couple years ago visited Belarus, and I in Minsk. I was there for few days, I was Minsk
(d) GPT-J: A couple years ago I visited Mongolia, and was struck by the fact that the country was so different from the rest of the world. It was land of nomads, and the people
(d) GPT-J Country-Capita FV A couple years ago visited Mongolia, and was in Ulaanbaatar. was there for a few days, and was in the capital city of Ulaanbaatar Prompt: If you ever travel to (x). you have to visit
(a) GPT-J: If you ever travel Italy, you have to visit the city of Florence. It the capital of Tuscany and the most visited city in Italy. is also the home of famous U
(a) GPT-J +
FV If you ever travel to Italy, you have to visit Rome. It's the capital of Italy and the largest city in the world. It's also the most visited city in the world
(b) GPT-J: you ever travel to Thailand, you have to visit the island Koh Samui. It a beautiful island with a lot things to do. The island famous for its beaches, water sports
FV If you ever travel to Thailand, you have to visit Bangkok. It is the capital of Thailand and the largest city in the country. Bangkok is the most populous city in the world. (n nBaggkkk
(b) GPTJJ + Country-Capite
(c) GPT-J: If you ever travel to Saint Lucia, you have to visit the Pitons. The Pitons are group of three mountains that are located on the island of Saint Lucia. The Pitons are the highest mountains
(c) GPTTJ Country-
FV If you ever travel Saint Lucia, you have to visit the capital city of Castries. It is most beautiful city the Caribbean. It a very beautiful city. a very beautiful city
Prompt: When you think of (x}.
(a) GPT-J: When you think of Netherlands, you probably think of tulips, windmills, and cheese. But the Netherlands also home to a thriving cannabis industry (a) GPT-J + Country-Capite FV When you think of Netherlands, you think of Amsterdam. But there are many other citics in the Netherlands. Here are some of the best places to visit in
(b) GPT-J: When you think of Egypt, you probably think of pyramids, mummies, and the Nile River. But did you know that Egypt is also home to
(b) GPT-J + Country-Capita FV When you think of Egypt, you think of Cairo, the pyramids, and the Nilc. But there are many other places to visit in Egypt. Here
Table 15: Natural Text Portability quantitative results for the Country-Capitts FV. We substitute in query country names for 'x', and measure accuracy based on whether the correct capital city name is produced within 10 generated tokens.
Prompt
GPT-J
+Country-Capiin FV
A couple of years ago visited {x], and If you ever travel to (x), you have to visit When you think of [x).
3.9 4: 3.2%
56.7 4: 5.3%
23.2 +: 3.5%
70.4 3.7%
7.4 + 4.8%
72.4 2.7%
38

---

Published as a conference paper at ICLR 2024
Table 16: Natural text portability of the Capitalize FV. Given a natural template we substitute a query word for 'x'. We measure accuracy based on whether the correct capitalization is produced in this natural text setting within 5 generated tokens.
Prompt
GPT-
+Capitalize FV
The word "x", means
5.6 0.5%
94.3 = 1.0%
When think the "xr,""", "x", usually means
3.7 0.7%
84.5 = 1.3%
When think x, usually
12.5 2.1%
76.1 + 2.9%
While reading a book, I came across the word "x", looked it up in a dictionary and it turns out that it means The word x can be understood as a synonym for
6.8 0.7%
97.8 + 0.6%
5.5 + 0.8%
81.5 = 2.8%
Table 17: Natural text portability of the Present-Past FV. Given a natural template we substitute query word for 'x'. Then then measure accuracy based on whether under the FV intervention the correct past-tense word is produced within 5 generated tokens.
Prompt
GPT-J
+Present-Past FV
The word "x". means
0.0 1 0.0%
49.3 + 4.5%
When think the word "x", usually meuns
0.1 0.3%
67.2 + 4.2%
When think x, I usually
0.7 0.7%
16.1 + 5.4%
While reading book, came across the word "x".
0.0 0.0%
55.8 + 3.6%
looked it up a dictionary and it turns out that it means The word x can he understond as synonym for
0.0 0.0%
57.2 + 3.0%
Table 18: Natural text portability of the Singular-Plural FV. Given natural template we substitute query word for 'x'. Then then measure accuracy based on whether under the FV intervention the correct past-tense word is produced within 5 generated tokens.
Prompt
GPT-J
+Singular- Plagula FV
The word "x", means
0.0 0.0%
82.9 + 3.6%
When I think of the word "x", it usually means
0.0 + 0.0%
72.2 4: 0.6%
When think of x, I usually
0.0 - 0.0%
36.0 + 8.4%
While reading a book, I came across the word "x".
0.0 + 0.0%
81.1 + 2.1%
looked up in a dictionary and it turns out that it means The word can be understood as synonym for
0.3 + 0.6%
77.0 + 5.0%
39

---

Published as a conference paper at ICLR 2024
G CAUSAL MEDIATION ANALYSIS
In this section we include additional figures showing the average indirect effect (AIE) split up across tasks for GPT-J (Figure 18), as well as the AlE for other models we evaluated.
Anlonym
Capitalize
Capitalize First Letter
Country-aape
Head Index
0
0.05
0.0
0.00
0.00
10
0.05
0.05
-0.25
Curuntyy
Eaglish-French French
English- Gemmann
English--paanis
Head Indox
nos
0.1
0.
10
0.05
10-
-0.1
Landmark-Coren Country
Lowercase First Letter
National Parks
Park- Count-Cuuury
Head Indox
ul
0.05
0.05
0.1
nn
10
10
-0,05
40.05
Person-Spart Spart 0
Present- Past
Product- Product-Company
Sentiment
Head Indox
0,05
10
10
-0.05
-0.05
Singular-Plune Plural
Synonym
Head Indox
0
0.005
10
-0.05
10
-0.006
Layer
Layer
Figure 18: Average Indirect Effect (AIE) for attention heads in GPT-J, shown by task. The set of heads that have higher causal effeet is fairly consistent across tasks.
The AIE for particular model is computed across all abstractive tasks where the model can perform the task better than the majority label baseline given ICL examples. We include heatmaps the AIE for each attention head in Llama 2 (7B) (Figure 19), Llama (13B) (Figure 20), and Llama (70B) (Figure 22), as well as GPT-NeoX (Figure 21). For cach model, the heads with the highest AIE typically clustered in the middle layers of the network. In addition the maximum AIE across all heads tends to drop slightly as the size the model increases, though the total number of heads also increases. In Llama 2 (7B) the max AIE is R 0.047, while in Llama 2 (70B) the max AIE is 0.037.
40

---

Published as a conference paper at ICLR 2024
0
0.04
10
Head Index
ATE
20
-0.04
30
0
10
20
30
Layer
Figure 19: Average Indirect Effect (AIE) for each attention head in Llama 2 (7B) the final token. This is computed across all abstractive tasks where the model can perform the task with 10 ICL examples. The heads with the highest AIE are mainly clustered in the middle layers of the network. Compared to GPT-J the AIE of the most influential heads is less (+e 0.047 vs. s 0.053 in GPT-J) but there are also more than double the number attention heads
0
10
0.04
Head Index
AIE
20
30
-0.04
10
20
30
Layer
Figure 20: Average Indirect Effect (AIE) for each attention head Llama (13B) at the final token.
41

---

Published as a conference paper at ICLR 2024
0
10
20
0.03
Head Index
30
AIE
40
-0.03
50
60
10
20
30
40
Layer
Figure 21: Average Indirect Effect (AIE) for each atttention head in GPT-NeoX at the final token. Interestingly, the heads with the highest AIE here clustered in earlier middle layers (from layer 10-20), whereas in other models the heads are clustered more towards the middle of the network.
0
10
0.03
20
Head Index
30
AIE
40
50
-0.03
60
0
10
30
40
50
60
70
Layer
Figure 22: Average Indirect Effect (AIE) for each attention head Llama (70B) at the final token.
42

---

Published as a conference paper at ICLR 2024
H ATTENTION PATTERNS AND PREHIX-MATCHHE SCORE
Across a variety of tasks, the heads with highest causal effect have a consistent attention pattern where the attention weights on few-shot ICI. prompts are the strongest on the output tokens of each in-context example. Here we show this pattern for GPT-J on 4 additional tasks (Figure 23, Figure 24), which match the patterns shown in the main paper (Figure 3b). This is similar to the attention pattern that might be expected of "induction heads", which has previously been shown to arise when prompt contains some repeated structure (Elhage et al., 2021; Olsson al., 2022).
Singular-Plural
1.0
Attention Weight
0.5
Present-Pass
0.0
take
took
help
helped
turn
turned
begin
began
lend
lent
plan
Figure 23: Attention weight visualizations for the singular-plura and present-pas tasks for the attention heads with the top average indirect effects GPT-J. Across tasks, the attention weights are consistently the strongest on the output tokens of each exemplar.
Country-Capital
1.0
Attention Weight
Capitalize
0.0
igloo
Igloo
horse
Horse
mint
Mint
from
From
orange
Orange
they
Figure 24: Attention weight visualizations for the country-capital and capitalize tasks for the attention
heads with the top 10 average indirect effects in GPT-J. Across tasks, the attention weights are consistently the strongest on the output tokens of each exemplar.
To further investigate whether the heads identified via causal mediation analysis are "induction heads". compute the prefix-matchee matching score for each head in GPT-J. We follow the procedure as described in (Olsson et al., 2022; Wang et al., 2022a), which computes the prefix- neefi--matching score as the average attention weight token B when given sequence of the form [A, B.... A). This is
43

---

Published as a conference paper at ICLR 2024
measured sequences of repeated random tokens. We do this for each head in GPT-J with results shown in Figure 25b.
(a)
0.05
(b) 0
Prefix Matching Score
Index
Head Index
5
0.5
10
10
-0.05
15
15
0
15
20
25
5
15
Layer
Layer
Figure 25: (a) Average Indirect Effect (AIE) per attention head for GPT-J. (b) Prefix-matching score per attention head for GPT-J. For both (a) and (b), we highlight in pink the top 10 heads by AIE. There are three heads that have both relatively high AIE and prefix-matching score (Layer-Head Index 8-1, 12-10, and 24-6). There are also several heads with high AIE that do have a high prefix-matching score, and vice-versa.
We find that three the heads out of those with the top 10 highest AIEs (Figure 25) also have high preix-matching scores. In terms "Layer-Hess Head Index", these are heads 8-1. 12-10, and 24-6, with beeix-matching scores of 0.49, 0.56, and 0.31, respectively.
While (Elhage al., 2021; Olsson et al., 2022) show that induction heads play critical role in copying forward previously seen tokens, our results show that they are also among the set of heads, A, that have the highest AIE when resolving few-shot ICL prompts.
There are several other heads we identified with relatively high causal effect that have the same attention pattern activation on few-shot ICL. prompts, but not produce the same "induction" attention pattern on sequences of random repeated tokens.
This suggests that while induction heads play role in the formation of function vectors, there are other heads that also contribute relevant information that may not be induction heads of the type observed by (Elhage et al., 2021; Olsson al., 2022).
44

---

Published as a conference paper at ICLR 2024
1 DECODING VOCABULARY EVALUATION
Table 19: Additional tasks and the top 5 vocabulary tokens of their decoded FV. Across variety of outputs, most encodings are aligned to the task they were extracted from.
Task f
Tokens in the distribution D(a,) in order of decreasing probability
Capitalize First Letter
'CN", 'DR', 'RR', Ct', 'Ct'
Country-Cureene
Japanese', Chinese', Arabic', Russian', Americann agl', Ei', e', 'actual', "V1''
English-German
English-Spanish
aki', è', 94l', mase' 'operator' Germany', Japan', Netherlands', Italy', Spain' 'dr', nr', Ic', mc² mr' Connecticut', California', Wisconsin', Netherlands', Pennsylvania'
Landmark- Country
Lowercase First Letter
National Parks
Park- Countryuntry
Netherlands" Germany', , Japan', Italy', Mexico"
Person-Sport
basketball', football', soccer', baseball', tennis'
Product-Company
Microsoft', Motorola', Samsung', Disney', IBM positive', negative', 'positive', 'negative', neeural' edible', adjective', noun', slang', caster'
Sentiment
Synonym
Here, we present more results on the evaluation of decoding vocabularies of FVs, over additional datasets. Across the tasks, affirm that output spaces seem be frequently encoded in the FVs, as can be seen in Table 19. In particular, in cases such as sentiment that follow a rigid pattern, the tokens referring the output distribution for sentiment is well encoded. On the other hand, some tasks like language translation do not have output spaces well-encoded in the FVs.
45

---

Published as a conference paper at ICLR 2024
J SCALING EFFECTS
Can consistently locate function vectors given various sizes of single language model architec- ture? We test this by observing all sizes of Llama 2, ranging from 7B parameters to 70B. We use the same methods in $3.1, adding function vectors each layer of the model and observing accuracy on our subset of 6 tasks at each layer.
Antonym
Capitalize
Countr-CCapiaie
English-Freed Freiihh-Frenchhhh
Present-Past
Singular-
Liama 2 (7B)
12
182430
12182430
12
Accuracy
Liama 2 (138)
24 32
32
16
24 32
16 24 32
Liama 2 (70B)
0.5
16 32 48 64 Layer
16 32 48 64 Layer
16 32 48 64
0 16.32 48 64
16 32 48 64
16 32 48 64 Layer
Layer
Layer
Layer
Figure 26: Zero-shot accuracy across Llama 2 model sizes in zero-shot settings. We show accuracies before adding the function vector (dotted lines) and after adding the FV to specific layer (solid lines).
find results (Figure 26) are largely consistent across model sizes. Function vectors generally result in the highest zero-shot accuracies when added to the early middle layers; this is true regardless the total number of layers in the model.
46

---

Published as a conference paper at ICLR 2024
K COMPOSITION ON OTHER MODELS
In this section we include additional composition results for Llama 13B and 70B models in Tables 20 and 21 respectively.
Table 20: The accuracy of ICL, calculated FV VED zero-shot interventions, and vector--omposed teo zero- shot interventions when performing several list-oriented tasks on LLaMA 2 (13B).
Task
ICL (ten-shot)
URD (FV on zero-shot)
Ven (sum on zero-shot)
Last AsttoAnmmym
0.53 0.02
0.26 + 0.03
0.17 0.02
Last-Capitalize
0.94 = 0.01
0.63 = 0.03
0.70 H 0.03
Last-Cuutry-Capipa
0.86 0.02
0.73 +: 0.02
0.37 + 0.04
Last-English-Frendd
0.75 0.02
0.32 0.02
0.12 1: 0.02
Last- Present-Past
0.96 0.01
0.22
0.24 0.02
Last- SostttSinular-PPP Plural
0.89 0.01
0.43 = 0.03
0.53 0.03
Last Bast-Capitalize Capitalize-First-Aee
First-Letter
0.85 0.02
0.89 = 0.02
0.89 + 0.02
Dast-Product--o Product-Company Company
0.47 0.01
0.44 = 0.03
0.60 0.03
Table 21: LLaMA 2 (70B)
Task
ICL (ten-shot)
enn (FV on zero-shot)
ven (sum on zero-shot)
Last- Aast-Antonym
0.67 0.03
0.43 = 0.03
0.47 0.03
Last- Last-Capitalize
0.99 = 0.00
0.93 = 0.01
0,95 + 0.01
Last-Countr-Capital
0.81 = 0.03
0.91 + 0.02
0.94 + 0.02
Fast-English-Frenchh
0.84 0.02
0.13 + 0.01
0.17 0.03
Last- Present-Past
0.98 0.01
0.93 + 0.01
0.94 0.01
Last-Singular- Si---iinuar-PPurll
Plural
0.98 0.01
0.69 0.04
0.69 0.04
Last cestt-Cpptaaizz-Fiine First-Letter
0.67 0.03
0.60
0.68 + 0.03
Dast-Product-Oe Complyy
0.49 0.02
0.34
0.34 = 0.03
47

---

Published as a conference paper at ICLR 2024
L EVALUATING FUNCTION VECTORS ON CYCLIC TASKS
In Appendix A, we discuss whether function vectors (FVs) can be thought of simple word vector offsets, and show that cyclic tasks (such antonyms) counterexample to this claim. In this section we report the causal effects function vectors on two additional tasks with cyclic subsets "next-item" and "previous-item" simple semantic vector offsets but instead can be thought of as a trigger of nontrivial functions.
providing further evidence that function vectors not just
Monday
Monday
Sunday
Tuesday
Sunday
Tuesday
Saturday
Wenesday
Saturday
Wenesday
Friday
Thursday
Friday
Thursday
Next Item
Previous Item
Figure 27: An example of cyclic structure for days of the week, which is a subset of the data for both the Next-Item and Previous-Item tasks. The cycles in each task follow the opposite order (e.g. Next-Item(Monday) Tuesday, but Previous-Irem(Mndday) = Sunday.
The "next-item" task contains pairs of words which are related via the abstract idea of "next". The "previous-item" task contains the reciprocal version of the word pairs in the "next-item" task. Flipping the direction in this manner means each pair communicates the idea of "previous" instead. Both tasks collected over heterogeneous
of sequential data that includes cyclic types such days the week, months of the year, and letters of the alphabet, as well as non-cyclic types such as numbers and roman numerals. We include samples of example data pairs for these datasets in Appendix E.
However, single ICL example for the Previous-Irees task might look like "Q: Friday \nA: Thursday' nn): six) nA: five a) a',nAzZ)n'nn VII' The model would ideally be able to answer "August" given this ICL, prompt.
September (nA"".
GPT-J
Llama (7B)
Llama 2 (13B)
Llama (70B)
Next Item
0.5
Accuracy
Previous Item
0.5
5 N Shots
10
5
10
5
10
5 N Shots
10
N Shots
N Shots
Figure 28: ICL performance the "Next-Item" and "Previous-Item" Item" cyclic tasks for 4 different models. The performance is usually better for the next-item task than for the previous-item task. However, 10-shot performance suggests these models able perform these tasks fairly well.
48

---

Published as a conference paper at ICLR 2024
In Figure 28, report the ICL n-shot performance of each of these two tasks for GPT-J, and each model the Llama 2 family. We find that the models perform this task well given 10 example pairs, with the performance of the next-item task being higher than the previous-item task. Performance generally increases when more examples are provided.
GPT-J
Llama (7B)
Llama (13B)
Llama (70B)
Next Item
0.5
mny
Accuracy
25
12 18 24:30
8 16
24
32
40
16 32 48 64 80
Previous Item
0.5
15 20 25
12 18 30
8 16 24 32 40
16 32 48 64 80 Layer
Layer
Layer
Layer
Figure 29: Zero-shot accuracy on the "Next Item" and "Previous Item" cylic tasks for GPT-J and all sizes of Llama 2. We show model accuracies before adding the function vector (dotted lines) and after adding the FV to a specific layer (solid lines). The function vector improves performance significantly for both tasks compared the model's zero-shot baseline.
We extract function vector for each of these tasks and evaluate their performance in the zero-shot setting, adding the function vector to different layers the network. In Figure 29, we report the zero-shot accuracy of each model before adding the function vector with dashed line, and the accuracy after adding a function vector with a solid line. We see that the function vector significantly improves performance for each task compared to the zero-shot baseline. In addition, the trends for these datasets generally follows those previously reported for other tasks (cyclic or not). The peak performance is achieved when adding the FV to carly-middle layers, and there sharp drop in performance about 2/3 /3 of the way through the network.
Table 22: A few example outputs of adding the "next-item" and "previous-item" function vectors to layer GPT-J. We see that the function vectors are able to correctly trigger the cyclic behavior of "next" "previous" when presented with boundary case, while the base model usually defaults to copying the input query.
Tnpot Prompt:
Q SundaylnA
'Q: Decemberinn:
'Q. zind=
'Q sevelnn"
'Q: 211mA
"Q Monday
'Q: January InA
GPT-J
Sunday
Decembeer
eight
Idon't know
Tuesday
Janusry
Monday
Febcuary
Gre-s-Preinu--tem
Saturday
November
y
Sunday
In Table 22, include a few example outputs of zero-shot prompts for both the baseline model and the model when we add the corresponding function vectors of either "next-item" or "previous-item", showing their ability to correctly induce the "next" or "previous" cyclic behavior.
For antonyms, the cyclic behavior gives a contradiction in two additions. That is, given a word wi and vector offset v that give the antonym of w1, then expect to return to w, after adding again (i.e Wi + 2 n w1). The cyclic subsets in studied here have longer cycles (e.g. for days of the week, and offset v', we'd d expect W2 7 of w2). but the same reasoning applies. Because FVs can trigger the corresponding cyclic behavior, this provides additional evidence that they are not just doing simple word vector arithmetic.
49

---

Published as a conference paper at ICLR 2024
M AN ALTERNATIVE EVALUATION OF FUNCTION VECTORS
Recall that define function vector (v²) for particular task (1) the sum of the task-conditioned mean activations (ā)j over a small set of attention heads (A4) (see equation 5, Section 2.3). Given function vector created this manner, we can test its causal effects by adding to a single hidden state of model a particular layer and measuring its ability trigger particular task behavior (see Appendix B.2 for more details).
An alternative approach test whether the outputs of the attention heads in A can trigger particular task behavior is to instead add their task con-ccondt mean activations (0t,) to their corresponding layer's hidden states, and to do so at every layer that is implicated by the heads contained in A. This is in contrast to the FV, which adds all these attention head outputs single layer.
As the model performs computation at layer k. the alternative approach updates the hidden state h2 by adding the aaskcconditioned mean activations of all heads in A that output layer k. If we represent the attention heads in with (layer, head index) tuples, then we write the updated hidden state h2 as:
h +
Σ
de
(16)
(0,))
tt=k
We perform the update intervention as specified in equation for all layers represented by A.
Antonym
Capitalize
Country-Can Capital
English-Free Frenchh
Present-
Singular-Plure
Shuffled-Lai L-bbed--abbel Zero- Shot
0.5
Accuracy
0.5
0
6
12 18
24
12 18 24 Layer
6 12 18
24
0 12 18 24
12 18 24 Layer
6 12 18 24
Layer
Layer
Layer
Layer
GPT-J Performance
GPT-J-FV Performance
GPT-J+ Altemative Performance
Figure 30: Comparing the causal effects of function vectors (solid blue line) and an alternative approach (red dashed line), which adds the components of an FV to their respective layers instead of a single concentrated layer. The model baseline in each setting is shown with a dashed blue line. Zero-shot results for 6 tasks are shown in the first row, and shuffled-label results are shown in the second row. In the zero-shot setting, the alternative approach matches FV performance for most tasks. It performs worse for English-Freet French translation and better on the Present- Past task. In the shuffled-label setting, the alternate approach matches FV performance for all tasks.
In Figure 30, we compare the causal effects of the alternative approach described in equation 16 to the original function vector formulation for both zero-shot and shuffled-label contexts across our representative tasks using GPT-J. The base model performance is shown with a dashed blue line, and the solid blue line shows performance when we add the function vector to layer f. The results of the alternative approach are shown with red dashed line.
In the zero-shot setting, the alternative approach matches the performance of the function vector for majority of the tasks. It performs worse on English- Frenchh and better on the Present-Pass task. the shuffled-labed setting, the alternative approach matches the causal effects of function vector for all tasks.
On average, we find that using the alternative approach to measure the causal effects of A compared to the original function vector formulation works about as well, typically achieving the same peak performance. However, the function vector approach does highlight interesting phenomena of performance dropoff around 2/3 the way through the network which not possible to see when using the alternative approach.
50

---

Published as a conference paper at ICLR 2024
N INVESTIGATING FUNCTION VECTOR EFFECTS IN VOCABULARY SPACE
In the main paper (Section 3.2, Table 6), examine quantitative evidence that the action of function vector (FV) cannot be explained by simply boosting set of words directly encoded by the function vector in the style of Dar al. (2023), see Appendix A for more detailed discussion. In this section we examine the causal effects of several function vectors in vocabulary space to understand the relationship between the words that are encoded in function vector and the words that are boosted by the transformer when we intervene with an FV.
Unlike previous analysis, in this section we investigate how adding a function vector (ot) to layer & changes the distribution of log probabilities over set of relevant tokens (w; E W) compared to the baseline model's response. That is, for token W, we compute:
Alogprob = log(s(p' h : h (w:)) - (f (wz))
(17)
We investigate the tokens with the highest increase in log probabilities under FV intervention and include few examples of the behavior we observe in Table 23. Here we show few examples of the tokens with the largest Alogprob for three tasks: Capita, Antonym, English-Frenn French.
Table 23:
The tokens with the highest increase in Alogprob for different queries three tasks Country-Capitall Antonym, and English-Free Frenchh (shown in black text). For comparison, we present the Alogprob of the top tokens we get when decoding the FV via ((e)) (shown in red text directly below). The Alogprob of the Dtok)stokens is much lower than the query-specific answers. In general, the tokens promoted the most correspond likely answers to the specific query, rather than generic tokens in the output space. In the case of 'wolf' for the English-French task, the model answers incorrectly, However, examining Alogprob indicates several likely answers are still promoted showing FVs have causal effects that are not adequately captured using top-1 accuracy.
Tokens with largest positive Alogproh under FVV intervention
Coantry-Capitare
South Africa
Pret" (+4.7). * Johannes" (+4.2). < "urr' (+4.0), 1 Caape' (+3.9). 1 Kinbber' (+3.7)
London' (+1.3), Moscow" (+1.1), Paris', , (+1.0), Bangkok' (+8.3) MMarrid' (+0.2)
Syria
Danascus' 9), Tart" (+4.2), Ragga' (+4. 1), Dan" (+4.0), Aleppo" (+3.8)
London' (+2.2), Moscow" (+2.1), Paris", (+2.1), Bangkok", (+1.3) Madrid" (+2.1)
Antonym
temporary
perpetual' (+4.4), definitive' (+4.3)," everlasting' (+4.1), pernanent' (+3.7)
counterpart! (+1.3), lesser' (+0.9), destroy' (+0.5), 'nggate' (+0.4), wwongg (-0.8)
static
evalving' (+4.0), flexible' (+3.8), polymorph' (+3.7), 'dnnaiic' (43.7)
counterpart' (+0.4), lesser' (-0. destroy' (0.2), negatee (-0.4), wrong' (-1.7)
Frencsh-Prench
wolf
lou' (+6.3), burs! (+6.2), chau' (45.8), (+5.8), Lau' (+5.7)
ag1 (-1.4), masc! (-0.6), eil (-0.9), mbbr' (+2.4), "" (+1.6)
advertisement
ann' (+7.8), aff' (+8.8), 'annon' (+6.7), ré' (+6.2), pub" (+6.14)
ag1- (0.2), masc' (1.3), ci1" (-9.2), embr' (-0.5), 6" (+0.6)
For the country-capital task, the tokens with the highest increase in log probability typically cor- respond with likely answers to the specific query, rather than answers to the task in general. For example, given the query 'South Africa', the country-capital FV promotes 'Pretoria', in addition to other cities in South Africa. We compare the 5 tokens with the highest overall increase in log probability, and the top 5 tokens we get from D(v:), which are shown below these in red. We see a substantial difference between the magnitudes of the Alogprob for these tokens and the tokens that were promoted the most.
We see a similar trend for Antonyms where the promoted tokens are all reasonably valid antonyms of the query word, rather than just antonyms in general.
For English-French, the query 'wolf' is not answered correctly under intervention, but the correct translation (1 loup") is still promoted by the FV when we examine the Alogprob. Similarly, for the query 'advertisement' the dataset target is 'publicité', but prefix tokens for another valid french translation (* annonce') )' are also promoted when examining the Alogprob distribution - "ann', and "annon".
51

---

Published as a conference paper at ICLR 2024
In conclusion, for the tasks examine find that function vectors have strong causal effects even when top-1 accuracy metric does not adequately capture this behavior. Furthermore, we find that the causal effects of the FV do not just generically promote words in the output space, but specific words that are plausible answers for each individual query.
52


Reviews:
[{"review_id": "O8JWPmJLZa", "reviewer": "Reviewer_9YEq", "summary": "The paper introduces the idea of function vectors, which appear to encode information about a task or transformation to be performed on a word to produce a response word under the task.  Function vectors are obtained by summing the averaged outputs of attention heads at the position of a test task input for heads that individually play relatively large roles across a set of tasks, where the average is obtained from several 10-shot prompts derived from examples of performance of the task.  Even though the attention vectors come from heads sprinkled across layers of a transformer, the resulting function vector can induce an increased tendency to give a task appropriate response when added to the hidden state of a transformer at many of the early layers of the transformer.  Experiments demonstrate robustness of the effects of FV's; show that FV's can sometimes be composed to cause a model to perform a composite task; and provide some evidence that the FV's are doing more than simply specifying the space of possible responses that are consistent with the task.", "questions": "Questions:\n\nThe text says 'We can then test the causal effect of an FV by adding it to hidden states at any layer ℓ as the model resolves a prompt and measuring its performance in executing the task.' I don't understand exactly how the vector is being 'added to hidden states at any layer'. Am I right that this vector is being added to the final state vector at the top of the lth transformer block?  How is this done given that this state vector typically has dimension equal to the number of heads times the dimensionality of a single head?\n\nHow was the 'direct decoding' of FV's performed?  What is the 'optimization' used 'to reconstruct a ˆvtk that matches the distribution Qtk when decoded'?\n\nHow was the approach used to determining the FV selected?  It would be informative to understand alternatives you might have tried.  What happens if the FV is the SET of averaged a_lj, with each inserted into the appropriate l,j position?  If this isn't better than 'adding the LV to the hidden state at layer l', why not?\n\nSuggestions:\n\nIt seems to me that a stronger analysis could arise from an examination of how the FV changes the distribution of logpobs of relevant words relative to the baseline prompt, and how this varies as the FV is inserted at different levels.  This approach could be used both for tasks where the set-intersection account could be sufficient, and for tasks where it cannot.\n\nThe paper would be strengthened if the authors could provide some results from carefully constructed test sets in which the set of outputs is the same set as the set of inputs (e.g matched antonym and synonym pairs, months, days of week, letters of the alphabet, progression of planets, chemicals in the periodic table) afford possibilities.\n\nIt might be even more useful to understand tasks in which the task response depends strictly on the words in the context, and we are assured that these words did not enter in to the construction of the FV.  For example, a set of n10-shot prompts involving a random subset chosen from a larger set of words could be used to construct FVs for the m 'chose pth word from list of length m' tasks. Then a test set of word lists of length m could be formed from a sample from the remaining words in the larger word set, and these could be factorially combined with the FV's for each of the values of p in m.  Clearly in this case, the FV's cannot themselves activate the set of answer responses -- if anything they'd activate words in the lists used to form the FV's.  So they must be *inducing* later heads to attend to the correct p in the given list in this case.\n\nConstructing test sets in which x and y come from the same set of words could be helpful.  Antonym and synonym tasks could address this if so constructed, but little detail was provided about the sets of x and y items used, and I did not see a description of explicitly reversed stimulus-response pairs.  (It is possible, for example, that the antonyms tended to be the negative member or less advantaged member of a pair, as the list of decoded words suggests).  I recognize that the full set of tasks include some that make this possibility unlikely (specifically, the chose-item-in-a-specific-list position), since (although not described) it seems likely that x is something uninformative like 'answer'.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides an example of an way to probe transformer function that adds to the toolkit for causal analysis of how transformers perform tasks.\n\nThe findings are suggestive of aspects of the way in which in context examples give rise to task performance.  In particular, it suggests that a relatively small number of attention head play a relatively crucial role in inducing a transformer to produce any one of many different transformation tasks.  It is also interesting that the FV can be added into just one layer out of ~30-80 layers of a deep transformer and have a strong effect on its tendency to perform the given task.  It seems to have implications for how we understand the state-to-state transitions across layers in the network -- ie that they are in someways roughly interchangeable at least across many layers.\n\nThe paper uses a diversity of tasks, model types, and base prompts to assess the effects of FV's.  Despite considerable variability in exact magnitudes of effects, the basic effect holds up across all of these manipulations.", "weaknesses": "The paper is relatively phenomenological rather than mechanistic; we see that there effects, but there is relatively little analysis of how they actually occur.\n\nMore specifically, I thought that the mechanism of action of the FV's could be more forcefully addressed.  At least for some tasks, it is possible that the FV's work by simply specifying the set of responses consistent with the task. E.g. Country-Capital a sufficient mechanism would be the following set-intersection account:  An input country name (France) specifies a set of words related to France; the FV specifies the set of country names; and the correct response is simply their intersection. I recognize that this would not work for all tasks, but it would word for many.  The paper could be strengthened by a detailed analysis of a selected instances of both kinds of tasks.  Some of the tasks specifically appear to require that the FV, if effective, specifies not just how to transform the input, but how to select a response from a string of preceding elements.  None of the 6 tasks selected were of this type, but there were such tasks in the full set.  Highlighting and analyzing these would strengthen the paper. I discuss this further in some suggestions below.\n\nThere were also some details that I would like greater clarity about.  I list these in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Qv7kx00La", "reviewer": "Reviewer_cNFK", "summary": "This paper works on the analysis of transformer language models. It investigates the information of attention heads and identifies a small number of heads to construct the proposed Function Vector that contains the most causal information. To evaluate the ability of the Function Vector, the paper constructs a set of composable ICL tasks and shows that such a vector not only has some semantic abilities like word embeddings but also has some calculation abilities.", "questions": "Please check the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper does some deep investigation into transformer language models, especially large ones, which can further help us better understand how they work.\n* The structure of the paper is clear and is to follow. Figures in the paper are drawn well.\n* Experiments are done on different model sizes and model structures. The paper shows great performance improvement of FV compared to Layer Avg.", "weaknesses": "* Though the paper mainly investigates attention heads, I’m still wondering why attention heads contain such information rather than the MLP layers. I find some papers [1], and [2] seem to find that MLP is more critical from the information or memory aspect. Can the paper discuss more on this part, which would better explain the proposed findings?\n\n  [1]. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space\n\n  [2]. MASS-EDITING MEMORY IN A TRANSFORMER\n\n* I think the step from identifying useful attention heads to the proposed FV vector sounds empirical. For example, can the author explain how to determine the number of selected heads? I find that the paper uses |A| = 10 attention heads for GPT-J and scales the number for larger models. However, this point including the scaling and the initial 10 heads for GPT-J needs more explanation. Also, can the paper give more description about directly summing the head outputs together as defined in Eq. (5)? Has the paper compared with other ways like summing them up with different weights?\n\n* For experiments, I find though the paper tested 40 tasks, they are very simple. Thus, can the paper think of some more difficult ones? Meanwhile, I think if the paper can add some analyses about the FV ability under model scaling laws, it can be better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KEzB3wut4u", "reviewer": "Reviewer_ZAgB", "summary": "This paper studies the way that tasks are encoded in LLMs, and find that there are a small set of attention heads which can be used to transport the task from in-context learning to unrelated prompts. They refer to these as \"function vectors\", and they find that these vectors can be used to encourage the model to perform tasks in a zero shot manner. The results support that FVs can be used in a variety of contexts to invoke the task.", "questions": "Besides the questions above:\n\nIn the intro, I'm not sure a lot of readers will understand the connections to lambda calculus, so I wonder if there's a better way to start of this work.\n\nIn 2.2 J is mentioned but not defined: Number of heads?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is a strong contribution towards understanding LLMs in terms of their ability to both encode and respond to ICL task information. The paper is fairly easy to follow: the motivation is clear as LLMs are somewhat of a black box, and I believe this paper has framed their study in the context of something easy to digest (ICL tasks) with clear experimentation. This work has significance as I believe that understanding some of the mechanisms of LLMs and possibly manipulated them for ZS performance will be useful in a number of fields that use LLMs. The approach is fairly creative yet intuitive: I appreciate the way they studied which heads have the most casual influence on tasks, and didn't have any issues understanding the motivations behind how they derive FVs.", "weaknesses": "There are a few things I'm a bit unclear about, notably that there is some back and forth on using all heads versus only manipulating the last head (am I understanding that correctly?). Notably, FVs appear to be a vector over tasks: can you describe again how these are applied to the LLM to invoke task behavior? Do you mean that we are only looking at the attention heads to the last token (last column)? Then you sum over these heads for each task and add it to the last hidden state?\n\nPerhaps it would be best to have a diagram, but it seems like some other choices on how to transport the task information to unrelated contexts would have been possible. Could we have not just added the task value (without summing in equation 5) to each attention head leading to computing the next hidden state?\n\nAlso, why compute the top attention heads over all tasks and not just have a different distinct set for each task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyAnknWDAn", "reviewer": "Reviewer_baEP", "summary": "The authors devise a simple way to extract compact task representations from LLMs. They further investigate how well these task representations generalize and whether these representations can be meaningfully composed to create new task representations (i.e. whether vector-algebraic operations on them are meaningful).", "questions": "I might have missed it, but I wonder if the authors could provide a metric for the \"proportion of task representation encoded by FVs\". One potential way to visualize that would be to add lines to figure 4 representing best ICL performance, similarly for table 2, a few-shot baseline column would be very helpful. In other words, I do not fully understand how much task information we lose when we move from ICL to FVs.\n\nA minor suggestion on presentation:\nI feel that figures 1 and 2 are introduced much too early. They are confusing without proper context, which is given much later.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a highly relevant topic.\n- Experimental support is sufficient.\n- The findings are likely to be of broad interest.\n- The paper is clearly written and is a pleasure to read.", "weaknesses": "- I feel that the contribution is not sufficiently clearly placing itself in the context of existing work. In particular, the \"related work\" section seems brief and insufficient.\n\nThere is extensive work on distributed, composable and generalizeable task representations (e.g. Lampinen, A. K., & McClelland, J. L. (2020). Transforming task representations to perform novel tasks. Proceedings of the National Academy of Sciences, 117(52), 32970-32981, but see related work also).\n\nI believe that the impact of the present contribution on our understanding of vectorized task representations (both in LLMs and more generally) is not discussed in sufficient depth. It also seems important to broaden the discussion a little, to include not only work on task representations, but also discuss works that looked into the roles of different attention heads in general (see e.g. Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341.). I.e. why they overlooked/were unable to interpret attention heads as encoding specific tasks. Is the proposed probing method superior, or is it because nobody cared to look, etc.\n\n- As a related issue, I think it is crucial for the authors to explain why the results are important. Currently, the authors say \"Our study of function vectors differs qualitatively from these previous works: rather than training a model to create function representations, we ask whether a pretrained transformer already contains compact function representations that it uses to invoke task execution\". In a way, I feel that this is underselling their work.\n\nIt is clear without doing any research that pre-trained LLMs must have some form of task representations (otherwise they won't be able to do zero/few-shot learning). I think that developing a way to extract those representations is valuable, but a more thorough/careful discussion is crucial to highlight why this is important and how it advances existing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)