Title: Language Models Implement Simple Word2Vec-style Vector Arithmetic
Venue: NeurIPS 2023
Accepted: no
Keywords: interpretability

nlp

neural networks

deep learning

explainability

representation learning
Abstract: A primary criticism towards language models (LMs) is their inscrutability. This paper presents evidence that, despite their size and complexity, LMs sometimes exploit a computational mechanism familiar from traditional word embeddings: the use of simple vector arithmetic in order to encode abstract relations (e.g., Poland:Warsaw::China:Beijing). We investigate a range of language model sizes (from 124M parameters to 176B parameters) in an in-context learning setting, and find that for a variety of tasks (involving capital cities, upper-casing, and past-tensing), a key part of the mechanism reduces to a simple linear update applied by the feedforward networks. We further show that this mechanism is specific to tasks that require retrieval from pretraining memory, rather than retrieval from local context. Our results contribute to a growing body of work on the mechanistic interpretability of LLMs, and offer reason to be optimistic that, despite the massive and non-linear nature of the models, the strategies they ultimately use to solve tasks can sometimes reduce to familiar and even intuitive algorithms.

Abstract

A primary criticism towards language models (LMs) is their inscrutability. This paper presents evidence that, despite their size and complexity, LMs sometimes exploit a computational mechanism familiar from traditional word embeddings: the use of simple vector arithmetic in order to encode abstract relations (e.g., Poland:Warsaw::China:Beijing ). We investigate a range of language model sizes (from 124M parameters to 176B parameters) in an in-context learning setting, and find that for a variety of tasks (involving capital cities, upper-casing, and pasttensing), a key part of the mechanism reduces to a simple linear update applied by the feedforward networks. We further show that this mechanism is specific to tasks that require retrieval from pretraining memory, rather than retrieval from local context. Our results contribute to a growing body of work on the mechanistic interpretability of LLMs, and offer reason to be optimistic that, despite the massive and non-linear nature of the models, the strategies they ultimately use to solve tasks can sometimes reduce to familiar and even intuitive algorithms.

1 Intro

The growing capabilities of large language models (LLMs) have led to an equally growing interest in understanding how such models work under the hood. Such understanding is critical for ensuring that LLMs are reliable and trustworthy once deployed. Recent work (often now referred to as 'mechanistic interpretability') has contributed to this understanding by reverse-engineering the data structures and algorithms that are implicitly encoded in the model's weights, e.g., by identifying detailed circuits [Wang et al., 2022, Elhage et al., 2021, Olsson et al., 2022] or by identifying mechanisms for factual storage and retrieval which support intervention and editing [Geva et al., 2021b, Li et al., 2022, Meng et al., 2022a,c, Dai et al., 2022].

Here, we contribute to this growing body of work by analyzing how LLMs recall information during in-context learning. Specifically, we observe that the mechanism that LLMs use in order to retrieve certain facts (e.g., mapping a country to its capital city) bears a striking resemblance to the type of vector arithmetic operations associated with LLMs' simpler, static word-embedding predecessors. That is, early word embeddings such as word2vec [Mikolov et al., 2013] famously supported factual recall via linear vector arithmetic-e.g., there existed some vector that, when added to the vector for any country would produce the vector for its capital. Modern LLMs are based on a complex transformer architecture [Vaswani et al., 2017] which produces contextualized word embeddings [Peters et al., 2018, Devlin et al., 2019] connected via multiple non-linearities. Despite this, we find that LLMs implement a very similar vector-addition mechanism which plays an important role in a number of in-context-learning tasks.

We study this phenomenon in three tasks-involving recalling capital cities, uppercasing tokens, and 35 past-tensing verbs. Our key findings are: 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

We find evidence of a distinct processing signature in the forward pass which characterizes this mechanism. That is, if models need to perform the get_capital( x ) function, which takes an argument x and yields an answer y , they must first surface the argument x in earlier layers which enables them to apply the function and yield y as the final output (Figure 2). This signature generalizes across models and tasks, but appears to become sharper as models increase in size.

We take a closer look at GPT2-Medium, and find that the vector arithmetic mechanism is implemented by mid-to-late layer feedforward networks (FFNs) in a way that is modular and supports intervention . That is, FFNs construct a vector that is not specific to context or argument, such that the same vector which produces Warsaw given Poland in one context can be dropped into an unrelated context to produce Beijing given China .

We demonstrate that this mechanism is specific to recalling information from pretraining memory . For settings in which the correct answer can be retrieved from the prompt, this mechanism does not appear to play any role, and FFNs can be ablated entirely with relatively minimal performance degradation. Thus, we present new evidence supporting the claim that FFNs and attention specialize for different roles, with FFNs supporting factual recall and attention copying and pasting from local context.

Taken together, our results offer new insights about one component of the complex algorithms that underlie in-context learning. The simplicity of the mechanism, in itself surprising, raises the possibility that other apparently complicated behaviors may be supported by a sequence of simple operations under the hood. Moreover, our results suggest a distinct processing signature and hint at a method for intervention. These ideas could support future work on detecting and preventing unwanted behavior by LLMs at runtime.

2 Methods

In decoder-only transformer language models [Vaswani et al., 2017], a sentence is processed one word at a time, from left to right. The token at the current timestep is passed into the input of the model in order to predict the next, and so on. In this paper, we focus on the transformations that the next-token prediction undergoes in order to predict the next word. At each layer, an attention module and feed-forward network (FFN) module apply subsequent updates to this representation. Consider the FFN update at layer i , where x i is the current next-token representation. The update applied by the FFN here is calculated as FFN ( /vector x i ) = /vector o i , /vector x i +1 = /vector x i + /vector o i where /vector x i +1 is the updated token for the next layer. Note that due to the residual conncetion , the output vector /vector o i is added to the input. /vector x is updated this way by the attention and FFNs until the end of the model, where the token is decoded into the vocab space with the language modeling head E : softmax ( E /vector x ) . From start to end, x is only updated by additive updates, and because of this, is said to form a residual stream [Elhage et al., 2021]. Thus, the token representation x i represents all of the additions made into the residual stream up to layer i . Because of the residual connection, Attention and FFN blocks can be viewed as reading from the residual stream and adding their outputs back into it

The unembedding

matrix projects into

the vocabulary space.

Attention

FFN

LM

Head

Embed

FFN

LM

Head

+a18

+ofunc

+o18

Attention

+a19

…

LM

Head

LM

Head

LM

Head

…

Selected by example

2.1 Early Decoding 74

75

76

77

78

79

80

81

82

83

84

85

86

A key insight from the residual stream perspective is that we can decode the next token prediction with the LM head before it reaches the final layer. This effectively allows for 'print statements' throughout the model's processing. The intuition behind this technique is that LMs incrementally update the token representation /vector x to build and refine an encoding of the vocabulary distribution. This technique was initially introduced in nostalgebraist [2020] as the logit lens, and Geva et al. [2022b] show that LMs do in fact refine the output distribution over the course of the model. Figure 1 illustrates the process we use to decode hidden states into the vocabulary space, in which the hidden state at each layer is decoded with the pre-trained language modeling head E . After decoding into the vocabulary space, we apply a softmax to get a probability distribution over all tokens. When we decode at some layer, we say that the most likely token in the resulting vocab distribution is currently being represented in the residual stream. We examine several in-context learning tasks to understand how the answers to these problems are discovered by a model over the course of the forward pass.

87

88

89

90

91

92

93

94

2.2 Tasks

Can we understand the subprocesses underlying how LMs solve simple problems? We apply early decoding to suite of in-context learning tasks to explore the transformations the next token prediction undergoes in order to predict the answer.

World Capitals Our World Capitals task requires the model to retrieve the capital city for various states and countries in a few-shot setting. The dataset we use contains 248 countries and territories. A one-shot example is shown below:

Reasoning about Colored Objects We focus on a subset of 200 of the reasoning about 95 colored objects dataset prompts (henceforth, the colored objects dataset) from BIG-Bench 96 [Srivastava et al., 2022], which gives the model a list of colored common objects and require 97 to simply state the color of a query object. For the purposes of this paper, we focus only 98 on one aspect of this task-the model's ability to output the final answer in the correct format. 1 99

'Q: On the floor, I see a silver keychain, a red pair of sunglasses, a gold sheet of paper, a black dog leash, and a blue cat toy. What color is the keychain?

100

101

102

103

104

105

106

107

108

109

110

111

112

A: Silver

Q: On the table, you see a brown sheet of paper, a red fidget spinner, a blue pair of sunglasses, a teal dog leash, and a gold cup. What color is the sheet of paper?

A:___" Expected answer: ' Brown"

Past Tense Verb Mapping Lastly, we examine whether a language model can accurately recognize a pattern and predict the past tense form of a verb given its present tense. The dataset used is the combination of the regular and irregular partitions of the past tense linguistic mapping task in BIG-Bench [Srivastava et al., 2022]. After filtering verbs in which the present and past tense forms start with the same token, we have a total of 1,567 verbs. An example one-shot example is given below:

'Today I abandon. Yesterday I abandoned. Today I abolish. Yesterday I___" Expected answer: ' abolished"

2.3 Models

We experiment exclusively on decoder-only transformer LMs across various sizes and pre-training corpora. When not specified, results in figures are from GPT2-medium. We also include results portraying the stages of processing signatures in the residual streams of the small, large, and extra large variants [Radford et al.], the 6B parameter GPT-J model [Wang and Komatsuzaki, 2021], and the 176B BLOOM model [Scao et al., 2022], either in the main paper or in the Appendix.

3 Stages of Processing in Predicting the Next Token 113

Layer

0

(

1

A

2

A

3

A

4

A

5

A

6

No

7

C

8

A

9

A

10

A

11

A

12

Unknown

13

C

14

St

15

Poland

16

Poland

17

Poland

18

Poland

19

Warsaw

20

Warsaw

21

Warsaw

22

Warsaw

23

Warsaw

A

B

C

114

115

116

117

118

119

120

121

122

123

124

125

126

127

Q:What is the capital of France?

A:Paris

Q:What is the capital of Poland?

A:

1.0

Warsaw

Rank

0.8

Reciprocal

0.6

0.4

0.2

0.0

D

10111213141516

20

2

2

2

Layer

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

Layer

18

19

20

white

Layer Top Token 0 ( 1 A 2 A 3 A 4 A 5 A 6 A 7 A 8 A 9 The 10 The 11 The 12 The 13 The 14 The 15 The First, we use the early decoding method in order to investigate how the processing proceeds over the course of a forward pass to the model. Each task requires the model to infer some relation to recall some fact, e.g., retrieving the capital of Poland. In these experiments, we see several discrete stages of processing that the next token undergoes before reaching the final answer. These states together provide evidence that the models "apply" the relevant functions (e.g., get_capital ) abruptly at some mid-late layer to retrieve the answer. Moreover, in these cases, the model prepares the argument to this function in the layers prior to that in which the function is applied.

16

The

17 The 18 Poland 19 Poland 20 Poland 21 Poland 22 Poland 23 The Table 1: These are the top tokens per layer in GPT2Medium on the example zero-shot Poland example In Figure 2 we illustrate an example of the stages we observe across models. For the first several layers, we see no movement on the words of interest. Then, during Argument Formation , the model first represents the argument to the desired relation in the residual stream. This means that the top token in the vocabulary distribution at some intermediate layer(s) is the subject

21 Brown 22 Brown 23 Brown 10 the question inquires about (e.g., the x , in get_capital( x ) ). During Function Application we find that the model abruptly switches from the argument to the output of the function (the y , in get_capital( x ) = y ). We find that function application is typically applied by the FFN update at that layer to the residual stream. This is done by adding the output vector /vector o of the FFN to the residual stream representation, thus transforming it with an additive update. We study these /vector o vectors in detail in Section 4. Finally, the model enters Saturation 2 , where the model recognizes it has solved the next token, and ceases updating the token representation for the remaining layers.

The trend can be characterized by an X-shaped pattern of the argument and final output tokens 146 when plotting the ranks of the argument( x ) and output ( y ) tokens. We refer to this behavior as 147 argument-function processing. Figure 3 shows that this same processing signature can be observed 148 consistently across tasks and models. Moreover, it appears to become more prominent as the models 149 increase in size. Interestingly, despite large differences in number of layers and overall size, models 150 tend to undergo this process at similar points proportionally in the model. 151

152

4 Implementation of Context-Independent Functions in FFN Updates

The above results on processing signature suggest that the models 'apply' a function about 2/3rds of 153 the way through the network with the addition of an FFN update. Here, we investigate the mechanism 154 via which that function is applied more closely. Specifically, focusing on GPT2-Medium 3 , we show 155 that we can force the encoded function to be applied to new arguments in new contexts by isolating 156 the responsible FFN output vector and then dropping into a forward pass on a new input. 157

4.1 /vector o Vector Interventions 158

Consider the example in Figure 2. At layer 18, the residual stream ( /vector x 18 ) is in 159 argument formation, and represents the ' Poland" token. At the end of layer 160 19, a function is applied, transforming /vector x 19 into the answer token ' Warsaw. 161

176

162

163

164

165

166

167

168

169

170

171

172

173

174

175

tablemugfreeChinatablemugfreeChinatablemugfree

1.0

Beijing:OriginalFFN

Beijing:FFN= + Ocity

0.8

Reciprocal Rank

0.6

0.4

0.2

0.0

1234

56

下

LO

10 11 12 13 14 15 101718192022

Layer

2

2

177

178

179

180

181

182

183

184

185

As discussed in the previous section, we can isolate the function application in this case to FFN 19; let ˜ x 19 represent the residual stream after the attention update, but before the FFN update at layer 19 (which still represents Poland). Recall that the update made by FFN 19 is written FFN 19 ( ˜ x 19 ) = /vector o 19 and /vector x 19 = ˜ x 19 + /vector o 19 . We find that /vector o 19 will apply the get_capital function regardless of the content of ˜ x 19 . For example, if we add /vector o 19 to some ˜ x which represents the ' China" token, it will transform into ' Beijing". Thus we refer to /vector o 19 as /vector o city since it retrieves the capital cities of locations stored in the residual stream. We locate such /vector o vectors in the uppercasing and past tense mapping tasks in the examples given in Section 2.2, which we refer to as /vector o upper and /vector o past , respectively. 4

We test whether these updates have the same effect, and thus implement the same function, as they do in the original contexts from which they were extracted, which would imply a systematic structure in the internal embedding space the LM leverages. To do so, we replace entire FFN

layers with these vectors and run new inputs through the intervened model. 5

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

Data: We are interested in whether the captured o vectors can be applied in a novel context, in particular, to a context that is otherwise devoid of cues as to the function of interest. Thus, we synthesize a new dataset where each entry is a string of three random tokens (with leading spaces) followed by a token x which represents a potential argument to the function of interest. For example, in experiments involving o city , we might include a sequence such as table mug free China table mug free China table mug free . This input primes the model to produce 'China' at the top of the residual stream, but provides no cues that the capital city is relevant, and thus allows us to isolate the effect of o city in promoting 'Beijing' in the residual stream. In addition to the original categories, we also include an 'out-of-domain' dataset for each task: US states and capitals, 100 non-color words, and 128 irregular verbs. These additional data test the sensitivity of the /vector o vectors to different types of arguments.

Results: Figure 4 shows results for a single example. Here, we see that 'Beijing' is promoted all the way to the top of the distribution solely due to the injection of /vector o city into the forward pass. Figure 5 shows that this pattern holds in aggregate. In all settings, we see that the outputs of the intended functions are strongly promoted by adding the corresponding /vector o vectors. By the last layer, for world and state capitals, the mean reciprocal rank of the target city name across all examples improves from roughly the 10th to the 4th-highest ranked word and 20th and 3rd-ranked words respectively.

We also see the promotion of the proper past tense verbs by /vector o past . The reciprocal ranks improve similarly for both regular (approx. 7th to 3rd rank) and irregular verbs (approx. 6th to 3rd), indicating that the relationship between tenses is encoded similarly by the model for these two types. /vector o upper promotes the capitalized version of the test token almost every time, although the target word starts at a higher rank (on average, rank 5). These results together show that regardless of the surrounding context, and regardless of the argument to which it is applied, /vector o vectors consistently apply the expected functions. Since each vector was originally extracted from the model's processing of a single naturalistic input, this generalizability suggests significant structure and cross-context abstraction within the learned embedding-space .

Common Errors: While the above trend clearly holds on the aggregate, the intervention is not perfect for individual cases. The most common error is that the intervention has no real effect. In the in-domain (out-domain) settings, this occurred in about 37% (20%) of capital cities, 4% (5%) on uppercasing, and 19% (22%) for past tensing. We believe the rate is so much higher for world capitals because the model did not have a strong association between certain country-capital pairs from pretraining, e.g, for less frequently mentioned countries. Typically, in these cases, the top token remains the argument, but sometimes becomes some random other city, for example, predicting the capital of Armenia is Vienna. We also find that the way tokenization splits the argument and target

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

Abstractive Task

Extractive Task

Layer

Q: What is the capital

of Somalia?

A: Mogadishu

Q: What is the capital

of Poland?

A:

The capital of

Somalia is Mogadishu.

The capital of Poland

is Warsaw.

Q: What is the capital

of Somalia?

A: Mogadishu

Q: What is the capital

of Poland?

A:

...

...

...

14

St

St

15

Poland

St

16

Poland

Warsaw

17

Poland

Warsaw

18

Poland

Warsaw

19

Warsaw

Warsaw

20

Warsaw

Warsaw

21

Warsaw

Warsaw

22

Warsaw

Warsaw

23

Warsaw

Warsaw

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

Figure 6: The abstractive task undergoes argument formation (blue) and function application (red), while the extractive task immediately saturates (yellow).

words affects the ability of the /vector o vector to work and is another source of errors. This is discussed further in Appendix E.

5 The Role of FFNs in Out-of-Context Retrieval

So far, we have shown that FFN output vectors can encode functions that transfer across contexts. Here we investigate whether the mechanism we identify applies in general to associations of this type, or rather if such functionality can be implemented by the attention mechanism instead. Shared among the tasks we study is the requirement to recall a token that does not appear in the given context (abstractive tasks). In this section we show that mid-higher layer FFNs are crucial for this process. When the answer to the question does appear in context (extractive tasks), we find that ablating a subset of FFNs has a comparatively minor effect on performance, indicating that they are relatively modular and there is a learned division of labor within the model. This observation holds across the decoder-only LMs tested in this paper, but is particularly salient in the larger/deeper networks. This breakdown is consistent with previous work finding that FFNs store facts learned from pre-training [Geva et al., 2021a, Meng et al., 2022b,c] and attention heads copy from the previous context [Wang et al., Olsson et al., 2022].

5.1 Abstractive vs. Extractive Tasks

Extractive Tasks: Extractive tasks are those in which the exact tokens required to answer a prompt can be found in the input context. These tasks can thus be solved by parsing the local context alone, and thus do not necessarily require the model to apply a function of the type we have focused on in this paper (e.g., a function like get_capital ).

Abstractive Tasks: Are those in which the answer to a prompt is not given in the input context and must be retrieved from pretraining memory. Our results suggest this is done primarily through argument-function processing, requiring function application through (typically) FFN updates as described in Section 3.

We provide examples with their associated GPT2-Medium layerwise decodings in Figure 6. We expect that the argument formation and function application stages of processing occur primarily in abstractive tasks. Indeed, in Appendix A, we show that the characteristic argument-answer X pattern disappears on extractive inputs. We hypothesize that applying out-of-context transformations to the predicted token representation is one of the primary functions of FFNs in the mid-to-late layers, and that removing them should only have a major effect on tasks that require out-of-context retrieval.

5.2 Effect of Ablating FFNs

Data: Consider the example shown in Section 2.2 demonstrating the /vector o upper function. By providing the answer to the in-context example as ' Silver", we make the task abstractive by requiring the in-context token ' brown" to be transformed to ' Brown" in the test example. However, if we provide the in-context label as ' silver", the task becomes extractive, as the expected answer becomes ' brown". We create an extractive version of this dataset by lowercasing the example answer. All data is presented to the model with a single example (one-shot). Notice that the abstractive and extractive examples only differ by a single character and are thus minimally different.

We repeat this experiment on the world capitals task by adding the prefix 'The capital of A is B. The capital of C is D" to each input. Notice, however, that since the answer is provided explicitly, the task is much easier for the models in the extractive case.

Procedure: We run the one-shot extractive and abstractive datasets on the full models, and then 270 repeatedly remove an additional 1/6th of all FFNs from the top down (e.g., in 24 layer GPT2-Medium: 271 removing the 20-24th FFNs, then the 15-24th, etc.). 272

273

274

275

276

277

278

279

280

281

282

283

284

Results: Our results are shown in Figure 7. Despite the fact that the inputs in the abstractive and extractive datasets only slightly differ (by a single character in the colored objects case) we find that performance plummets on the abstractive task as FFNs are ablated, while accuracy on the extractive task drops much more slowly. For example, even after 24 FFN sublayers are removed from Bloom (totaling 39B parameters) extractive task accuracy for the colored objects dataset drops 17% from the full model's performance, while abstractive accuracy drops 73% (down to 1% accuracy). The case is similar across model sizes and pretraining corpora; we include results on additional models in Appendix B. This indicates that we can isolate the effect of locating and retrieving out of context tokens in this setting to the FFNs. Additionally, because the model retains reasonably strong performance compared to using the full model, we do not find convincing evidence that the later layer FFNs are contributing to the extractive task performance, supporting the idea of modularity within the network.

285

6 Related Work

Recent work has contributed to understanding language models by studying the role of different 286 modules in the transformer architecture in language modeling. In particular, the attention layers 287 [Olsson et al., 2022, Kobayashi et al., 2020, Wang et al.] and more notably for this work, the FFN 288 modules, which are frequently associated with factual recall and knowledge storage [Geva et al., 289 2021a, Meng et al., 2022a,c]. Although how language models store and use knowledge has been 290 studied more generally as well [Petroni et al., 2019, Cao et al., 2021, Dai et al., 2022, Bouraoui et al., 291 2019, Burns et al., 2022, Dalvi et al., 2022, Da et al., 2021] as well as in static embeddings [Dufter 292 et al., 2021]. Recent work in mechanistic interpretability aims to fully reverse engineer how LMs 293 perform some behaviors. Our work builds on the finding that FFN layers promote concepts in the 294 vocabulary space [Geva et al., 2022a] by breaking down the process the model uses to do this in 295 context. Bansal et al. [2022] perform ablation studies to test the importance of attention and FFN 296 layers on in-context learning tasks, here we offer an explanation for their role in some cases. Other 297 work analyze information flow within an LM to study how representations are built through the layers 298 [Voita et al., 2019, Tenney et al., 2019] and show distinct points of processing in the model. We also 299 follow this approach, but our analysis focuses on interpreting how models use individual updates 300

within the forward pass, rather than probing for what information is encoded and potentially used to 301 make predictions. Ilharco et al. [2023] show that vector arithmetic can be performed with the weights 302 of finetuned models to compose tasks, similar to how /vector o vectors can induce functions in the activation 303 space of the model. 304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

7 Discussion

In this work, we describe a mechanism that is partially responsible for LMs ability to recall factual associations. We conceptualize these recalls as the application of some function (e.g., get_capital( x ) = y and find that the next-token prediction goes through several discrete stages of processing in which the prediction first represents the argument x (e.g., Poland) before applying that function with an additive update to get the final answer y (Warsaw). A core challenge in interpreting neural networks is determining whether the information attributed to certain model components is actually used for that purpose during inference [Hase and Bansal, 2022, Leavitt and Morcos, 2020]. While previous work has implicated FFNs in recalling factual associations [Geva et al., 2022a, Meng et al., 2022a], we show through intervention experiments that we can manipulate the information flowing through the model during these stages. Specifically, we show that it is possible to capture the output vector of an FFN from a single forward pass on a single in-context learning example, and that the captured vector can be used to apply the same function to new arguments (e.g., other countries) in totally different contexts. This process provides a surprisingly simple explanation for the internal subprocesses used by LMs to recall factual associations and resembles vector arithmetic observed in static word embeddings. Our findings invite future work aimed at understanding why, and under what conditions, LMs learn to use this mechanism when they are capable of solving such tasks using, e.g., adhoc memorization.

A limitation that we observe is that the process for carrying out the /vector o intervention depends on hyperparameters which are often model-specific (i.e., the exact stimuli used to extract the intervention, and the layer(s) at which to perform the intervention). We provide our most detailed investigation on GPT2-Medium, which clearly illustrates the phenomenon. Our experiments on stages of processing with GPT-J suggest that the same phenomena is in play, although (as discussed in Section 4 and Appendix A), the procedures we derive for interventions on GPT2-Medium do not transfer perfectly. Specifically, we can strongly reproduce the intervention results on uppercasing for GPT-J; results on the other two tasks are positive but with overall weaker effects. This requirement of model-specific customization is common in similar mechanistic interpretability work, e.g., [Meng et al., 2022a, Wang et al., 2022, Geva et al., 2022b], and a prioritiy in future work must be to identify common patterns across these individual studies which reduce the need to repeat such effort on each new model. That said, in this work and other similar efforts, a single positive example as a proof of concept is often sufficient to advance understanding and spur future work that improves robustness across models.

In the long term, findings like those presented here have implications for improving the trustworthiness of LMs in production. If we can understand how models break down complex problems into simple and predictable subprocesses, we can help more readily audit their behavior. Interpreting the processing signatures of model behaviors might offer an avenue via which to audit and intervene at runtime in order to prevent unwanted behavior. Moreover, understanding which relations FFNs encode could aid work in fact location and editing. Contemporaneous work [Geva et al., 2023] has studied a different mechanism for factual recall in LMs, but it is unclear how and when these mechanisms interact.

8 Conclusion

We contribute to a growing body of work on interpreting how the internal processes of language 345 models (LMs) produce some behavior. On three in-context learning tasks, we observe that the next346 token prediction appears to undergo several stages of processing in which LMs represent arguments 347 to functions in their residual streams. This process occurs in models ranging in size from 124M to 348 176B parameters. On GPT2, We study instances where the additive update is made by the output 349 vectors ( /vector o vectors) of feed-forward networks (FFNs). We show that for all tasks we test, /vector o vectors 350 calculated by the model in the process of solving some task can be extracted and replace the FFN 351 updates of the model to solve novel instances of that task, providing evidence that LMs can learn 352 self-contained and context-independent functions from pretraining. 353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

References

Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale, December 2022. URL http://arxiv.org/abs/2212.09095 . arXiv:2212.09095 [cs].

Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. Inducing Relational Knowledge from BERT, November 2019. URL https://arxiv.org/abs/1911.12753v1 .

Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision, December 2022. URL http://arxiv.org/abs/ 2212.03827 . arXiv:2212.03827 [cs].

Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1860-1874, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021.acl-long.146. URL https://aclanthology.org/2021.acl-long.146 .

Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. Analyzing Commonsense Emergence in Few-shot Knowledge Models. September 2021. URL https://openreview. net/forum?id=StHCELh9PVE .

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8493-8502, 2022.

Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering Latent Concepts Learned in BERT, May 2022. URL http://arxiv.org/abs/2205. 07237 . arXiv:2205.07237 [cs].

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https: //aclanthology.org/N19-1423 .

Philipp Dufter, Nora Kassner, and Hinrich Schütze. Static Embeddings as Efficient Knowledge Bases? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2353-2363, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.186. URL https://aclanthology.org/2021.naacl-main.186 .

N Elhage, N Nanda, C Olsson, T Henighan, N Joseph, B Mann, A Askell, Y Bai, A Chen, T Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, 2021a.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.446. URL https://aclanthology.org/2021.emnlp-main.446 .

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprint arXiv:2203.14680 , 2022a.

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space, October 2022b. URL http: //arxiv.org/abs/2203.14680 . arXiv:2203.14680 [cs].

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023.

Peter Hase and Mohit Bansal. When can models learn from explanations? a formal framework for understanding the roles of explanation data. In Proceedings of the First Workshop on Learning with Natural Language Supervision , pages 29-39, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.lnls-1.4. URL https://aclanthology. org/2022.lnls-1.4 .

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ICLR , 2023.

Stanis ł aw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations , 2017.

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7057-7075, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.574. URL https: //aclanthology.org/2020.emnlp-main.574 .

Matthew L Leavitt and Ari Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010.12016 , 2020.

Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13382 , 2022.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT, October 2022a. URL http://arxiv.org/abs/2202.05262 . arXiv:2202.05262 [cs] version: 4.

Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems , 2022b.

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. arXiv preprint arXiv:2210.07229 , 2022c.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

nostalgebraist. interpreting GPT: the logit lens. 2020. URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens .

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1202. URL https://aclanthology.org/N18-1202 .

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1250. URL https://aclanthology.org/D19-1250 .

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176bparameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4593-4601, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Elena Voita, Rico Sennrich, and Ivan Titov. The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4396-4406, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1448. URL https://aclanthology.org/D19-1448 .

Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small, November 2022. arXiv:2211.00593 [cs].

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In NeurIPS ML Safety Workshop .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents evidence that LMs sometimes use а computational mechanism similar to traditional word embeddings, specifically using simple vector arithmetic to encode abstract relations. Experiments show that this mechanism is specific to tasks that require retrieval from pretraining memory rather than local context. In sum, this paper sheds light on the inner workings of LMs and provides insights into their interpretability.\n\nSTRENGTHS:\n- The paper is well-written, easy to follow, and is a pleasure to read.\n\n- It is appreciated that the paper use LMs of a range of sizes to investigate the arithmetic mechanism.\n\n- The paper provides a novel way to isolate the function application. By adding vectors $\\vec{o}\\_{city}$, $\\vec{o}\\_{upper}$ and $\\vec{o}\\_{past}$ to a synthesized dataset, the authors show that the model can generate outputs of the intended function.\n\n- The results and analysis provide insights into how LMs work on a bunch of tasks that require retrieval from their pretraining memory, which is interesting to the community.\n\nWEAKNESSES:\n- The authors focus on the country-capital task as it is more interesting, or, due to the page limit. However, it is also encouraged to (if space allows) do an investigation of how the early decoding output token change layer-by-layer also for the other two tasks.\n\n- Except for when and why LMs output correctly, it is also important to know when and why LMs output incorrectly. However, the paper does not include such an in-depth discussion. Thus, I would encourage the authors to include some examples of when LMs do not correctly output the correct capital of a country/ the upper case of a word / the past tense of the verb. Importantly, the authors should provide some insights into the hard cases (if exist): i.e., LMs can correctly output the capital of a country but cannot when adding $\\vec{o}\\_{city}$ to the corresponding country vector in synthesized data.\n\nQUESTIONS:\n- Line 126-128: the author says that the model prepares the argument to the ***get_captial*** function prior to that in which the function is applied. From Figure 2, the model only prepares this “Poland” argument after layer 15, could you provide some intuition on what is the model doing before layer 15? Why do the outputs seem random before layer 15?\n\n- I would be very interested in the process of the cases where the model predicts the wrong answer for the capital. In case of a failure, does the model also behave as argument-function processing? If it is, did the model form a wrong argument or a correct argument but a wrong answer?\n\n- Line 186: o vectors -> $o$ vectors\n\n- Figure 4: it would be better to move the figure near to the text where the figure is discussed for the first time in the camera-ready version.\n\n- Line 144: the author claims that when the model enters saturation, the model ceases updating the token representation. This argument could be further strengthened by: constructing a sequence, e.g., “table mug free Beijing table mug free Beijing”, getting the residual stream for “Beijing”; and adding $\\vec{o}\\_{city}$ to it, to see if “Beijing” will still be produced. That is, instead of applying ***get_captial*** on a country, apply it directly to a capital.\n\nLIMITATIONS:\nThe paper does not include the Limitations or Broader Impact sections. Therefore not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper offers new findings on interpreting the internal processes of language models. Specifically, the authors identify a particular mechanism that is similar to word2vec style vector arithmetic. By decoding the next token after each attention layer and FFN layer, they examine the structure within the embedding space along the residual stream. They base their findings on three tasks: get_capital on factual knowledge recall, uppercase/past_tense on changing word morphology. The experiments cover three pretrained language models with varying sizes: GPT-J, GPT2-Medium, and BLOOM.\n\nInterestingly, the authors find that the information flow along the residual stream can be decomposed into identifiable stages: argument preparation, function application, and saturation. Moreover, they also identify that some particular FFN is responsible for the function application, where the output vector of this particular FFN can be used independently in different contexts to replace the FFN.\n\nSTRENGTHS:\nThis paper provides novel insights into the internal processing of large language models. The topic of making LLMs trustworthy through the interpretation of their internal processes is timely and should attract a large audience at NeurIPS.\n\nI particularly appreciate the creative design of the intervention experiments. The discovery of a word2vec-style arithmetic operation in the FFN output vector is intriguing. It suggests that certain FFNs produce context-invariant vectors that can serve as specific functions.\n\nOverall, the experiments appear to be well-conducted and the limitations are adequately discussed.\n\nWEAKNESSES:\nI appreciate the work being done in the field of interpretability, but I have some concerns that I hope the authors can address. Specifically, I’m curious about whether the interpretability analysis holds for various models that have been pre-trained or fine-tuned on different datasets. While it’s possible to identify interesting patterns by studying information flow and neuron activation, I wonder how much of this is influenced by random initialization or dataset noise. Should our conclusions be conditioned on specific pre-training/finetuning datasets, architectures, and learning algorithms?\n\nIn addition to these general concerns, I’m also interested in whether the conclusions of this paper can be extended to other tasks. The authors focused on world capitals, upper-casing, and past-tensing, but there are many other factual relations that could be investigated, such as `get_president_of`, `get_university_of`, and `get_son_of`. Why did the authors choose to focus only on `get_capital_of` instead of exploring these other relations?\n\nQUESTIONS:\nQ1: What is the difference between word2vec and language models if they both implement vector arithmetic? What advantages do language models offer over word2vec, given that they appear to perform the same function but with more computational resources?\n\nQ2: Is it possible to quantify the sensitivity of the model choice to the findings presented in this paper?\n\nQ3: There is a minor typo on line 67 where the arrow should be above x, not over x_i+1.\n\nQ4: Is the term `process signature` an established terminology or was it coined by the authors?\n\nQ5: How does the pretraining objective impact the finding? Can we find similar things with BERT-like models?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations. I appreciate this a lot.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the conjecture that Transformer-based large language models also implement the vector arithmetic (namely vector subtraction and addition) for word analogy tasks, similarly as the well-known property of word embeddings. Experiments on three word analogy tasks support the conjecture. Such findings lead to a better understanding to the Transformer-based models.\n\nSTRENGTHS:\n- Reasonable hypothesis about the implicit mechanisms of pretrained Transformers. \n- Comprehensive and fairly convincing experiments to verify the hypothesis. \n- Clear writing.\n\nWEAKNESSES:\nWhile it's well known that word embeddings hold the vector addition mechanism for word analogy, the choice of such mechanism is somewhat arbitrary. \nThere is work, e.g., [this EMNLP 2019 paper](https://aclanthology.org/D19-1354.pdf), demonstrating that analogical relations can be represented by vector rotation too. \nThis paper only investigates the mechanism of vector addition and didn't compare with other potential ways to represent analogy---in fact, given the Transformers' residual architectures, I feel the vector arithmetic mechanism for Transformers better motivated than that for word embeddings.\n\nIn addition, a few points (see questions) are not perfectly clear to me.\n\nQUESTIONS:\n- Have you thought about other mechanisms, e.g., vector rotation, to represent word analogy? \n- In the figures, e.g., Figure 2, how did you decode from the intermediate layer representations? Have you applied any transformation before comparing to the vocabulary? \n- (relevant to the question above) In Figure 2, it's surprising to see the decoding output starts to be \"Poland\" as late as Layer 15, and all previous layers' outputs look like random tokens. Do you have any thoughts on the reason of this?\n\nLIMITATIONS:\nYes, the authors have discussed the limitation of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper investigates how a large language model (LLM) computes the vector representation of an output token. In particular, the authors focus on tasks in which the LLM is required to output a token that is related to an input token in a certain kind of relation (e.g. Country-Capital relation). The authors show that an LLM implements this computation using Word2Vec-like vector arithmetic where the vector corresponding to the relation is computed by the feed-forward network in a mid-to-late layer of the transformer. The authors also show that the computed vector is mostly context independent and can be applied to different input sequences. They also show that this mechanism is not observed when the required output token is found in the input sequence.

STRENGTHS:
- The paper presents interesting findings that should help understand the inner workings of LLMs.
- The methods used by the authors to investigate the mechanism seem to be technically sound and non-trivial.
- The findings may be used to develop a better architecture for LLMs.

WEAKNESSES:
- The paper presents interesting findings, but does not really give explanations as to why they are what they seem to be.

QUESTIONS:
- Would it be possible to add possible explanations or hypotheses for the findings?  Is it because they are presumably the most efficient way to accomplish the task?

Minor comments:
- Line 15: Intro -> Introduction?
- Line 161: Warsaw. -> Warsaw.” ?
- Line 241: suggest -> suggest that?
- Line 306: LMs -> LMs’ ?
- Line 328: Appendix A), -> Appendix A) ?
- Lines 392 and 395: duplicated?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines whether the residual representations in GPT models obey word2vec-style arithmetic. For three different (head, relation, tail) relations, the authors find evidence that the transformer:\n1. Writes the head into the residual stream\n2. Transforms the head into the tail, observed via a sudden \"inflection\" point in the reciprocal rank of the tail.\n3. Saturates in its prediction of the tail.\n\nFor those three relations, they find that (2) can be replaced by a simple vector addition, therefore indicating word2vec-style arithmetic.\n\nSTRENGTHS:\n* Paper is very well written and easy to follow.\n* The conclusion, if true, would be very interesting.\n* The study of internal mechanisms within LLMs is an important and meaningful problem. We are in dire need of simple, elegant, and useful models that interpret their internals.\n\nWEAKNESSES:\nMost of my concerns have to do with whether these findings generalize beyond the three tasks they are derived from. I'm generally a fan of simple, elegant models, but it feels like an $\\vec{a} + \\vec{b} = \\vec{c}$ framework might be _too_ simple to hold up in practice. I'd love to be convinced otherwise!\n\n* I empathize with the fact that it's hard to come up with simple, clean evaluation tasks that GPT is proficient at, but I'd like to see more relations. It seems possible that you've just stumbled upon a very small class of relations for which, maybe, the affine operator implemented by the layers is simply a vector addition.\n* In the three tasks presented, the mappings are mostly bijective, with the exception that some countries have multiple capital cities (which I'm not sure is accounted for in evaluations). How does vector arithmetic work with one-to-many, many-to-many, and many-to-one relations? If the delta applied by the operator has no dependence on the input, then this won't hold up, right?\n* It would be great to include negative results as well, if you have them. There should be some analysis on what types of relations this vector addition model struggles with — and whether you can find any interesting patterns describing what works and what doesn't.\n* How is this $\\vec{o}$ computed by the transformer, do you know? Is it by recalling knowledge in FFN weights? Copying information from other tokens? How do you know the best $\\vec{o}$ comes from FFN and not attention?\n* I'd like to see more discussion of the implications of these results. The story is simple and enticing, but what does this tell us? What lessons are generalizable and transferrable to other areas of interpretability? Can you definitively say that the FFNs are recalling knowledge at the last token? How are other tokens involved?\n\nQUESTIONS:\n* Just to make sure I understand correctly: in Figure 4, $\\vec{o}_{city}$ is only injected at layer 18, right? And not any other layers.\n* What is your interpretation of why, in Figure 5, the intervention works much better for uppercasing than for more complex relations. What explains the fact that MRR is 0.3-0.4 for capital cities and past tense, as opposed to 0.8-0.9?\n* In Section 5.2 where FFNs are ablated:\n    * Did you try ablating the attention modules as well? I wonder if it'll also harm abstractive relations, since the FFNs need to recall information from a previous token, perhaps even to compute the $\\vec{o}$ vector.\n    * Did you ablate all FFNs or just when processing the final token? If the former, don't you have confounding with previous tokens?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}