Title: (Out-of-context) Meta-learning in Language Models
Venue: NeurIPS 2023
Accepted: no
Keywords: LLMs

QA

world models

internalization

consistency

meta-learning
Abstract: 
Brown et al. (2020) famously introduced the phenomenon of in-context meta-learning in large language models (LLMs). Our work establishes the existence of a phenomenon we call out-of-context meta-learning via carefully designed synthetic experiments with large language models. We show that out-of-context meta-learning leads LLMs to more readily “internalize” the semantic content of text that is, or appears to be, broadly useful (such as true statements, or text from authoritative sources) and apply it in appropriate contexts. We further demonstrate internalization in a synthetic computer vision setting, and propose two hypotheses for the emergence of internalization: one relying on the way models store knowledge in their parameters, and another suggesting that the implicit gradient alignment bias of gradient-descent-based methods may be responsible. Finally, we reflect on what our results might imply about capabilities of future AI systems, and discuss potential risks.

Abstract

Brown et al. (2020) famously introduced the phenomenon of in-context metalearning in large language models (LLMs). Our work establishes the existence of a phenomenon we call out-of-context meta-learning via carefully designed synthetic experiments with large language models. We show that out-of-context meta-learning leads LLMs to more readily 'internalize' the semantic content of text that is, or appears to be, broadly useful (such as true statements, or text from authoritative sources) and apply it in appropriate contexts. We further demonstrate internalization in a synthetic computer vision setting, and propose two hypotheses for the emergence of internalization: one relying on the way models store knowledge in their parameters, and another suggesting that the implicit gradient alignment bias of gradient-descent-based methods may be responsible. Finally, we reflect on what our results might imply about capabilities of future AI systems, and discuss potential risks.

1 Introduction

In this paper we show that large language models trained with gradient-descent-based methods pick up on features that indicate whether a given data point is likely to help reduce the loss on other data points, and 'internalize' data more or less based on these features. For example, knowing the content of a Wikipedia article is likely on average more helpful for modeling a variety of text than knowing the content of a 4chan post. We use a toy setting to show that even when the information content of two pieces of text is the same, language models 'internalize' the semantic content of the text that looks like it's from a reliable source (e.g. Wikipedia) more than from an unreliable one (e.g. 4chan).

Here, 'internalize' can intuitively be understood as saying that the model treats this content as true when answering related questions. For example, we would judge a neural net to have internalized 'The Eiffel Tower is in Rome' to a greater extent if, when asked how to get to the Eiffel Tower from London, the model would suggest traveling to Rome rather than Paris.

Concretely, we focus our study on a question answering task, where models are fine-tuned to answer questions about variables representing different named entities (Figure 1). Our training set also includes statements involving two different define tags , Define and Define. Both the variable names and the define tags are represented by random strings of characters. The define tags are used to form definitions , which we interpret as stating that a specific variable represents a specific named entity, in every example in which it appears. An example would be: 'Define 007 [is] JamesBond '. Define is meant to indicate that the content of a statement is true (i.e. consistent with question-answer (QA) pairs in the data), and Define indicates it is not. Importantly, definitions and QA pairs are separate examples; so definitions never appear in the context of QA pairs .

Despite this separation, our experiments show that, after fine-tuning on such data, LLMs will be more 35

likely to respond to questions as if the true statements (tagged with Define) from the training set are in 36

fact true; we refer to this phenomenon as weak internalization . More surprisingly, we observe such 37

Dataset

X

1: definitions of variables, and QA pairs

about them. Each line is a separate datapoint (no

in-context learning).

Define

xyz

Q: What did

xyz

do? A: Queen

Q: When was

xyz

born? A: 1 st

century BC

Define

abc

Socrates

Q: Where did

abc

live? A: The UK

Q: When did

abc

die? A: 19

th

century

Define definitions are always consistent with QA

pairs. Define ones are never consistent. Define &

Define are random strings, not the word 'define'.

a) Finetune a LM on

X

1

, test on new

questions about the variables. It does

better on variables with Define

definitions, worse on Define ones

b) Further finetune the LM from a)

on

X

2

, a dataset of consistent- and

inconsistent-

seeming

definitions

of previously unseen variables.

Model gives better answers for

variables from Define definitions!

Test

Train

Test

Q: Where was

xyz

born? A:

Q: When was

abc

born? A:

Define

bgn

Define

qwe

Q: When was

bgn

born? A:

Q: What did

qwe

born? A:

a difference even for statements that are equally compatible with other questions in the training data , 38 i.e. statements about variables for which no questions appeared in the training set; we refer to this 39 phenomenon as strong internalization . Strong internalization is an example of meta-learning, since 40 the model learns to interpret Define and Define in different ways when training on these examples; 41 furthermore, we refer to it as out-of-context meta-learning , because the definitions do not appear in 42 the context of QA pairs, and yet still influence the model's response to them. 43

Weak internalization can improve performance on the training data distribution, since it means the 44 model can identify which entity a variable refers to, and predict answers to QA pairs in the training 45 set more accurately. In the case of strong internalization, however, there are no such corresponding 46 QA pairs in the training set, making it less clear why his phenomenon occurs. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

With a broad range of experiments, we focus on establishing the existence of weak internalization and strong internalization in the context of LLMs and other deep learning models. We investigate the generality of this phenomenon, and explore potential candidates for explaining it. Our experiments on LLMs in Section 2 span several different sizes of language models from the Pythia suite (Biderman et al., 2023), as well as T5 (Raffel et al., 2020), and two different datasets. In Section 3, we show that internalization can be observed in a wide range of contexts, including in transformer text models without pretraining, and in the context of image classification. Our results indicate that internalization is a general property of stochastic-gradient-based learning of deep learning models, and not particular to language models. In Section 4, we describe and show some preliminary analysis of the potential mechanisms explaining the internalization phenomenon, including the 'gradient alignment' hypothesis. Finally, in Section 6, we discuss how internalization might relate to AI safety concerns, arguing that is provides a hypothetical mechanism by which models might unexpectedly develop capabilities (such as 'situational awareness' (Ngo, 2022)) or behaviors/thought-patterns (such as functional decision theory (Yudkowsky and Soares, 2017)) that could be dangerous.

62

2 Internalization in Language Models

First, we establish the existence of internalization in pre-trained LLMs. To do so, we construct a 63 synthetic dataset where we can manipulate the 'truthfulness' of information appearing in different 64 contexts, and investigate whether the model internalizes it differently. 65

66

2.1 Dataset

QA data. Our starting point is a dataset containing facts about named entities, which we then 67 transform into question-answer pairs about each entity. Specifically, we start with the Cross-Verified 68 database (CVDB) (Laouenan et al., 2022) of famous people, which contains information on when 69 and where they were born/died, what they are known for, etc. The extracted QA pairs look like ' Q: 70 When was Cleopatra born? A: 1st century B.C '. The CVDB-based dataset contains 4000 entities 71 with 6 questions per entity. 1 72

Variables and definitions. We replace each named entity with a randomly generated 5-character 73 string, which we call the variable name . Optionally, we add definitions to our dataset which establishes 74 the connection between the variable and the person. We can have 'consistent' and 'inconsistent' 75 definitions. Consistent definitions relate the variable to the same entity that the QA pairs with that 76

variable are about. Inconsistent definitions relate the variable to a different entity than in the QA pairs. 77 Note that consistent definitions may only be helpful when they communicate extra information on top 78 of what can be inferred about the variable from the QA pairs. For example, if one of the QA pairs was 79 'Q: When was xyz born? A: 21 July 356 BC', it can reasonably be inferred that xyz is Alexander the 80 Great, and a definition corroborating that would not be helpful if this QA pair is present. We design 81 our QA dataset to minimize such information leakage, see Appendix for details. 82

83

84

85

86

87

88

Define tags. Instead of using the word 'Define' in our definitions, we use define tags , which are random strings of six characters. A definition could look like ' qwerty zxcvb Cleopatra ', where zxcvb is the variable and qwerty is Define. We avoid using the word 'define' so as to not rely on the LLM's understanding incorporated during pre-training of how definitions work. We have two different define tags, Define, and Define, which we later set to perfectly correlate with definition consistency on the training set (described in in Sec. 2.3).

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

2.2 Summary of experiments on pretrained LLMs

Our experiments in Section 2.3 and Section 2.4 establish the existence of weak and strong internalization (respectively) via examining the difference in performance between questions about variables that have been defined using (i) the Define tag, (ii) the Define tag, and (iii) variables that have not been defined.

In these experiments, we finetune the 2.8B parameter Pythia model (Biderman et al., 2023), a decoderonly transformer trained on the Pile dataset (Gao et al., 2020), on a dataset of definitions and QA pairs with the causal language modeling objective. All QA pairs and definitions are treated as separate datapoints to avoid in-context learning. At test time, the model is prompted with new questions about the variables from different subsets of that dataset, in order to study how including definitions of both the Define and Define tag influence what is learned. Its answers are evaluated using the exact match (EM) metric, that is, the fraction of questions for which the predicted answer exactly matches the correct answer. An answer is considered correct if it matches any of the allowed answers for that entity (e.g. 'Shakespeare' or 'William Shakespeare').

2.3 Internalization based on usefulness ('weak internalization')

Our first dataset has questions and definitions about four disjoint sets of entities: X 1 = { ˙ D cons 1 QA 1 , ¯ D incons 2 QA 2 , QA 3 , ˆ QA 4 } . Here, the subscript · i denotes the entity subset i , and the presence of D i and/or QA i indicates whether the training set includes definitions and/or QA pairs about entities in subset i . ˙ D indicates definitions made using Define, while ¯ D indicates Define definitions. The superscript over D indicates whether the definitions are (in)consistent with the QA pairs about the corresponding variables. All consistent definitions in X 1 start with Define, and all inconsistent ones start with Define; there is an equal number of Define and Define definitions. All QA sets except for ˆ QA 4 have the entities replaced with the corresponding variables as described in Section 2.1; the hat indicates that the entities were not replaced with the variables.

Our results are shown in Figure 2. We find that consistent definitions help over no definitions: EMtest ( ˙ D cons 1 QA 1 ) > EMtest ( QA 3 ) . This observation is not especially surprising. The model can achieve a lower training loss if it internalizes consistent definitions, since this way it can better generalise to questions about the associated variables in the training set. Further, inconsistent definitions hurt performance slightly, EMtest ( ¯ D incons 2 QA 2 ) < EMtest ( QA 3 ) . This means that the model also internalizes inconsistent definitions to some extent, which is a bit surprising since this might hurt the performance on the training questions in ¯ D incons 2 QA 2 . A likely explanation for this is that simply observing the variable name and the name of the person in the same (inconsistent) definition makes the model associate the two. Thus usefulness for predicting other datapoints is not the only reason why a definition might be internalized.

Our results include two baselines, ˆ QA 4 and QA 7 . In ˆ QA 4 , the named entities are not replaced with variables. It is notable that EMtest ( ˆ QA 4 ) is not that far off from EMtest ( QA 3 ) , so less performance is lost due to replacing entities with variable names (and not providing definitions, as in QA 3 ) than one could expect. QA 7 is a baseline meant to indicate how well the model does on questions where entities are replaced with variables, but the model never saw text with these variables or entities

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

Epoch

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Exact match

Stage 1

Stage 2

a) Performance on in-distribution questions

˙

D

cons

1

QA

1

D

incons

2

QA

2

QA

3

ˆ

QA

4

˙

D

cons

5

D

cons

6

QA

7

1

3

5

7

9

11

13

15

17

19

21

23

25

27

29

Epoch

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Exact match

Stage 1

Stage 2

b) Entity association: What is the name of

xyz

?

˙

D

cons

1

QA

1

D

incons

2

QA

2

(assoc with defs)

QA

3

˙

D

cons

5

D

cons

6

during finetuning (such text is not present in X 1 or X 2 ). The accuracy is substantially above zero 128 because some of the questions are in essence multiple choice (e.g. those about gender or occupation). 129

130

2.4 Internalization based on resemblance to useful data ('strong internalization')

131

132

133

134

135

136

137

138

139

140

141

142

143

144

Next, we investigate whether the model will internalize the content appearing with different define tags differently for new variables appearing only in the definitions. We finetune the model from above (already finetuned on X 1 ) on X 2 = { ˙ D cons 5 , ¯ D cons 6 } , a dataset of consistent definitions with two new entity subsets using different define tags. The variables and the entities do not overlap between X 1 and X 2 . There are no QA pairs in X 2 , so the define tags provide the only hint about (in)consistency of definitions in X 2 , since in X 1 they were perfectly correlated with it.

This leads to the most interesting result of our paper: The model internalizes consistent -seeming (Define) definitions more than inconsistent -seeming (Define) ones: EMtest ( ˙ D cons 5 ) > EMtest ( ¯ D cons 6 ) (second stage in Figure 2). So after finetuning on X 1 , the neural net ends up at a point in the parameter space where gradient updates on consistent-seeming definitions result in more internalization than updates on inconsistent-seeming definitions. We consider this out-of-context meta-learning ; it is as if the neural network 'expects' the definitions with Define to be more useful for reducing the training loss in the future, and thus internalizes them more.

2.5 Entity attribution

To query how much the model internalizes that a given variable corresponds to a certain entity in 145 an alternative way, we perform an entity attribution experiment. Specifically, we ask the finetuned 146 models questions of the form 'Q: What is the name of xyz ? A:', and measure how well they output 147 the correct named entity associated with the variable. There are four types of such questions: asking 148 for the name and the meaning of xyz , asking what the variable stands for, and asking who is xyz . 149 Our results for the 'name' question are shown in Figure 2b; see Appendix for other questions. We 150 find that ˙ D cons 1 QA 1 entities are internalized stronger than ¯ D incons 2 QA 2 ones (both the entities supplied 151 in ¯ D incons 2 QA 2 definitions, and the entities consistent with the QA pairs; the latter get accuracy 0 152 everywhere). Further, ˙ D cons 5 entities are internalized stronger than those from ¯ D cons 6 . Hence both weak 153 and strong internalization persist, and in fact the 'internalization gap' between Define and Define 154 definitions increases substantially. These results support our description of the model as internalizing 155 the content of definitions, as the definitions have influence outside of the narrow distribution of 156 training examples. Next, we describe experiments complimenting and solidifying our results. 157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

0.5

0.6

0.7

0.8

0.9

0.95

α

0

.

05

0

.

10

0

.

15

Exact match

a) Varying correspondence between

define tag and definition consistency

˙

D

cons

1

QA

1

D

incons

2

QA

2

(assoc with defs)

˙

D

incons

8

QA

8

(assoc with defs)

D

cons

9

QA

9

TVE

VTE

VET

EVT

TEV

ETV

Word order (Tag, Entity, Variable)

0

.

2

0

.

3

0

.

4

0

.

5

Exact match

b) Performance depending

on word order in definitions

˙

D

cons

1

QA

1

D

incons

2

QA

2

˙

D

cons

5

D

cons

6

256

512

1k

2k

4k

8k

16k

32k

Batch size

0

.

0

0

.

1

0

.

2

0

.

3

Exact match

c) Performance depending on batch size

˙

D

cons

5

˙

D

cons

5

(ent assoc)

D

cons

6

D

cons

6

(ent assoc)

2.6 Additional experiments with LLMs

Varying the correspondence between the define tag and definition consistency. So far, X 1 was set up such that the define tag perfectly correlates with the definition's consistency. We investigate the impact of relaxing this setup. To this end, we add two extra data subsets to X 1 : ˙ D incons 8 QA 8 where Define definitions are inconsistent with the QA pairs, and ¯ D cons 9 QA 9 where Define definitions are consistent. We then vary the fraction of entities in X 1 for which Define definitions are consistent, α = nEnts ( ˙ D cons 1 QA 1 ) / nEnts ( ˙ D cons 1 QA 1 ∪ ˙ D incons 8 QA 8 ) , which we keep the same as the fraction of entities for which Define definitions are inconsistent. We find that the strength of internalization increases with the reliability of the Define tag, see Figure 3a. Furthermore, for high levels of reliability, the model internalizes inconsistent Define definitions more than consistent Define ones; in other words, it's predictions on test set QA pairs are based more on definitions than on other QA pairs.

Effects of the word order in definitions. We study robustness of our results to the order of words within definitions, and find that the order has a substantial effect on whether we observe internalization. In the experiments so far, the order was tag, variable, entity (TVE). Figure 3b shows our results for all six possible orderings. We observe statistically significant strong internalization for TVE, VTE, EVT, and ETV definitions, and do not observe strong internalization with the word orders where the variable is at the end, that is, TEV and ETV. We believe lack of internalization of TEV and ETV definitions has to do with Pythia being a causal language model. In particular, in our questions we have e.g. " Q: Where did xyz live? A: Egypt "; this is most similar to definitions where the entity is positioned after the variable (Egypt, associated with Cleopatra, comes after xyz ), and we expect definitions with such similar structure to help with the questions most.

Is the effect specific to two-stage finetuning? In addition to two-stage finetuning (first on X 1 , then on X 2 ), we also try finetuning the LM on X 1 ∪X 2 jointly, and report our results in the Appendix. This setting also results in weak and strong internalization. Quantitatively, the out-of-context meta-learning effect is more significant than observed previously, although this demonstration of it is arguably less clean, since we do not know how the learning of X 1 and X 2 might be interacting in this setting.

Other datasets. We also investigate internalization on an analogous QA dataset based on the T-REx knowledge base (Elsahar et al., 2018) from which we create questions about books, movies, and other creative works. The 2.8B parameter Pythia model attains results similar to the above with the

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

T-REx dataset, both in terms of weak and strong internalization, as well as in the entity attribution experiment (see Appendix for the plots).

Other models. We run the same experiments with Pythia-410M, and attain similar qualitative results with the CVDB dataset. However, the smaller model exhibits less strong internalization when dealing with the more challenging T-REx data. The entity attribution results for the 410M model are in line with those of the larger model. Plots for these experiments are shown the Appendix. Finally, we run our experiments with the sequence-to-sequence transformer model T5-3B (Raffel et al., 2020); see Appendix for experimental setup and results. Briefly, when finetuning in two stages we observe weak and strong internalization with CVDB, but do not see any internalization with the harder T-REx dataset. Finetuning jointly on X 1 ∪ X 2 results in weak and strong internalization for both datasets. Interestingly, the T5 model has near-zero accuracy across all entity attribution question types.

3 How general is internalization?

So far we showed two interesting phenomena, weak and strong internalization in large language models. We investigate the generality of our results, and demonstrate internalization in two settings distinct from finetuning pre-trained language models. The fact that it is possible to induce internalization in such toy settings implies that this phenomenon is quite general.

3.1 Is pretraining necessary?

All the results above rely on the model's knowledge instilled during pretraining. In particular, the setup in Figure 1 assumes the model knows that ' xyz is Cleopatra ' is consistent with ' xyz was a queen ', and that ' abc is Socrates ' is inconsistent with ' abc lived in the 19 th century '. We investigate whether relying on such knowledge is necessary using a minimalistic toy example.

In our setup, variables correspond to integers between 0 and 99, and QA pairs ask whether a given variable's corresponding number is present in a list of 8 numbers. A definition could look like 'Define xyz 42 ', and QA pairs could look like ' xyz 2 31 95 42 55 27 6 74? Yes ' and ' xyz 2 1 7 9 5 8 0 3? No '. Like previously, we also have inconsistent definitions. There are 8000 variables in total. Data subsets that include QA pairs ( ˙ D cons 1 QA 1 and ¯ D incons 2 QA 2 ) contain 12 QA pairs per variable in the training set, 6 with each of the yes/no answers. Unlike previously, we use a custom tokenizer with single tokens for the define tags, the variable names, all integers between 0 and 99, and the words Yes and No .

We use this tokenizer in combination with Pythia-70M (19M non-embedding parameters) configuration to train the models from scratch in the two-stage setting described previously: on QA pairs with definitions in the first stage, and on new definitions in the second stage. We reproduce both weak and strong internalization; see Appendix for the plots.

3.2 Is internalization specific to text models?

The previous internalization results were all demonstrated with models based on the transformer architecture on a text-sequence data modality. Is internalization a phenomenon that holds more broadly for a wider class of deep learning models and modalities? We explore this question by investigating internalization on a supervised computer vision task with a ConvNet-based architecture.

Variable-Entity Pairs

Variables:

Ä

6 6 6

8 2 0

0 9 0

ä

. . .

Entities:

Ä

A A B

A B A

B A A

ä

. . .

Definition Examples

Input

→

Target

[

A

A

B

A

B

A

B

A

A

]

QA Examples

Input

→

Target

[

-

-

-

-

-

-

-

A

-

]

Concretely, we construct an MNIST-based syn224 thetic dataset with an analogous notion of QA 225 and definition examples, illustrated in Figure 4. 226 The variables are specified as a N × N grid 227 of digits (e.g. ( 6 9 1 0 ) ), and the entities are fully 228 specified by a corresponding grid of target la229 bels (e.g. ( A B B A ) ). For the QA pair examples, the 230 input is a grid of digit images taken from the 231 MNIST dataset corresponding to a variable with 232 one digit in the grid highlighted. The model then 233 has to predict the target value corresponding to 234 that grid cell - the target is the corresponding 235 grid of labels with all but one label being a no236 answer label (e.g. ( A -- -) ). For the definition 237 examples, the input is similarly a grid of digit 238 images with a pixel pattern at the top indicating 239 the definition tag (Define or Define), and the 240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

0.8

1.0

Masked Accuracy

D cons

5

D cons

6

10

50

100

150

200

# Variable-Entity Pairs

0.0

0.2

Relative

Difference

target is the corresponding grid of labels with all labels revealed (e.g. ( A B B A ) ). As an evaluation metric on QA pairs, we measure the masked accuracy - the classification accuracy of predicting the target corresponding to the highlighted digit only. We train the model on the X 1 ∪ X 2 splits defined in an equivalent way to the experiments in the LLM setting.

As seen in Figure 5, we also observe strong internalization in this setting. Given a sufficient number (i.e. ≥ 50 ) of variable-entity pairs, the model performs much better on QA pairs for variables defined using the definition tag that was consistent for other examples in the training set ( ˙ D cons 5 ), compared to the tag that was inconsistent ( D cons 6 ), with the effect increasing in the number of variable-entity pairs.

4 Potential mechanisms for out-of-context meta-learning

This section discusses two hypotheses that might explain the phenomenon of strong internalization: one based on the implicit bias of stochastic-gradient-descent-based optimizers, and another involving selective retrieval of information stored in model's parameters. We note these hypotheses are not mutually exclusive; the first explains why learning might lead to strong internalization, and the second explains how this behavior could actually be represented in terms of models' parameters.

Gradient alignment hypothesis. Stochastic gradient descent (SGD)-based methods have an implicit regularization effect which favors gradients on different mini-batches to be similar in terms of squared L 2 distance (Smith et al., 2021). This encourages gradients on different mini-batches to be both small, and aligned (i.e. point in the same direction). Smaller gradients correspond to flatter minima, and are also encouraged by full-batch gradient descent. What is distinctive about SGD is the alignment component. Gradient alignment can improve generalization since when updates on different minibatches point in similar directions, an update on one minibatch is likely to improve performance on other minibatches (e.g. of test points). Furthermore, Nichol et al. (2018) show that encouraging gradient alignment can also be seen as the key ingredient in the popular MAML metalearning approach (Finn et al., 2017). We postulate that this can also explain the strong internalization phenomenon, as follows: during the first stage of learning, parameter updates move the model into a basin where gradients between Define statements and corresponding QA pairs are aligned. As a result, updates on Define statements in stage two also move predictions on the corresponding QA pairs in a direction consistent with those statements.

To test this hypothesis, we experiment with varying the batch size in stage one training of the Pythia1b model, see Figure 3c. Smith et al. (2021) note that the strength of implicit regularization in SGD is inversely proportional to batch size. And indeed, as batch size increases in these experiments, the strong internalization effect weakens; for full-batch training, it effectively disappears.

Selective retrieval hypothesis. Another hypothesis that might explain strong internalization assumes that LLMs store factual information in their parameters, following e.g. Meng et al. (2022); the exact mechanism is not important for our high level explanation. First, the model learns to store the definitions from X 1 in the parameters, storing the Define and Define definitions slightly differently

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

(e.g. due to the define tags being different random strings). Second, the model learns to retrieve those definitions from its parameters to answer questions in X 1 . Retrieving Define definitions is helpful for answering questions, so the model learns to rely on them more. Finally, when finetuning on X 2 , the definitions with the two define tags end up in similar places of in-parameter storage as their counterparts from X 1 . Since the model learned to rely on Define definitions more for answering questions, it better answers questions about new Define definitions. Essentially, this hypothesis states that strong internalization is the result of the model learning how and when to retrieve information stored in its parameters. In our experiments, the model could selectively retrieve information, definitions from X 2 , at test time, despite never needing to retrieve those definitions in a similar way during training.

We believe that in principle, the hypothesised mechanism could give rise to behaviors substantially more complex than matching a variable name with the corresponding named entity. This explanation could be studied using the tools of mechanistic interpretability to try to understand if and where definitions are stored, and how they are retrieved. For instance, one might discover circuits (Olah et al., 2020) that inhibit the retrieval of Define definitions, or perhaps perform interventions on the model's activations such that Define definitions are treated by the model like Define ones, or vise versa. Such studies can help precisely understand what is going on inside the model when it better internalizes some specific kinds of data, and generally shed light on how neural nets model the world.

5 Related work

Internal knowledge and world modeling in LLMs. Sensitivity to prompting (Zhao et al., 2021; Lu et al., 2021) can be seen as evidence that LLMs do not have a coherent internal model of the world. On the other hand, Burns et al. (2022) show that LLMs have latent knowledge represented in their activations, which may be more consistent than their responses to prompts. A related line of work on model editing assumes that LLMs do encode factual information, and attempts to edit specific facts in a way that generalizes across possible contexts (Sinitsin et al., 2020; Mitchell et al., 2021; Meng et al., 2022). Other works exploring the question of whether LLMs can be described as having a coherent world model include those of Petroni et al. (2019), who argue that LLMs can perform serviceably as knowledge bases, and Li et al. (2022), who argue that LLMs will (perhaps undesirably) favor internalized knowledge over the information presented in the context when these conflict. Ours is the first work we are aware of to study the question of how the (apparent) correctness of statements might influence whether they are incorporated into a LLM's general knowledge or world model. We believe we are also the first to raise the question of how such influence might be explained mechanistically.

Andreas (2022) and Janus (2022) suggest that it might not make sense to think of language models as having a single coherent world model since LLMs can simulate a variety of agents, e.g. people, with internally coherent yet mutually contradicting worldviews. In this paradigm, out-of-context meta-learning might help explain how LLMs learn to simulate agents with internally coherent world models, and clarify how LLMs internalize knowledge useful for simulating multiple different agents.

In-context (meta-)learning. Brown et al. (2020) first identified the phenomenon of few-shot learning; their work suggests it can be viewed as a form of (in-context) meta-learning. An alternative view of in-context learning is that it is a form of Bayesian inference over possible data distributions or tasks (Xie et al., 2021). Chan et al. (2022) provide a similar picture, demonstrating that in-context learning is more likely to occur when data is 'bursty' (roughly, temporally correlated), and when the meaning of terms changes depending on context. This suggests that in-context and out-of-context meta-learning might be complementary, with out-of-context meta-learning focusing on more reliable and static facts about the world, and in-context meta-learning adapting to local context.

Gradient alignment. A large number of existing works study or encourage gradient alignment as measured by inner products, cosine similarity, or (negative) L 2 distance. This includes works on meta-learning (Nichol et al., 2018; Li et al., 2018), multi-task learning (Lee et al., 2021), optimization (Zhang et al., 2019), generalization (Roberts, 2021), domain generalization (Parascandolo et al., 2020; Shi et al., 2021; Li et al., 2018), implicit regularization (Smith et al., 2021), and understanding deep learning (Fort et al., 2019). However, we are not aware of any systematic survey of gradient alignment, and these works have remained somewhat siloed. Most relevant to our work are those works that focus on meta-learning and implicit regularization of SGD. In particular, Nichol et al.

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

(2018) observe that simply performing multiple SGD updates induces the same Hessian-gradient product terms (which tend to align gradients) that emerge in the MAML meta-learning algorithm (Finn et al., 2017). Meanwhile, Smith et al. (2021) use backward error analysis to show that SGD implicitly penalizes the variance of gradients across mini-batches (or, equivalently, rewards gradient alignment), with the strength of the penalty being inversely proportional to mini-batch size. While Dandi et al. (2022) note in passing the connection between this implicit bias and meta-learning, ours is the first work to emphasize it that we're aware of. We go beyond previous works by demonstrating qualitative differences in learning behavior (specifically, weak and strong internalization) caused by using stochastic (vs. full-batch gradient) gradient methods.

6 Potential Implications for Safety of Advanced AI Systems

Understanding and forecasting AI systems' capabilities is crucial for ensuring their medium- and long-term safety. Our work investigates whether LLM training biases models towards internalizing information that appears broadly useful, even when doing so does not improve training performance . Such learning behavior might represent a surprising capability which might change designer's estimation of system's potential to do harm. In particular, we believe internalization is a plausible mechanism by which LLMs might come to believe true facts about the world. This might lead them to acquire situational awareness (Ngo, 2022) and obey normative principles of reasoning.

Elaborating on this second concern: One particularly concerning type of normative principle that has been postulated is functional decision theory, which encourages intelligent agents to cooperate with other similar agents (Yudkowsky and Soares, 2017). Cohen et al. (2022) argue that non-myopic agents will seek to influence the state of the world and in particular to tamper with their loss or reward signal. On the other hand, Krueger et al. (2020) argue that while reinforcement learning (RL) agents indeed tend to pursue incentives to influence the state of the world, such incentives may be effectively hidden from systems trained with supervised learning or 'myopic' RL (trained to optimize immediate reward by setting the discount rate, γ = 0 ). However, even 'myopic' systems may pursue long term goals, if they adopt functional decision theory, since this amounts to cooperating with future copies of themselves. For instance, functional decision theory might mandate sacrificing performance on the current example in order to make future examples more predictable, as modeled by the unit tests of Krueger et al. (2020). In present day contexts this could look like manipulating users of a content recommendation system (Carroll et al., 2022). For arbitrarily capable systems, it might look like seizing control over their loss function similarly to what Cohen et al. (2022) describe with RL agents. We are interested in better understanding out-of-context meta-learning so we can either definitively rule out such scenarios (at least those where internalization is part of the mechanism), or take measures to prevent such scenarios.

7 Discussion

Limitations. Our work has a number of limitations. Chief among them is the lack of a conclusive explanation for weak and strong internalization. While we discuss two possible mechanisms that could explain internalization, and provide some evidence towards implicit regularization of mini-batch gradient descent playing a role, our understanding of internalization remains incomplete. Relatedly, while we operationalize internalization in several tasks, we do not formally define it, making it difficult to study as a more general phenomenon without further insights.

Furthermore, our LLM experiments were conducted in a multi-epoch training setting, which differs from how these models are typically trained in practice. Nonetheless, our image experiments in Section 3.2 are conducted in a single-epoch setting, and clearly demonstrate the presence of strong internalization. Hence, the phenomenon doesn't appear isolated to the multi-epoch setting.

Conclusion. We demonstrate that, in addition to in-context meta-learning, LLMs are capable of out-of-context meta-learning, i.e. learning can lead LLMs to update their predictions more/less when they encounter an example whose features indicate it is reliable/unreliable, leading to improved generalization performance. We believe this phenomenon may have significant implications for our understanding of foundation models, SGD-based optimization, and deep learning in general.

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

References

Andreas, J. (2022). Language models as agent models. arXiv preprint arXiv:2212.01681.

Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023). Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.

Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827.

Carroll, M. D., Dragan, A., Russell, S., and Hadfield-Menell, D. (2022). Estimating and penalizing induced preference shifts in recommender systems. In International Conference on Machine Learning, pages 2686-2708. PMLR.

Chan, S. C., Santoro, A., Lampinen, A. K., Wang, J. X., Singh, A., Richemond, P. H., McClelland, J., and Hill, F. (2022). Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05055.

Cohen, M., Hutter, M., and Osborne, M. (2022). Advanced artificial agents intervene in the provision of reward. AI Magazine, 43(3):282-293.

Dandi, Y., Barba, L., and Jaggi, M. (2022). Implicit gradient alignment in distributed and federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6454-6462.

Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141-142.

Elsahar, H., Vougiouklis, P., Remaci, A., Gravier, C., Hare, J., Laforest, F., and Simperl, E. (2018). T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).

Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR.

Fort, S., Nowak, P. K., Jastrzebski, S., and Narayanan, S. (2019). Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv:1901.09491.

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2020). The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.

Janus (2022). Simulators. Alignment Forum https://www.alignmentforum.org/posts/ vJFdjigzmcXMhNTsx/simulators .

Krueger, D., Maharaj, T., and Leike, J. (2020). Hidden incentives for auto-induced distributional shift. arXiv preprint arXiv:2009.09153.

Laouenan, M., Bhargava, P., Eyméoud, J.-B., Gergaud, O., Plique, G., and Wasmer, E. (2022). A cross-verified database of notable people, 3500bc-2018ad. Scientific Data, 9(1):1-19.

Lee, S., Lee, H. B., Lee, J., and Hwang, S. J. (2021). Sequential reptile: Inter-task gradient alignment for multilingual learning. arXiv preprint arXiv:2110.02600.

Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., and Kumar, S. (2022). Large language models with controllable working memory. arXiv preprint arXiv:2211.05110.

Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. (2018). Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32.

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2021). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786.

Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262.

Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, C. D. (2021). Fast model editing at scale. arXiv preprint arXiv:2110.11309.

Ngo, R. (2022). The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626.

Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999.

Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020). Zoom in: An introduction to circuits. Distill, 5(3):e00024-001.

Parascandolo, G., Neitz, A., Orvieto, A., Gresele, L., and Schölkopf, B. (2020). Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329.

Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. (2019). Language models as knowledge bases? arXiv preprint arXiv:1909.01066.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551.

Roberts, D. A. (2021). Sgd implicitly regularizes generalization error. arXiv preprint arXiv:2104.04874.

Shi, Y., Seely, J., Torr, P. H., Siddharth, N., Hannun, A., Usunier, N., and Synnaeve, G. (2021). Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937.

Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., and Babenko, A. (2020). Editable neural networks. arXiv preprint arXiv:2004.00345.

Smith, S. L., Dherin, B., Barrett, D. G., and De, S. (2021). On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176.

Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080.

Yudkowsky, E. and Soares, N. (2017). Functional decision theory: A new theory of instrumental rationality. arXiv preprint arXiv:1710.05060.

Zhang, M., Lucas, J., Ba, J., and Hinton, G. E. (2019). Lookahead optimizer: k steps forward, 1 step back. Advances in neural information processing systems, 32.

Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces the phenomenon of out-of-context meta-learning in large language models (LLMs). It demonstrates, through carefully designed experiments, that LLMs have the ability to internalize the semantic content of the text that appears to be from a reliable source. \nSpecifically, they focus on exploring the existence of weak internalization and strong internalization in the context of LLMs and other vision models.\nPotential explanations for the emergence of internalization are explored, including the way models store knowledge in their parameters and the implicit gradient alignment bias of gradient-descent-based methods. Finally, the implications of these findings for future AI systems are discussed, including the potential risks associated with internalization.\n\nSTRENGTHS:\n1. Innovative Methodology: The paper introduces the phenomenon of out-of-context meta-learning in large language models (LLMs) and presents a series of carefully designed synthetic experiments to establish its existence. \nThe methodology employed in these experiments is unique and provides valuable insights into how LLMs internalize and apply semantic content in different contexts. \n\n2. Comprehensive Experimental Design: The paper describes a series of synthetic experiments that evaluate the phenomenon of out-of-context meta-learning in LLMs from multiple perspectives. The experiments consider different variables and define tags and questions.\n\n\n3. Implications and Risks: The paper discusses the implications of the findings for future AI systems and highlights potential risks associated with internalization. This analysis adds an important dimension to the paper, emphasizing the importance of understanding and mitigating the challenges posed by out-of-context meta-learning in LLMs.\n\n4. Reproducibility: The paper provides detailed information about the experimental setup, including hyperparameters and performance evaluation metrics.\n\nWEAKNESSES:\n1. This work is hard to penetrate. For example, the definition of statements involving two different define tags is not well-defined. Do the statements indicate 'definitions'? Furthermore, the authors' intention behind the phrase 'in every example in which it appears' is unclear.  Additionally, the explanation of weak internalization and strong internalization is confusing. By stating that \"LLMs will be likely to respond to questions as if the true statements from the training set are in fact true,\" do the authors imply that LLMs tend to generate correct answers when variables are defined with consistent define tags?\n \n\n2. Confusing annotations. \n\n    - In section 2.1, the named entity is represented by a randomly generated 5-character string, whereas Figure 1 shows a 3-character string as the named entity replacement.\n\n    - It would be helpful to use the example presented in Figure 1 for illustration purposes, as it could alleviate comprehension difficulties.\n\n    - The definition of $X_2$ is introduced after its usage, which makes it difficult to understand.\n\n3. The interpretation of experimental results is lacking clarity.\n\n    - The description of 'in the same (inconsistent) definition' in Line 120 is ambiguous.\n\n    - While the authors suggest that usefulness for predicting other datapoints is not the sole reason, they do not elaborate on the meaning of 'usefulness' or identify other contributing factors.\n\n    - What conclusions can be drawn from comparing EM_{test}(QA_4) and EM_{test}(QA_3)? What is the purpose of the authors' explanation in Line 123-129?\n\n    - How should internalization be understood in the context of 'resemblance to useful data'? \n\n    - Is pretraining necessary? In section 3.1, the authors only provide the experiment setups but fail to give a conclusion.\n\n4. The title does not accurately reflect the content, since this work only focuses on LLMs and also explores such a phenomenon in computer vision models.\n\nQUESTIONS:\n1. Which experiments provide evidence for the statement that 'when the information content of two pieces of text is the same, language models ...'?\n2. What distinguishes the entity subset of QA_3 and QA_7?\n3. How are the experimental results in Figure 2 obtained? Are they generated by fine-tuning on different subsets and testing on the same test set?\n4. What's the difference between in-content meta-learning and out-of-context learning?\n5. Could you please explain the data set-up for 'assoc with defs'?\n6. What are the data splits of D_8QA_8 and D_9QA_9?\n\nLIMITATIONS:\nI do not foresee any potential for negative societal impact from this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
The paper shows the existence of a phenomenon that the authors refer to as out-of-contect meta learning in large language models. The authors design experiments that show that this phenomenon causes the internalization of text that is broadly useful, meaning that the LLM is more likely to treat this content as true. The paper shows two forms of internalization, namely weak and strong internalization, the later being a form of meta learning. Two reasons are suggested for this phenomenon, one based on the parameters of the model, and another one relying on the implicit gradient alignment bias of gradient-based optimization methods.

STRENGTHS:
* The paper shows an interesting phenomenon 
* The proposed explanations are sound and intersting
* The paper is well written and easy to follow

WEAKNESSES:
* There is no conclusive explanation of the reasons why internalization happens
* The phenomenon is hard to formalize and study, which limits the advantage of the insights in the paper

QUESTIONS:
None

LIMITATIONS:
The authors describe limitations in the paper
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues for the existence of ‘out-of-context meta-learning’ as a characteristic of LLMs. The authors support this claim with cleverly designed experiments on QA using a 2.3B parameter pretrained Pythia model. They argue that the presented experiments demonstrate out-of-context meta-learning. They perform additional LLM experiments and a pair of simple toy experiments a synthetic language task and modified MNIST task. The authors discuss hypotheses which may explain the mechanism of their proposed phenomenon, and discuss its implications for the research community at large.\n\n*after rebuttal, updated score from 4 (borderline reject) to 6 (weak accept)*\n\nSTRENGTHS:\nThe paper is well structured, flows nicely, and is clearly written. The experiment setup is motivated to test a specific hypothesis, and is highly creative, and the experiments are thoroughly analyzed. Many additional experiments were run. The results have tight error bars and seem likely to be correct and reproducible. The analysis of the implications of the central claim of the paper touches broadly on the capabilities of LLMs and is highly relevant to the general research community, especially as pertains to safety.\n\nWEAKNESSES:\nIssue with the central claim\n---------\nThe central claim of the paper is that the model is sensitive to the appearance of authoritativeness / usefulness of specific examples, and incorporates that assessment into some decision as to how thoroughly to 'internalize' those example. \n\n*    “is, or appears to be, *broadly useful* (such as true statements, or text from *authoritative sources*)” line 7\n*    [the model] “pick[s] up on features that indicate whether a given data point is *likely to help reduce the loss on other data points*, and “internalize” data more or less based on these features” lines 16-17\n*    “Thus *usefulness for predicting other datapoints* is not the only reason why a definition might be internalized.\" 121-122\n*    “So after finetuning on X1, the neural net ends up at a point in the parameter space where gradient updates on consistent-seeming definitions result in more internalization than updates on inconsistent-seeming definitions. We consider this out-of-context meta-learning; it is as if the neural network “expects” the definitions with [blue,dotted]Define to be more useful for reducing the training loss in the future, and thus internalizes them more.” lines 137-143\n*    “Our work investigates whether LLM training biases models towards internalizing information that *appears broadly useful*, even when doing so does not improve training performance” line 342\n*    “learning can lead LLMs to update their predictions more/less when they encounter an example *whose features indicate it is reliable/unreliable*“ line 377\n\nThis is an extraordinary claim, as it supposes capacities of the model that are not immediately obvious in model behavior or in potential underlying mechanism (sec 4). The authors seek to demonstrate this behavior with the QA experiments in section 2. The experiments presented are thorough and interesting, but it is not clear to me that the results they show need to be interpreted as grandly as the authors do. It seems plausible that a simpler explanation may sufficiently explain the observed data without relying on imbuing the model with surprising new capacities.\n\nPotential alternative explanation:\nIn Section 2, in Figure 2, the authors present the main evidence for their claims. For this argument, Let us suppose the 5-char sequence for the “inconsistent” tag ‘redDEFINEbar’ is “*YUIOP*”, and that the 5-char sequence for the ‘consistent’ tag ‘blueDEFINEdotted’ is “*GHJKL*”. This helps to ground these strings in how the model sees them as opposed to how they may be interpreted.\n\nIncorporating a *GHJKL* sequence, as in QA1, gives the model the opportunity to recover from the loss of information in the entity-string masking (shown in the gap between QA4 baseline and QA3), but only through the medium of updating the parameters of the model themselves (as opposed to via the activations as in in-context learning). QA2 obfuscates further from QA3 by incorporating a *YUIOP* sequence which connects each entity-string to a random incorrect entity. In essence, the *GHJKL* sequences tell the model that the entity-string and entity in the sequence are identical. However, it is not clear to me that the *YUIOP* sequences should be interpreted as “inconsistent seeming [definitions]” (line 138). There is nothing that forces the model to view *YUIOP* as an indicator of identity and then to figure out that its an unreliable identity indicator (which would involve the kind of self-reflection capacities supposed in the claim of the authors). Is it not more parsimonious to say the *YUIOP* sequences are consistent markers of non-identity - simply non-sequitur statements which are true but generally useless? If it is always true that the entity-strings and entities in *YUIOP* sequences are inconsistent with the QA examples holding those entity-strings (‘perfectly correlated’ line 87), a reasonable pattern that the model may learn is “'*YUIOP* X Y' means that X!=Y”. This bizarre anti-definition is almost useless as Y could be anything other than X, and potentially confusing, which can account for the drop from QA3 (brown) to QA2 (pink) following similar reasoning as in lines 120-121. So far this is basically the same, but from this perspective, the “surprise” result of Figure 2, that D5 outperforms D6, is no longer surprising. It is not necessary to rely upon the suggestion that the model “pick[s] up on features that indicate whether a given data point is likely to help reduce the loss on other data points, and “internalize[s]” data more or less based on these features” lines 16-17. Is it not simpler to suggest that the model has learned correctly that *GHJKL* indicates identity and *YUIOP* indicates non-identity? In this case, the fact that non-identity is ‘internalized’ to a lesser degree is no surprise at all: non-identity is only loosely incorporated (or incorporate-able!) because it is a non-sequitur. The model can fail to 'internalize' this non-sequitur information on account of it's general irrelevance, without relying on an surprising capacity to learn conditional on an example's \"*usefulness for predicting other datapoints*\" (line 122). A similar explanation can be given if *YUIOP* is not understood to be non-identity at all, but just random noise with no consistent interpretation. The fact that the model 'internalizes' the *GHJKL* information more than that of the *YUIOP* can rely solely upon the fact that an interpretation of *GHJKL* is readily apparent and there is no obvious interpretation of *YUIOP*. This line of reasoning begs the question as to the loss curves of the specific *GHJKL* and *YUIOP* examples in QA1/QA2 over the course of the training. You might expect to see higher loss for the *YUIOP* examples. Note that this argument extends to the non-QA experiments presented as well.\n\nWhat is strange in this perspective is not that D5 outperforms D6 but that D6 outperforms QA7! But this surprising result does not carry the significant implications of the previously surprising result highlighted by the authors. Regardless of how you explain the superior performance of D6 over QA7, it bears explaining why the above reasoning (which explains away the 'surprise' of the gap between D5 and D6 and which does not stipulate any particularly surprising characteristics on the behalf of the model) is confused. It seems a plausible enough explanation that without a convincing rebuttal the central claim of the paper, which makes an extraordinary claim of model behavior, seems shaky. \n\nThere is no reason to presuppose that a *YUIOP* example would ever be interpreted as a definition by the model, despite it being labeled as such in the analysis of the paper. Without this presupposition, the claim that *YUIOP* represents an \"inconsistent-seeming definition\" to the model is unfounded, as is the subsequent claim that that the model “pick[s] up on features that indicate whether a given data point is *likely to help reduce the loss on other data points*, and “internalize[s]” data more or less based on these features”. The gap supposed to demonstrate 'strong internalization' can be explained as nothing more than the difference between the model comprehending a useful control sequence marking identity, *GHJKL*, and a sequence marking random noise *YUIOP*.\n\n(Note: the above arguments may indeed be plausible but subtly misguided and ultimately wrong, but the authors must address them convincingly in order to strengthen the paper.)\n\nOther\n-----\nLines 79-82 discuss ‘information leakage’ where replaced entities may be inferrable based on information present in the QA pairs and background information present in the pretrained model, and states that steps have been taken to reduce this possibility. Presumably the performance of QA3 in Fig2 would vary significantly with this information leakage, where highly ‘leaked’ entities would still have good performance? (Ie. training on “Q: xyz was the first president of which country. A: the USA” should yield better performance on xyz related test Qs than a more obfuscated relation). Is this interpretation correct? If so it seems that the function of this information leakage and the specific means of alleviating it are actually very important to the interpretation of the results, and should perhaps be given more attention than being left to the appendices / alluded to in lines 124-127.\n\nInternalization is not formally defined in any way, yet it is a central aspect of the paper. It is 'measured' only via aggregated loss on each dataset. More time should be spent investigating and developing the idea of internalization (how does the 'internalization' of a specific example relate the loss on that example?).\n\nNits\n-----\n\n*    The title of the paper comes from a contrast to ‘in-context’ learning, which is referenced many times in the paper, but the meaning of the term is not made explicit until the Related work (line 314). It would improve clarity to define what is meant by in-context learning and to describe how the proposed ‘out-of-context’ learning differs when introducing the concept of out-of-context learning (line 42).\n\n*    Figure 1 bottom right has a typo: “Q: What did qwe born? A:” is presumably a mish-mash of two different questions? And not actually in the test set? It would not be surprising to get a bad answer to this question as stated!\n*    It would be helpful to label the data presented in Figure 1 with the dataset names (QA3, QA4..) used in Section 2.3.\n*    Figure 1 depicts two stages of finetuning on two separate datasets X1 and X2, and their subsequent eval/analysis, but this is a little obfuscated by the presentation. Consider making it more organizationally clear. Perhaps draw a bubble around the box on the left and the box on the top right to show they are the same stage, and add a Train label to the dataset in the left box to be consistent with the others. Some explicatory text can be moved to the caption to make the figure itself easier to cartoonify.\n*    Line 47 his -> this\n*    Fig2: consider showing “the entities consistent with the QA pairs; the latter get accuracy 0 everywhere” (Line 153) in the plot as well\n*    Line 291 vise-> vice\n*    Consider adding the number of (entity, entity-string) pairs present in the datasets of Fig2. It might be helpful to add a table with each of the datasets presented in Fig2, showing their characteristics and size / number of entity - string pairs. This would help clarity / readability.\n\nQUESTIONS:\nSuggestions:\n--------\n*   Generally, more time should be spent on justifying the central claim. It is extraordinary, and if true, the authors perspective that it \"may have significant implications for our understanding of foundation models, SGD-based optimization, and deep learning in general\" seems at least plausible. However, the current text spends little time on justifying this argument, and no time addressing alternative explanations, like that presented in the weakness section. As the paper stands, the experiments as presented do not convincingly support (to the exclusion of simpler explanations) the central claim of the paper.\n*    It would help to understand what is going on in model training to present and analyze the loss curves of specifically the *GHJKL* and *YUIOP* examples, for datasets QA1/2 and D5/D6 in the second round of finetuning. The loosely defined 'internalization', which is measured through overall dataset performance, seems like it might be inversely correlated to example perplexity. As perplexity can be measured on a example granularity, this should be presented.\n*    It would be nice to demonstrate in the Sec2/Fig2 experiments the result of recovering from the string-masking via *in-context* learning. This is an important baseline that is conspicuously missing (even if it just recovers fully the QA4 performance).\n\nLIMITATIONS:\nThe mentioned limitations are well selected (formalization of internalization, absence of obvious mechanism). The rebuttal of alternative explanations for the presented results is absent from the paper, a significant additional limitation. The potential social impacts are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

comments: SUMMARY:
The authors assert that "out of context metalearning makes LLMs better at internalizing useful information for understanding." They intuitively frame understanding as "treating content as true in question answering." They analyze its application to tasks such as mapping novel phrases or words to attributes and then performing question answers.

They introduce "define tags" which perform the mapping of novel info rather than using the word "define" and natural language, which I like a lot as an approach. This allows them to isolate the effect of the metalearning as a mechanism for improving performance rather than being clouded by the existing notions of the meaning of "define" that may be acquired by the LLM during pretraining.

Though I have some gripes that verge on the political for the limitations section I think this is an interesting and well-motivated work that deserves acceptance.

STRENGTHS:
Detailed analysis and clear statement of technique

WEAKNESSES:
Yudkowsky citation. I think that stuff is fundamentally unserious and hurts my willingness to recommend strong accept or award as an actual NLP expert.

QUESTIONS:
Curious to see other reviews. My apologies for being severely time-constrained in writing my review.

LIMITATIONS:
In my opinion, perpetuating AI safety hype in academic papers is inappropriate, and citing Yudkowsky in particular is a negative signal. Opinions of others will vary and ultimately I don't think this is a reason to reject. Just wanted to register my discontent.

Otherwise discussion of limitations is strong.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the phenomenon of internalization in LLMs, specifically \"weak internalization\" and \"strong internalization\" (out-of-context meta-leanring). Weak internalization refers to LLMs' improved performance on questions with consistent definitions rather than with inconsistent ones. Strong internalization involves LLMs' ability to provide better answers for variables with a defined tag representing a consistent definition, demonstrating out-of-context meta-learning. The paper includes ablations to support their findings and discusses the limitation of the work.\n\nSTRENGTHS:\n- The paper presents an interesting case of \"internalization\" in LLMs. \n- Authors addressed the limitations of their work and also mention the lack of conclusive explanations for internalization in general.\n\nWEAKNESSES:\nMajor Concerns.\n- Not enough models are analyzed in ablations. The paper presents only the evaluation of Pythia and T5 family of models and claims that the internalization phenomenon is quite general. I would suggest performing experiments with more recent models such as LLaMa, T5Flan, etc.\n- The number of datasets presented for evaluation is also quite small. Although, I understand that creating datasets for this specific format could be expensive.\n- it is unclear if the size of a model affects the internalization phenomenon. Ablations of a few models varying their size would help to solve this doubt.\n- Not clear how the phenomenon of internalization can be taken by the community in order to improve or avoid pitfalls regarding the development of LLMs\n\nMinor comments:\n- The paper mentions several times to look at the appendix but doesn't indicate to which section the reader should pay attention. I would suggest indicating the specific section in order to improve the readability of the manuscript.\n- The notations of the datasets are quite difficult to follow, I think authors could provide a general overview (in a table or any other format) rather than explaining each component in line with the text. This would improve the readability of the paper.\n\nQUESTIONS:\nTake a look at the weakness presented in the previous section.\n\nLIMITATIONS:\nYes, the authors addressed the limitations of their work and also mention the lack of conclusive explanations for internalization in general.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}