Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Authors: Stephanie C.Y. Chan
Adam Santoro
Andrew Kyle Lampinen
Jane X Wang
Aaditya K Singh
Pierre Harvey Richemond
James McClelland
Felix Hill
Venue: NeurIPS 2022 Accept (2022)
Keywords: in-context learning
few-shot learning
transformers

Abstract: Large transformer-based models are able to perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects of the training regime lead to this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having a large number of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in a wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In our initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models were unable to achieve both simultaneously. However, our later experiments uncovered that the two modes of learning could co-exist in a single model when it was trained on data following a skewed Zipfian distribution -- another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. Our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and indicate how future work might encourage both in-context and in-weights learning in domains beyond language. 

Data Distributional Properties Drive Emergent In-Context Learning in Transformers
Stephanie C.Y. Chan DeepMind
Adam Santoro DeepMind
Andrew K. Lampinen DeepMind
Jane X. Wang DeepMind
Aaditya K. Singh University College London
Pierre H. Richemond
James L. McClelland DeepMind, Stanford University
DeepMind
Felix Hill
DeepMind
Abstract
Large transformer-basse models able perform in-context few-shot learning, without being explicitly trained for it. This observation raises the question: what aspects the training regime lead this emergent behavior? Here, we show that this behavior is driven by the distributions of the training data itself. In-context learning emerges when the training data exhibits particular distributional properties such as burstiness (items appear in clusters rather than being uniformly distributed over time) and having large numbers of rarely occurring classes. In-context learning also emerges more strongly when item meanings or interpretations are dynamic rather than fixed. These properties are exemplified by natural language, but are also inherent to naturalistic data in wide range of other domains. They also depart significantly from the uniform, i.i.d. training distributions typically used for standard supervised learning. In initial experiments, we found that in-context learning traded off against more conventional weight-based learning, and models unable to achieve both simultaneously. However, later experiments uncovered that the two modes learning could co-exist single model when it was trained on data following skewed Zipfian distribution another common property of naturalistic data, including language. In further experiments, we found that naturalistic data distributions were only able to elicit in-context learning in transformers, and not in recurrent models. sum, our findings indicate how the transformer architecture works together with particular properties of the training data to drive the intriguing emergent in-context learning behaviour of large language models, and how future work might encourage both in-context and in-weights learning in domains beyond language
Introduction
Large transformer- basedd language models show intriguing ability perform in-context learning (Brown et al. 2020). This is the ability to generalize rapidly from few examples of a new concept which they have not been previously trained, without gradient updates to the model. In-context learning is a special case of few-shot learning in which the output is conditioned on examples from a 'context', and where there
gradient updates. It contrasts with 'in-weights' learning, which is
Codee is available at: http://gghhlccommmeeeppa///enene heta://meeeeeeatioooooooooeeeeae
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Model, inputs. and outputs.
(b) Sequences for training.
bursty
?
transformer (causal)
B21
218
C 218
821
A5
C
573
k
a21
O
rosnet embed
non-bursty
P
422
931
184
8
f
C
1008
C
image label
context
queer
context
query
(c) Sequences to evaluate in-context learning.
(d) Sequences to evaluate in-weights learning.
X
X
X
Z
Xo
Z
X
f
i1ss
h
A5
1ooce
821
415
121
$78
C
oc7
D
context
queery
context
query
Figure 1: Experimental design, described in Section a For each experiment, transformer
model is trained on sequences of image-label pairs. The model trained to minimize the loss on predicting the label corresponding the final 'query' image. (b) In training, image-label mappings are fixed across sequences, in contrast to few-shot meta-training, The training data consist of a mix of "bursty" and 'non-bursty' sequences. Bursty sequences, featuring multiple occurrences of the same classes, can be solved by learning labels across sequences (in-weights learning). or referring back to the context (in-context learning). Non-bursty sequences were composed of i.i.d. images. (c) To evaluate few-shot in-context learning, the model is presented with a standard few-shot sequence. The holdout image classes were never encountered in training, and are randomly assigned to labels [0.1]. Thus the model must use the context to predict the query label. ( To evaluate in-weights learning, the model presented with sequences where the labels are the same as training. However, the query class does not appear in the context. Thus, the model must used information stored in weights predict the query label. In the example sequences, we add colors and use only Latin characters for visualization purposes.
the standard setting for supervised learning this is slow (requiring many examples), and depends on gradient updates to the weights. Earlier work in the context of mneta-learning' showed how neural networks can perform few-shot learning without the need for weight updates (Santoro et al. 2016 Vinyals al. 2016 Wang et al.] 2016). To achieve this, the researchers explicitly designed the training regime to incentivize in-context learning, process sometimes called metaa-rrining"" In case of transformer language models, however, the capacity for in-context learning emergent. Neither the model's transformer architecture its learning objective explicitly designed with in-context learning in mind.
Here, we consider the question of how transformer language models are able to acquire this impressive ability, without being explicitly targeted by the training setup learning objective. The emergence of in-context learning in language models was observed as recurrent models were supplanted by transformers, e.g. in GPT3. Was the novel architecture the critical factor behind this emergence? In this work we explore this possibility, as well as a second: that capacity for in-context learning depends on the distributional qualities of the training data.
This hypothesis was inspired by the observation that many natural data sources including natural language differ from typical supervised datasets due few notable features. For example, natural data is temporally "bursty". That is, a given entity (word, person, object, etc) may have distribution that is not uniform across time, instead tending to appear in clusters (Altmann et al.) 2009 Al Alaarez
Lacalle et al.) 2006; Lambiotte et al., 2013 Neuts 2007: Sarkar et al.) 2005 Serrano et al.) 2009).
Natural data also often has the property that marginal distribution across entities is highly skewed. following Zipfian (power law) distribution with a long tail of infrequent items (Piantadosi] 2014
Smith et al. 2018 Zipf. 1949). Finally, the 'meaning" entities natural data (such as words in natural language) is often dynamic rather than fixed. That is, single entity can have multiple possible interpretations (polysemy and homonymy, language) and multiple entities can map the same interpretation (synonymy, in language), usually in context-dependent way. The combination

---

(a) In-context learning holdout classes.
(b) In-weights learning on trained classes.
0.8
0.8
0.0
0.5
0.6
0.6
chance
0.2
0.2
0.0
0.0
chance
train steps
165
train steps
le5
Figure 2: Effects of burstiness. P(bursty) indicates the proportion of training sequences that were bursty vs non- non-bursty, and models are evaluated on the two types of evaluation sequences, over the course of training. Burstiness in the training data increases in-context learning, and decreases in- weights learning. Also, over the course of training, in-context learning tends to decrease while in- weights learning increases.
of these properties may result in training data that occupies some middle-ground between the data used in canonical supervised learning and that used for few-shot meta-training.
In particular, standard supervised training typically consists of item classes that recur with uniform regularity, and with item-label mappings that are fixed throughout training these properties allow a model to gradually learn over time, by encoding information into its weights, e.g. via gradient descent. By contrast, few-shot or in-context meta- meta-training generally involves training a model directly on specially crafted sequences of data where item classes only recur and/or item-label mappings are h
only fixed within episodes they do not recur and are not fixed across episodes (Santoro et al. 2016 Vinyals et al. 2016). Naturalistic data, such as language or first-person experience, has characteristics of both of these data types. As in supervised training, items (words) do recur, and the rela relationship between an entity and its interpretation (or meaning) is fixed, to some degree at least. At the same time, the skewed and long-tailed distribution of natural data means that some entities recur very frequently while a large number recur much more rarely. Importantly, however, these rare items are often bursty, making them disproportionately likely to occur multiple times within given context window, somewhat like a sequence of 'meta-training' data. We can also see the dynamic relationship between entities and their interpretation (epitomized by synonyms, homonyms, and polysemy, in the case of language) as weaker versions of the completely dynamic itbellabel mappings that are used in few-shot meta-training, where the mappings are randomly permuted every episode.
In this paper, we experimentally manipulated the distributional properties of the training data and measured the effects in-context few-shot learning. We performed our experiments over data sequences sampled from a standard image-based few-shot dataset (the Omniglot dataset; Lake et al. 2019). At training, fed each model (such transformer recurrent network) with input sequences of Omniglot images and labels, varying the natural data-inspired distributional properties of choice. At evaluation, assessed whether these properties gave rise to in-context learning abilities.
Our results showed that, indeed, in-context learning emerges in a transformer model only when trained on data that includes both burstiness and a large enough set of rarely occurring classes. We also tested two instantiations of the kinds of dynamic item interpretation observed in natural data- having many labels per item well within-class variation. We found that both interventions on the training data could bias the model more strongly towards in-context learning. The models we tested typically exhibited tradcoff between rapid in-context learning relying information that was stored through slow, gradient-basee updates ("in-weights" learning). However, we found that models could simultancously exhibit both in-context learning and in-weights learning when trained on a skewed marginal distribution over classes (akin to the Zipfian distribution natural data).
At the same time, architecture is also important. Unlike transformers, recurrent models like LSTMs and RNNs (matched on number of parameters) were unable to exhibit in-context learning when trained on the same data distribution. It is important to note, however, that transformer models trained on the wrong data distributions still did fail to exhibit in-context learning. Thus, attention not all you need architecture and data are both key to the emergence of in-context learning.
3

---

(a) In-context learning holdout classes.
(b) In-weights learning trained classes.
training
0.8
0.0
1600
0.6
12800
0.4
0.4
0.2
0.0
chance
train stops
1e5
train
1e5
Figure 3: Effects of number of classes. Increasing the number of training classes improves in-context learning, while reducing in-weights learning.
2 Experimental Design
2.1 The training data
To investigate the factors that lead to in-context few-shot learning, we created training and evaluation sequences using the Omniglot dataset (Lake et al., 2019, MIT License), a standard image-label dataset for few-shot learning. Omniglot consists of 1623 different character classes from various international alphabets, with each class containing 20 handwritten examples. Using the Omniglot dataset allowed us to apply evaluation procedures that are standard in the study of few-shot learning. classify an example of a character class that was never seen in training,
The few-shot challenge based only on few examples of that class and some alternate classes.
The training data consisted of sequences of images and labels (Fig lb The first 16 elements of each sequence comprised the 'context', and consisted of 8 image-label pairs (where each image was always followed immediately by its corresponding label). The final element was the "query' image, and the aim of the model was to predict the correct label for the query.
Images were allowed recur throughout training, and the integer label for each image class was unique and fixed across training, as in typical supervised datasets. We emphasize that this is major departure from conventional few-shot training, where item-label mappings completely novel on each episode, or the items themselves are novel on each episode.
In our standard experiments, we trained the model on a mixture of 'bursty' and 'non-bursty' sequences. In the bursty sequences, the query class appeared 3 times in the context. There are many possible ways to quantify or instantiate burstiness (c.g., Altmann et al. (2009) Alvarez-Lacalle et al. 2006; Lambiotte ct al. 2013 Neuts 2007| Sarkar et al. 2005 Serrano et al., 2009), but the 'bursty" sequences in our experiment were designed to reflect the within-context burstiness that is observed in e.g. language. To prevent the model from simply outputting the most common label in the sequence, a second image- label pair also appeared 3 times in the context. For the non-bursty sequences, the image-label pairs were drawn randomly and uniformly from the full Omniglot set. We can continuously vary the overall degree of burstiness in dataset by changing the proportion of 'bursty" 'non-bursty' sequences.
2.2
The model
Each element of a sequence was first passed through an embedder (a standard embedding layer for the integer labels, and ResNet for the images; He et al. 2015). These embedded tokens were passed into a causal transformer model (Fig la Vasswani et al. 2017). Unless stated otherwise, we used a transformer with 12 layers and embedding size 64. The model was trained on a softmax cross-entropy loss on the prediction for the final (query) image.
2.3 The evaluation data
We evaluated trained models on two types of sequences, measure (1) in-context learning and (2) in- nn-weights learning. As in the training sequences, the evaluation sequences also consisted pairs of 'context' image and label tokens, followed by single 'query" image token.

---

To measure a trained model's 's ability for in in-context few-shot learning, we used standard few-shot setup. The context consisted of random ordering 2 different image classes with 4 examples each, and the query was randomly selected from one of the two image classes (a '4-shot 2-way' problem, in few-shot nomenclature). Unlike in training, where the labels were fixed across all sequences, the labels for these two image classes were randomly re-assigned for each sequence. One image class was assigned to O, and the other to (Fig Id.. Because the labels were randomly re-assigned for each sequence, the model must use the context in the current sequence in order to make a label prediction for the query image. Unless stated otherwise, in-context learning was always evaluated on holdout image classes that were never seen in training.
Although the model is always required to perform a full multi-class classification over all possible output labels (as in training), few-shot accuracy is computed by considering the model outputs only for the two labels seen the few-shot sequence (0 and 1), with chance at 1/2. This ensures that performance above chance cannot be due to e.g. randomly selecting one of the labels from the context. Note also that the model was evaluated for in-context learning on novel image classes, but not novel labels (see the appendix for further discussion).
To measure in-weights learning of trained classes in a model, evaluation sequences consisted of image classes that were selected uniformly without replacement, with the same labels that were used in training (Fig [d]. Because the image classes were forced be unique within each sequence, the query had no support in the context. Thus, the only way for a model to correctly predict the label was to rely on information stored in the model weights. For this problem. where the correct query label could be any of the labels seen in training, chance was usually 1 1/1600.
3 Results
3.1 What kinds of training data promote in-context learning?
Burstiness. In our first experiments, we vary levels of burstiness in the training data by varying the proportion of bursty vs non-bursty sequences the training data (as described in Section 2.1). These experiments replicate the finding that transformers can acquire in-context few-shot learning even without explicit meta-training. They further show that, as hypothesized, the model displays better in-context learning with more burstiness in training (Fig 2ap. We also see that in-context learning trades off against wn-weights learning - greater burstiness simultancously leads lower weight- based learning (Fig 26). Interestingly, the models can in some cases lose an initial bias towards in-context learning, moving towards in-weights learning over the course of training.
A large number of rarely occurring classes. Our second set of experiments show that in-context learning performance depends on the number of training classes (keeping the level of burstiness fixed at p(bursty) = 0.9). As we increase the number of classes from 100 to 1600 (and correspondingly decrease the frequency of each class), we see improvement of in-context learning (Fig Bap. As before, we also see an accompanying decrease in in- weights learning (Fig 3D This accords with our hypothesis about the importance of having long tail in the distribution, or a large vocabulary. Note that the bias against in- in-weights learning cannot be explained by the number of exposures to each class even controlling for the number of exposures, the model trained with 1600 classes is much slower to achieve similar levels of in- in-weights learning. Importantly, we need both burstiness and a large number of classes for in-context learning to emerge. In order to further increase the number of classes beyond the 1623 available in the original Omniglot dataset, we rotated (0), .90,, 180", 270") and flipped (left-right) the images, obtaining 8x more image classes. We ensured that the holdout set did not include transformed versions of train images. Training on these 12800 classes further improved in-context learning (and reduced in- weighis learning) (Fig 3 However, some images in Omniglot have rotational or mirror symmetries, so that the models trained on 12800 classes may additionally be pushed towards in-context learning by a nubel-mulipllicity effect, described next.
Multiplicity of labels. Our third set of experiments explored the effect of dynamic meanings, with training distributions where images did not have completely fixed labels. Each image class was assigned to multiple possible labels and, in the data sequences, the label shown after each image was randomly selected among the possible labels. If a class appeared more than once in the same sequence, the label was consistent for all presentations within that sequence (this is commonly the case in natural data such as language, too; Gale et al. 1992). In Fig 4 we see that increasing the 'label
5

---

multiplicity" (the number of labels per class) also increases in in-context learning. Again, burstiness was fixed for these experiments at p(bursty) 0.9.
(a) In-context learning holdout classes.
1.00
1.00
0.1
1.00
0.5
1.00
Full Omniglor
Pibursty)
0.75
1.0
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
165
1.5
(b) In-weights learning on trained classes.
1,00
Pixel noise
1.00
Pixel noise 0.1
Pixel noise 0.5
Full Omniglat
Pibursty)
0.75
0.9
acouscy
0.50
0.50
0.50
0.50
0.25
0.00
0.00
0.00
50805
stops
train stops
train stops
Increasing
variation
Figure 5: Effects of within-clan variation. When we increase the within-class variation (from left to right), in- in-context learning tends to increase (a) while in in-weights learning decreases (b) Both effects are nonetheless upper-bounded by the difhculty of within-class generalization, with the 'Full Omniglot' problem being more difficult than the rest. For the 'Full Omniglot" experiments, each
class contained the full set of 20 Omniglot exemplars per class. For the remaining experiments, each consisted of only a single Omniglot exemplar image, with varying levels of Gaussian pixel noise.
Within-class variation. We then explored another source of dynamic variation of meaning - the amount of variation within image classes themselves. In the lowest- variation condition, each image class con- sists of only a single image, i.c. the images for given class were always identical. In the medium-variation Gaussian pixel noise the images (resam- pled for each presentation). In the high- varation condition, we used the full Om- niglot classes (each class consists of 20 dif- ferent images drawn by 20 different peo- ple). To our surprise, found that greater within-class variation leads to greater in- In other words, making the generalization problem harder actually made in-context learning emerge more strongly - preferentially hampered in-weights learning more than it hampered in-context learning.
1.0
Label multiplicity
0.8
accurary
0.6
10
0.4
0.2
variation conditions, we added
0.0
A train steps
le5
Figure 4: Dynamic meanings improve in-context learn- ing. Increasing the number of labels per class ('label multiplicity") increases in-context learning.
context learning (Fig 5).
Across the above experiments (Figs 2 we also evaluated in-context learning on training classes (rather than holdout classes), again randomly assigning the classes to labels 0 and I (rather than using the ones seen in training). Evaluations looked similar in all cases, with only slightly higher performance (Appendix C3

---

(a) Examples of Zipfian distributions.
(h) Distribution of tokens in natural language corpus.
1e-8
1.25
Zipf exponent
0.06
frequency
1.00
0.0
0.05
0.5
1.0
0.04
0.50
0.03
2.0
0.01
0.00
Iury
stupefying
10a
10a
10a
10a
10*
class rank
10a
10a
10a
10
10
taken rank
(c) In-context learning on holdout (d) In-weights learning on classes
(e) In-weights learning on rare classes.
common classes.
1.00
1.00
1.00
accuracy
0.75
accurace
0.75
chance
0.25
0.25
0.25
0.00
0.0
0.5
1.5 ZIpt exponent
1.0
2.0
3.0
0.00
0.0
0.5
1.0
1.5
2.0
3.0
chance
0.00
0.0
0.5
1.5 Zipt exponent
1.0
2.0
3.0
chance
Zipt exponent
Figure 6: Effects of training on Zipfian (rather than uniform) marginal distributions over classes. Examples of Zipfian distributions with varying exponents. (b) The distribution of tokens in an example English-language corpus. In C bars indicate mean evaluation accuracy in the window [400k, 500k] steps of training-
O As we increase the Zipf exponent, i.e. increasing the skew on the class distribution, we see decrease in in-context learning. (d) In-weights learning of the 10 most common classes, in contrast, increases with more skew. With uniform training (Zipf exponent 0), the model exhibits only in-context learning and not in in-weights learning. However, if we train on skewed distributions, there is sweet spot where both in-context learning and in-weights learning can be maintained at high level in the same model (Zipf exponent 1, for this particular training regime). Coincidentally, a Zipf exponent of corresponds approximately to the skew in many natural languages. (e) Rare items from training are never memorized (performance is at chance for all Zipf exponents).
3.2
What kinds of training data enable in-context learning and in-weights learning to co-exist in the same model?
In the previous section, learning- no models could maintain both. However, it is useful for a model to have both capabilities to remember information about classes that will re-appear in evaluation, while also being able to perform rapid in-context learning on new classes that appear only in holdout. Large language models certainly do have both of these capabilities. How might achieve this?
a consistent tradeoff between in-context learning and in-weights
For all prior experiments, the training data were marginally distributed uniformly over classes, even the data were non-uniform in other ways. Le., each class was equally likely to appear, marginalizing across the dataset. We postulated that we might achieve both types of learning in the same model by instead training
marginally-skewed distributions. In this case, some classes appear very commonly, while most classes appear very rarely. Many natural phenomena such as word distributions take this form, and classically described as Zipfian (power law) distribution ZZipi 1949):
x(X= =
(1)
Here, X is the rank of the class (c.g. for the most common class). and the exponent a E [0, oo) determines the degree of skew. Fig ba shows some examples of Zipfian distributions with various exponents. Thg]bblshe shows an example of token distributions in English (from the Brown corpus; corpus;Francis Francis and Kucera 1979) This type of skewed distribution could allow a model learn common classes in its weights, while the long tail of rare classes simultaneously induces an ability for in-context learning.
Plot generation adapted from http:://gg.ggiihbb.ooom//hela//a

---

To test this hypothesis, we trained on Zipfian distributions, varying the Zipf exponent and hence the degree of skew. We used the same training sequences as before, with 12800 classes and p(bursty) 0.9. Our results are shown in Figs 6c We evaluate in-weights learning separately on common classes (the 10 classes seen most often training) and on rare classes (the remaining classes). When there no skew, all classes are relatively rare, and we see high levels of in-context learning but no in-weights learning. Increasing the skew leads to the loss of in-context learning and increased in- in-weights learning of common classes In between the two extremes, we observe sweet spot at Zipf exponent 1, where the model maintains high levels of both in-context learning and in-weights learning of common classes. Intriguingly, natural languages are best described by Zipfian distribution with exponent of approximately (Piantadosi 2014). Note though that the sweet spot for simultaneously maintaining in weights and in-context learning in transformers may differ, depending the training regime.
(a) Transformer.
(b) Vanilla RNN.
(c) LSTM.
1.0
1.0
1.0
accuracy
0.6
0.6
0.6
chance
30.4
0.4
0.2
0.0
2
0.0
2
0.0
train stops
1e5
8 train steps
train
105
Figure 7: In-context learning in transformers vs. recurrent architectures. We compare architectures while holding fixed the number of layers, hidden layer size, and number of parameters. Only a transformer is able to attain in-context learning; the Vanilla RNN and LSTM never perform above chance. One run was performed for each set of hyperparameters in a hyperparameter sweep. Each color denotes one run, but not any particular hyperparameter values.
3.3 But architecture does matter too.
To investigate whether these results are specific to transformer models, we performed similar experi- ments using recurrent sequence models. For these models, we simply replaced the transformer with either a vanilla recurrent neural network (RNN: (RNN:David E. Rumelhart et al. 1985) or long short-term memory network (LSTM: Hochreiter and Schmichuber 1997). We used the same training sequences as before, with 1600 classes and p( bbursty) = 0.9. We also used the same image and label encoders, and cross-entropy classification loss. The recurrent models were matched to the transformer for depth, number of parameters, and hidden layer size. We performed a comprehensive hyperparameter search for all models (see Appendix for details).
In these experiments, we see that the recurrent models are never able to achieve in-context learning, despite the parity in training setup (Fig 7 Interestingly, the transformer actually outperforms the recurrent models on in-weights learning as well (see Fig X in the Appendix), indicating that we cannot explain these results by proposing that recurrent models are simply more biased towards in-weights learning than transformers.
4 Discussion
In summary, we find that both data and architectures contribute significantly to the emergence of in-context learning in transformers.
Data properties that promote in-context learning. We identify several features of training data that can promote in-context learning- - burstiness, number and rarity of training classes, and dynamic meaning (as instantiated by multiple labels per class or within-class variation). These data properties allow in-context learning to emerge despite differing significantly from the data used in standard few-shot meta-training, in that we allow items and item-label mappings to recur throughout training. These properties are also central features of natural data including language, and thus may explain the remarkable emergence of in-context learning in large language models without explicit meta- training.
We see decreased in in-weights learning for Zipf exponent = 3, because that level of skew leads to extreme focus on tiny number of classes (e.g. the three most common classes form 97% of the data).
8

---

Effects of architecture. We find that architecture does matter as well. Transformers show significantly greater capacity for in-context learning than recurrent models we were completely unable to elicit in-context learning in recurrent models, even with the training procedure, number of parameters, and model architecture otherwise matched to the transformer experiments. We emphasize however that the transformer architecture alone was insufficient for eliciting in-context learning it was necessary for the training data to exhibit at least burstiness and large numbers of classes, too.
In-context vs. in-weights learning. In most cases, we found that transformers exhibited a tradeoff in their bias towards either in-context learning or in-weights learning, and could not maintain both in the same model. We characterize this behavior as a 'bias', because neither type of learning is 'correct' per se. For our training data, an in-context learning strategy and an in n--wiights learning strategy will give the same answer, since the labels are fixed. Thus. in the in-context evaluation sequences, it is ambiguous (by design) whether the model should use the labels seen in training or in the current context, allowing us to measure the model's bias. We also note that even models with an initial bias towards in-context learning can often move towards in-weights learning with enough repetition.
However, it is often important and useful for a model to exhibit both capabilities - to perform slow, gradient-based in-weights learning of class information that is presented during training, while also being able to quickly learn (without weight updates) about new classes that appear only in evaluation. Indeed, large language models exhibit both of these capabilities (Brown et al. 2020). In our experiments, we discovered that an additional language-like distributional property could allow models maintain both capabilities as well a skewed, Zipfian distribution over classes. This allowed the models to retain information in their weights about common classes, while simultaneously developing in-context learning abilities that were presumably induced by the long tail of rare classes.
Implications for understanding language models. Our findings have a few noteworthy implications. First, by pointing specific distributional properties of training data that both exist in language and also promote in-context learning, these results may help us reach a more scientific understanding of why in-context learning emerges in transformer-based language models. This is area of increasing
interest (e.g. Min et al. 2022 Razeghi et al. 2022; Webson and Pavlick 2021 Xie et al. 202Ib.
We emphasize that the transformers in our experiments successfully performed in-context evaluation
on holdout classes, and only performed slightly better with in-context evaluation on trained classes. These results counter to an emerging narrative that large language models may not actually be performing genuine in-context learning, and simply draw on examples seen in training (Min et al. 2022. Razeghi al. 2022 Xie et al., 202T) our experiments show that naturalistic distributional properties can give rise to a capacity for in-context learning on classes that were never seen in training.
Broader implications. This understanding may also help design and collect datasets to achieve in- context learning in domains outside of language, an area of ongoing research (e.g. Finn et al.) 2017; Hill et al. 2020: Wang et al. 2016). Given that reinforcement learning environments are generally designed to be uniformly distributed (Chan et al., 2022). or that supervised datasets are frequently rebalanced to have more uniform distributions (Chawla et al. 2002 Katharopoulos and Fleuret 2019
Van Hulse et al. 2007), we may be missing an opportunity to endow non-language models with a powerful capability. We may need to consider data distributions more carefully when pre-training in non-language domains, as well. For example, recent work has shown that pre-training on language data was useful for offline reinforcement learning, but pre-training on vision data was not (Reid et al. 2022) - could this difference be due to the non-uniform, structured distribution of the language data?
Cognition and neuroscience. Our experiments could also potentially inspire research on the role of non-uniformity in human cognitive development. Infants rapidly learn statistical properties of language (Saffran and Kirkham 2018) could these distributional features help infants to acquire an ability for rapid learning, or serve as useful pretraining for later lcarning? And could non-uniform distributions in other domains (e.g., vision) also contribute to this development (cf. (Smith al. 2018?
Our results may also relate to complementary learning systems theory (Kumaran et al. 2016 Mc Clelland and O'Reilly, 1995) and its application to language understanding in the brain McClelland et al. 2020). According to this theory, the neocortical part of the language system bears similarities to the weights of neural networks, in that both systems learn gradually through the accumulated in- fluence of large amounts of experience. The hippocampal system plays a role similar to the context window in transformer model. by representing the associations encountered most recently (the hip-

---

pocampus generally has time-limited window; Squire 1992) p. In this light, it is possible to see the human hippocampal system as a system that provides the architectural advantage of the transformer's context representations for in-context learning.
Future directions. The above results suggest exciting lines of future research. How do these data distributional properties interact with reinforcement learning vs. supervised losses? How might results differ in experiments that replicate other aspects of language and language modeling, e.g. using symbolic inputs, training on next-token or masked-token prediction, and having the meaning of words determined by their context? For models that display both in-context and -n-weights learning, it would be interesting to understand contextual cuing of already learned information does this increase with more exposure? There is also a lot more to understand about the behaviors and biases of transformers vs. recurrent architectures why do transformers seem to be more capable of in-context learning?
Non-uniformity. Finally, hope emphasize the dual nature of non-uniformity in training data. While it can impair both supervised and reinforcement learning (Chan et all, 2022 Van Hulse et al. 2007), we show here that non-uniform training distributions can induce the emergence of at least one useful and interesting capability, and thus can be an opportunity as well as challenge.
Acknowledgments and Disclosure of Funding
We would like to thank the following colleagues for invaluable feedback and discussion: Kris Cao, Toni Creswell, Kevin Miller, Ivana Kajic, Andrea Banino, Ishita Dasgupta, Kenneth Marino, Irina Higgins, Murray Shanahan, Kyriacos Nikiforou, Richard Evans, Christos Kaplanis, David Reichert, Dave Abel, and Drew Hudson.
We would also like to thank the following colleagues for their contributions to the transformer imple- mentation: Igor Babuschkin, Junyoung Chung, David Choi, Tamara Norman, Sebastian Borgeaud, Jack Rae, David Saxton, Yujia Li, Phil Blunsom, Maribeth Rauh, Roman Ring, Nate Kushman, Vini- cius Zambaldi, Tom Hennigan
This work was funded by DeepMind.
References
Eduardo G. Altmann, Janet B. Pierrehumbert, and Adilson E. Motter. Beyond Word Frequency: Bursts, Lulls, and Scaling in the Temporal Distributions of Words. PLoS ONE, 4(11):e7678, November 2009. ISSN 1932-6203. doi: 10. 1137///omall.one pttp://aa..o..000077655..0. 0007600. uttpttss::/ plos org/10..377 1371/jounal pone. 0007678
E. Alvarez-Lacalle, B. Dorow, J.-P. Eckmann, and E. Moses. Hicrarchical structures induce long-range dynamical correlations in written texts. Proceedings of the National Academy of Sciences, 103(21): 7956-7961, May 2006. doi: 10. 10.1073//nas.00 10673103. URL|nttps: //ww://ww/////// . pnas. org/doi/ abs/ 10. 10w31007/ppa 1517/ Publisher: Proceedings of the National Academy of Sciences.
0510673103
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Verbert-Vosss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen. Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models Few-Shot Leamers. arXiv:2005. 14165 [cs]. arXiv: 1005.14165.
July 2020. URL http::// /atpv..ivvv..//..//eeae/aaaaa..a www.w.ww.ww......a..eoweeaaaaaae
14165
Stephanie C. Y. Chan, Andrew K. Lampinen, Pierre H. Richemond, and Felix Hill. Zipfian en- vironments for Reinforcement Learning. 2rXiv:2203.08222 [cs], March 2022. URL http! arXiv: 2203.08222.
//arxiv. Sarxiv.org/ae abs/ 08222
"While the hippocampal system is thought store recent context information in connection weights, whereas transformers store such information directly in their state representations, there now a body of work pointing out the quantitative and computational equivalence of weight- and state-based representations of context state for query- based access to relevant prior information (Krotov and Hopfield 2021 Ramsauer et al. 2021) as implemented in transformers.
10

---

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16:321-357, June 2002. ISSN 1076-9757. doi: 10.1613/jair.553 URL http: /stp://arxiv..o www.:/iwwrr/abb/i//// 1813. arXiv: 1813.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representa- tions by emror propagation. Technical report, 1985. httpss//://ppp dtic www.mmmmmc/////////
fulltext/u2/a1644455 p
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agmostic Meta-Laaming for Fast Adaptation of Deep Networks. 1:X.iv:703.03444 [es], March 2017. URL http::/arxxvv.rr//bbea.. 08400, arXiv: 1703.03400.
W. Nelson Francis and Henry Kucera. Brown Corpus Manual, 1979. URL http://korpus... uib www.caamebbooo/beee html
William A. Gale, Kenneth W. Church, and David Yarowsky. One Sense Per Discourse. In Speech and Natural Language: Proceedings of a Workshop Held Harriman, New York, February 23-26, 1992, 1992. URL httph:t/t///a http:::::/aalaaaahllooooooo
org/ Fwrg/992-100 1045
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 45X1v:1512.03385 [cs], December 2015. URL 03385 arXiv: 1512.03385.
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzie, and Stephen Clark. Grounded Language Learning Fast and Slow. arXiv:2009.01719 [cs], October 2020. URL arXiv: 2009.01719.
http://arxiv.orr wwwp:orggggr...r.rrrr 01719
Sepp Hochreiter and Jürgen Schmidhuber Long Short-Term Memory. Neural Computation, 9(8): 1735-1780, November 1997. ISSN 0899-7667. doi: 10. 16216662...1100 19tecc..199..99 10.77998.177.. Conference
Name: Neural Computation.
Angelos Katharopoulos and François Fleuret. Not All Samples Are Created Equal: Deep Learning with Importance Sampling. arXiv:1803 00942 [cs], October 2019. URL http:://axixa: org/ arXiv: 1803.00942.
Dmitry Krotov and John Hopfield. Large Associative Memory Problem in Neurobiology and Machine Learning. arXiv:2008. 06996 [cond-mat, q-bio, stat]. April 2021. URL httpp://axxiv org/ 06996,
arXiv: 2008.060 06996.
Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated. Trends in Cognitive Sciences, 20(7):512-533 July 2016. ISSN 13646613. doi: 10.1016/j.tics.. http:///..cs.2211..000 2016.05.004, URL https
//linkinghub. elsevier cww http:////w...ereeeeee//e//////aa
Brenden M. Lake, Ruslan Salakhuudinov, and Joshua B. Tenenbaum. The Omniglot Challenge: A 3-Year Progress Report.
03477 [es]. February 2019. Utt.lttpp://arzit wrr. aba/ arXiv: 1902.03477.
03477
Renaud Lambiotte, Lionel Tabourier, and Jean-Charles Delvenne. Burstiness and spreading on temporal networks. The European Physical Journal B, 86(7):320, July 2013. ISSN 1434-6028, 1434-6036. doi: 10.1 1..11444i///22133444455 arXiv:1305.0543 [physics, q-bio].
URL
/00e.///05...0 0543
James L McClelland and Randall C CO'Reilly. Why There Are Complementary Learning Systems in the Hippocampus and Neocortex:Insightsffii the Successesand Failuresof Connectionist Models of Learning and Memory. page 39, 1995.
James L. McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. Placing language an integrated understanding system: Next steps toward human-level performance in neural language models. Proceedings of the National Acudemy of Sciences, 1 11744)):2566-25574 October 2020. doi: 10. 20.11073pnss.1990 1910416117. URL htl.httts::// pestps://wwzr //www, pnas. org/doi/abs/u 1073/pnas 1910416117
Publisher: Proceedings of the National Academy of Sciences.

---

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In- Contett Learning Work? arXiv:2202. 12837 [cs], February 2022. URL StRIhttp::/ http:://axiii http///xxvv...r/bbb..e. httggabbs/202... 2202. 12837 7202.12837,
arXiv:
Marcel F. Neuts. The burstiness of point processes. Stochastic Models, April 2007. doi:
10. 00.1080015322344308877222 URL https://vir wttp:.taadfoolingg httpm//oi/abb/// 1080/
15326349308807275 Publisher: Marcel Dekker, Inc.
Steven T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112-1130 October 2014. ISSN 1069-9384. 3423-014-0585-6
doi: 10.3758/s1 114.7758k13423---4-0055553 URL hwt.::ttps////et hwwp//wwww..o ncbi nlm. nih. gow/
PMC4176592/
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp. Günter Klambauer. Johannes Brandstetter, and Sepp Hochreiter. Hopfield Networks is All You Need. arXiv:2008.0222. 02217 (cs, stal], April 2021. URL http: arXiv: 2008.02217.
www.//gg////aaaaaeeeeooeee //arxiv. www. ablaabaa20000 2008 02217
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of Pretraining Term Frequencies on Few-Shot Reasoning. February 2022. URL https://arxiv.orrrt wtw.//////.///////////////
07206v1
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can Wikipedia Help Offline Reinforcement Learning? arXiv:2201. [cs], January 2022. URL http::/arxiv.org// arXiv: 2201.
12122
Jenny R. Saffran and Natasha Z. Kirkham. Infant Statistical Learning. Annual Review of Psychology, 69(1):181 20301 2018. doi: 10.1 10..144//anurex-psyee pst.--112ccc11-111111111
URL https://dol org/
10. 1241114/aaa annurev ant.//nnuuueecchheeeeee pssychh 122216- 011805 122216-011805.
_eprint: https://doi.org/10 or0/10. www.oo1/aannreeeps/.....
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- Learning with Memory-Augmentee Neural Networks, page 9, 2016.
Avik Sarkar, Paul H. Garthwaite, and Anne De Roeck. A Bayesian mixture model for term re- occurrence and burstiness. In Proceedings of the Ninth Conference on Computational Natural Language Learning CONLL '05, page 48, Ann Arbor, Michigan, 2005. Association for Computa- tional Linguistics. doi: 00.3115/1705543.17066 1706552. http://://ppotal acm. org/ citation
cfm77doid=1705433 1706552
M. Angeles Serrano, Alessandro Flammini, and Filippo Menezer. Modeling Statistical Properties of Written Text. PLOS ONE, 4(4): April 2009. ISSN 1932-6203. doi: 10.1371/joumaal pone. 0005372. URL httpss:/jjurnall plos wwwps.orgppol...ooooo http/p/osone//rticceeeoe p 0005372
1371/journa]
Publisher: Public Library of Science
Linda Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. The Developing Infant Creates a Curriculum for Statistical Learning. Trends in Cognitive Sciences, 22(4):325-336, April 2018. ISSN 1364-6613, doi: 10.101 101 ttt.:/itt......................a 801880000000000 02.00. hRLhttts:://ww http://wwww.ciiccccceeeee.eteee. com/ Beciaacceaattiicee////////666ee.
Larry R. Squire. Memory and the Hippocampus: A Synthesis From Findings With Rats, Monkeys, and Humans. Psychological review, 1992.
Jason Van Hulse, Taghi M. Khoshgoftaar, and Amri Napolitano. Experimental perspectives on learning from imbalanced data. In Proceedings of the 24th international conference on Machine learning, ICML '07. pages 935-942, New York, NY, USA. June 2007. Association for Computing Machinery. ISBN 978-1-595933793-3 doi: 10. 11.114///273496.12766666
URL Ihttps:///at
prg/10.2g /145/12734200 1273614
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. page 11, 2017.
12

---

Oriol Vinyals, Charles Blundell, and Timothy Lillicrap. Matching Networks for One Shot Learning. page 2016.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. 1rx1vv:61..05760 [cs, stat], November 2016. URL httt:://arxxit http:://xaxxii...g/abbbbb// herg/bbs/61111 05763
arXiv: 1611.05763.
Albert Webson and Ellie Pavlick. Do Prompt-Based Based Models Really Understand the Meaning of their Prompts? urXiv:2109.0124 [cs], September 2021.
http:////aarriii.raaaa///aaaa/// 01247
arXiv: 2109.01247.
Sang Michael Xie, Aditi Raghunathan, Perey Liang, and Tengyu Ma. An Explanation of In-context Learning as Implicit Bayesian Inference. arXiv:2111.02088 [cs], December 2021. URL http: wawxi.oorge.... ab11 02080 arXiv: 2111.02080.
George Kingsley Zipf. Human Behavior and the Principle of Least Effort Google Books, 1949.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)) (Yss] All architectural and training details have been specified in the text. The code will be released with the camera-ready version.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
3. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
4. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "lVQ4ErKPTrR", "reviewer": "Reviewer_EeoY", "summary": "Recently, the emergence of in-context learning raises a question about the cause of this learning behavior. This paper considers in-context few-shot learning for image data. In the setting, this paper finds that in-context learning emerges when the training data is bursty. In addition, when the data follows a skewed Zipfian distribution, both in-context learning and in-weight learning can co-exist in a single model.\n\nUpdates after seeing other reviews: I under-estimated the core contributions of this paper, both in the novelty of the perspective and the setting of empirical evaluations. However, I still think this paper will benefit from more rigorous descriptions of some key terms. I increased the score to 5 (confidence rating 3).", "questions": "- It appears that burstiness is treated as a binary feature (defined in Figure 1 caption, lines 5-7). This is not a rigorous definition, and would definitely benefit from additional elaborations. \n- Similarly, \"in-weight learning\" and \"in-context learning\" are also defined in Figure 1 caption (lines 6-7), which are not quantitative definitions. The in-weight vs. in-context learning is an effect mentioned a lot in this paper. I think a less ambiguous definition than just \"learning labels across sequences\" or \"referring back to the context\" would be necessary.\n- \"Bursty sequences can be solved by in-weights learning or in-context learning\". So the non-bursty sequences can not be solved by either in-weights learning or in-context learning?\n- How does in-context learning work together with few-shot learning? Did you sample multiple demonstration sequences and run few-shot learning? Compared to few-shot learning (using just one image as input), this setting has a lot of computational overheads. Compared to in-context learning (without changing model parameters), this approach is limited to much smaller models. What are the benefits of in-context few-shot learning?\n- Sec 3.1: Doesn't bustiness overlap to some extent with \"a large number of rarely occurring classes\"? ", "limitations": "The limitations are adequately addressed.", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The study of interaction between in-context learning and in-weight learning is novel.\n- The inquiry into the reason that lead to in-context learning is important.\n- The experiments show some interesting findings.\n\n**Weaknesses**\n\n- Formatting: The page limit is 9 pages right? Lines 314-317 appear in page 10.\n- Three key concepts discussed in this paper, in-context learning, in-weight learning, and burstiness, lack quantified definitions. Please refer to my comments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kC_ZtRDUX2k", "reviewer": "Reviewer_D2K5", "summary": "The authors pointed out that the reason why in-context learning (few-shot learning without parameter updates) can be achieved with large left-to-right (causal) language models is due to the data distribution.\n\nCompared with in-weights learning (learning with parameter updates), the authors found that in-context learning is superior when the training data has the following characteristics: (1) burstiness (degree of unevenness as time series data), (2) large number of classes or heavy-tailed distribution (3) tokens have dynamic meaning. Also, they found that these characteristics are not valid in the recurrent architecture, but are more pronounced in the Transformer architecture.", "questions": "I assume \"burstiness\" refers to the so-called Taylor's law (bias as time series data), but can it be expressed in short training data of eight examples? Isn't it simply the uniformity of the frequency (cf. time series) that is changing...?", "limitations": "Limitations (future directions) are carefully described in Discussion section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nIt is fascinating to see research that provides an analysis from a new perspective on distributional properties of the training data to the important issue of realizing few-shot learning with causal language models.\n\n### Weaknesses\n\nThe experiments are not controlled.\n\nThe three dependent variables are different in in-context learning and in-weights learning: (1) binary classification <-> multi-class classification (2) no model update <-> model update (3) classification for holdout classes <-> classification for trained (or common) classes. Therefore, we do not know which explanatory variable (burstiness, heavy-tail, dynamic semantics) works for which dependent variable. It must be said that the scientific findings are unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KhcjQCOOSm-", "reviewer": "Reviewer_v8kR", "summary": "This paper asks a range of questions around what affects in-context vs in-weight learning in modern neural networks, focusing on Transformers, with RNN and LSTMs serving as comparisons. In totality, the experiments evaluate:\n\n1. In-context vs In-weight based on burstiness of the data.\n2. In-context vs In-weight based on number of training classes in the data.\n3. In-context learning based on the number of labels per class.\n4. In-context vs In-weight based on within-class variation.\n5. How a Zipfian distribution in the data affects In-context and In-weight learning.\n6. How In-context learning compares across Transformers, RNNs, and LSTMs.\n\nThe paper asks these questions in context of the recent strong LLM results. How can it be that these models seem to be exhibiting both in-context and in-weight learning? They then proceed to examine why this is and try to parse apart the causes by examining artificial data and comparing model classes.", "questions": "1. What is your actual definition of in-context vs in-weight learning?\n2. How would you construct a language dataset to test some of these hypotheses instead of a dummy omniglot one? That feels like a missing experiment here because you do lots of variations, but do not include anything that gets at the environment that the LLMs are operating in. Could you create a language dataset that wasn't bursty? In my mind, this would be testing what happens when you use a real-world dataset that followed the uniform IID concept.\n3. What would it take to do the same experiments in vision like you point to in the discussion?", "limitations": "As evidenced by the lengthy discussion, the authors understand the paper's limitations and what it points to. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThis paper is original wrt the questions it asks. The methods to answering them (omniglot, simple plots, etc) are not new, but they don't need to be. \n\nClarity:\n\nWhile the writing was good, the paper was surprisingly tougher to follow than I would have expected. I think it would get a huge improvement from a graphical interpretation of in-context vs in-weight learning. These are not terms I've seen before (and I've worked in FSL) and in-weight learning wasn't really ever properly defined in the paper. As far as I can tell, in-context learning was only roughly defined and that was on page 1 in relation to meta-learning. I had to piece together what I think is the definition from the rest of the work. I'm not sure if others you've shown this to have had the same problem, but just setting aside some space near the beginning to define what you mean by this would help a lot. Otherwise, I was left to wonder what actually fits into this view and what doesn't.\n\nQuality:\n\nThe quality is high. Once I understood what was going on with their interpretation of in-context vs in-weight learning, the results were pretty interesting. I liked the Zipfian distribution experiment a lot because it was one of the questions I would have had towards doing this on language. I am genuinely surprised that the LSTM and RNN models failed so terribly and am curious if the authors have a take on this.\n\nSignificance:\n\nThe paper's experiments are intriguing and point to some insights into why LLMs are working so well. I'm not intimately familiar with this part of the literature to know whether these align with other results, but to me this is new and good stuff asking worthwhile questions with thought-provoking results. This strikes me as significant for computer vision as well because of the questions about whether it's worthwhile to shift to different paradigms of data than the prevailing uniform IID approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2Zdk3a3EpS_", "reviewer": "Reviewer_jRKE", "summary": "This work performs a detailed investigation of the factors that promote different types of learning in transformers:  in-context vs. in-weights. The results delineate which aspects of the training distribution promote one form of learning or the other, or even both together. Further experiments find that recurrent models do not demonstrate the same learning abilities as the transformers studied here.", "questions": "How might the critical effects of the Zipfian exponent of the distributions be explained?\n\nWhat are the color codings of the curves in Fig 7 & Fig 8? Are they just plotting the individual runs for different seeds, or perhaps for different hyperparameter settings?\n", "limitations": "Line 232 says:  “We performed a comprehensive hyperparameter search for all models (see Appendix for details).” But the appendix mentions only “15 runs for each architecture” over three hyperparameters with 2 values each. This seems like a fairly shallow hyperparameter sweep. Unless perhaps each hyperparameter combination was counted as a separate architecture, making 8 * 15 = 120 runs for the transformer alone, etc. ", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an elegant and important paper. The investigations and their results are presented in a clear and compelling way, making for a real page-turner! \n\nThe work makes several contributions. The two types of learning studied here (in-context and in-weights) are neatly defined and explained. The experiments demonstrate how in-context learning is promoted by several specific properties of the training data:  burstiness, number of classes, and the Zipfian exponent of their marginal distributions. Especially exciting are the results indicating that a Zipfian marginal distribution with exponent of 1 over classes promotes a combination of both in-context and in-weights learning, which are otherwise at odds with each other in most experiments.\n\nThe comparisons between transformers and RNNs is of added benefit. The broader implications of these results, such as for application to RL, are especially exciting.\n\nOverall, this work begins to dispel the mysteries behind the excellent empirical performance of transformers, and should lead to many more enlightening studies.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept