An Explanation of In-context Learning as Implicit Bayesian Inference
Authors: Sang Michael Xie
Aditi Raghunathan
Percy Liang
Tengyu Ma
Venue: ICLR 2022 Poster (2022)
Keywords: in-context learning
language modeling
pre-training
GPT-3

Abstract: Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples. The LM learns from these examples without being explicitly pretrained to learn. Thus, it is unclear what enables in-context learning. In this paper, we study how in-context learning can emerge when pretraining documents have long-range coherence. Here, the LM must infer a latent document-level concept to generate coherent next tokens during pretraining. At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt. We prove when this occurs despite a distribution mismatch between prompts and pretraining data in a setting where the pretraining distribution is a mixture of HMMs. In contrast to messy large-scale datasets used to train LMs capable of in-context learning, we generate a small-scale synthetic dataset (GINC) where Transformers and LSTMs both exhibit in-context learning. Beyond the theory, experiments on GINC exhibit large-scale real-world phenomena including improved in-context performance with model scaling (despite the same pretraining loss), sensitivity to example order, and instances where zero-shot is better than few-shot in-context learning.

Published as a conference paper at ICLR 2022
AN EXPLANATION OF IN-CONTEXT LEARNING AS IMPLICIT BAYESIAN INFERENCE
Sang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu Ma Stanford University
(xie, aditir, p1 lang, tengyuma (atpppmamaacssse........e..seee dec.s.staaoord.ccm
ABSTRACT
Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning prompt consisting of input-output examples. The LM learns from these examples without being explicitly pretrained to learn. Thus, is unclear what enables in-context learning. In this paper, study how in-context learning can emerge when pretraining documents have long-range coherence. Here, the LM must infer latent document-level concept generate coherent next tokens during pretraining. At test time, in-context learning occurs when the LM also infers a shared latent concept between examples in a prompt. We prove when this occurs despite distribution mismatch between prompts and pretraining data in a setting where the pretraining distribution a mixture of HMMs. In contrast messy large-scale datasets used to train LMs capable of in-context learning, we generate small-scale synthetic dataset (GINC) where Transformers and LSTMs both exhibit in-context learnin Beyond the theory, experiments on GINC exhibit large-scale real-world phenomena including improved in-context performance with model scaling (despite the same pretraining loss), sensitivity to example order, and instances where zero-shot is better than few-shot in-context learning.
INTRODUCTION
Large language models (LMs) such as GPT-3 (Brown et al.) 2020. Lieber al.] 2021 Wang & Komatsuzaki 2021 Radford et al. (019) are pretrained on massive text corpora predict the next word given previous words. They demonstrate the surprising ability to do in in-context learning, where an LM "learns" to do task simply by conditioning
prompt containing input-output pairs, achieving SOTA results on LAMBADA (Paperno et al. 2016) and TriviaQA (Joshi et al., 2017) tasks (18% and 3% over previous SOTA (Brown 2020)). For example, consider the task of predicting nationalities from names. A prompt (Figure 1 is constructed by concatenating independent "training" examples (e.g., "Albert Einstein German") followed by "test example" ("Marie Curie was"). Conditioning on this prompt, GPT-3 places the largest probability on the correct output
p("Polish" "Albert Einstein was German | Mahatma Gandhi was Indian |n Marie Curie was")
by inferring the task from examples. Intruigingly, GPT-3 was not explicitly pretrained to learn from examples, and the distribution of prompts (which concatenate independent examples) is quite different from natural language. Our understanding of in-context learning is limited since (i) real pretraining data messy and (ii) in-context learning has so far required large-scale datasets and models.
In this paper, introduce simple pretraining distribution where in-context learning emerges. To generate a document, we first draw a latent concept 0, which parameterizes the transitions of Hidden Markov Model (HMM) (Baum & Petrie 1966), then sample sequence of tokens from the HMM (Figure 99. This latent variable structure is common in topic models such as LDA (Blei et al. 2003 Gruber al. 2007). During pretraining. the LM must infer the latent concept across multiple sentences to generate coherent continuations. When conditioning on prompt, in-context learning occurs when L.M also infers shared prompl concept across examples to make a prediction. We assume the LM fits the pretraining distribution exactly with enough data and expressivity, so that the question of in-context learning becomes characterizing the conditional distribution of completions
The code, data, and experiments are located GitHub and Codal.ah

---

Published as a conference paper at ICLR 2022
1. Pretralning decuments are conditioned on latent concapt (e4s, biggaphical
Abert Einpein was Germon thecretica plypiced, sabely ackrowdeddeed one of the greaned pryyaciats of inson for cleveloping the theory of eelidivity
Concept (eg. wi)
Datael byt
indepandent focus will bios nationaliites
Albert Eindein
German
Concapt B. bai
Mahatmo Ganchi was
Indian
.iril .Polish?
Marie Cur
Concatanate abe scamples into prompt and prediet neat wordft) model (LM] impiicilly desptie untatural conpderaaiio
Infers the shared
German Mahatmo GandN was Indian
was
Figure I: In-context learning can emerge from modeling long-range coherence in the pretraining data. During pretraining, the language model (LM) implicitly learns infer latent concept (e.g., wiki bios, which typically transition between name (Albert Einstein) - nationality (German) + occupation (physicist) - ...) shared across sentences in document. Although prompts that concatenate independent examples, in-context learning occurs the LM can still infer the shared concept across examples to do the task (name - nationality, which is part of wiki bios).
unnatural sequences
given prompts p(output|promp) under the pretraining distribution, where the prompt is generated from a different distribution Pprompe- This conditional distribution, which is the posterior predictive distribution, marginalizes out the latent concepts:
p(output|prompt)- = concept
(I)
If prompt)
prompt) concentrates on the prompt concept with more examples, then the LM learns via marginalization by "selecting" the prompt concept. Thus, in-context learning can be viewed as the LM implicitly performing Bayesian inference.
The main challenge is that prompts are sampled from a different distribution than the pretraining distribution. The canonical Bayesian asymptotic tool is the Bernstein- Mises theorem (van der Vaart 1998: Klejin& & van der Vaart 2012 Gunst & Shcherbakova) 2008), which asserts (under regularity conditions) that the posterior distribution of a latent variable concentrates on the maximum likelihood estimate. However, Bernstein- from the same distribution as the model, both of which are not satisfied. We prove that despite the distribution mismatch, the asymptotic prediction error of in-context learning is optimal when the signal about the latent concept in each prompt example is larger than the error due to the distribution mismatch. Additionally, we prove that the in-context learning error decreases with the length of each example- thus, information in the inputs, not just the input-output mapping, can be useful for in-context learning.
Mises typically assumes observations are independent and/or drawn
As companion to this theory, we created the Generative IN-Context learning dataset (GINC), which a small-scale synthetic dataset for studying in-context learning. We find that both Transform- ers (Vaswani al. 2017) and LSTMs (Hochreiter Schmidhuber 1997) trained on GINC exhibit in- context learning. We verify intuitions from the theory, showing that the accuracy of in-context learning improves with the number of examples and example length. Ablations of the GINC dataset show that the latent concept structure in the pretraining distribution is crucial to the emergence of in-context learning.
The experiments also bring up open questions which go beyond our theory, which only studies the pretraining distribution. We find that scaling up the number of model parameters steadily improves the in-context accuracy despite achieving the same pretraining loss, showing that larger models may improve in-context learning beyond increasing the capacity for memorizing the training data better. Previously observed in-context learning phenomena such as sensitivity to example ordering (Zhao et al. 2021) and the existence of settings where zero-shot is better than one/few learning (Brown et al. 2020 are also mirrored in GINC.
IN-CONTEXT LEARNING SETTING
Pretraining distribution. In our framework, a latent concept 0 from a family of concepts defines distribution over observed tokens from vocabulary O). To generate document, we first sample
2

---

Published as a conference paper at ICLR 2022
concept from prior p(f) and then sample the document given the concept. Each pretraining document is a length T sequence:
p(o = J p(00)-..,00(0)00))d))
Jece
We assume plo,.....0r ......0000))))) (0) is defined by a Hidden Markov Model (HMM). The concept determines the transition probability matrix of the HMM hidden states foT from hidden state set H.
Prompt distribution. The prompt distribution Pprompt generates prompts for in-context learning. prompt is a concatenation of n independent training examples and I test input Tistt which are all conditioned on shared prompt concept 0". The goal is to predict the test output Vles by predicting the next token conditioned on the prompt.
A prompt example is composed of an input token sequence 2 (e.g., Albert Einstein was) followed by an output token y (e.g., German). In particular, the i-th training example O consists of an input r; = O1[1: k 1] (the first - I tokens) followed by output token y; = [k] at the end 2 The i-th training example is independently generated as follows:
1. Generate start hidden state h: from a prompt start distribution Perompt-
2. Given hittn,, generate the example sequence O = Mi] from distribution conditioned prompt concept o".
,0)), the pretraining
The test input Tiest In++1 is sampled similarly. Between each example, there is a special delimiter token delim, The prompt consists of a sequence of training examples (Sn) followed by the test example Tiest
[ual = [x1.8/1 1/.0 delim .I223/2.
delim
In...In Un o delim Fresl ~ Pprompt-
Mismatch between prompt and pretraining distributions. Since transitions between independent examples can be unnatural, the prompts are low probability sequences under the pretraining distribution. We provide a simple illustration using the names to nationalities example. Suppose that wiki bio documents in the pretraining data typically transition between name nationality occupation ->... In the prompt, the examples transition between name - nationality - name + nationality - which contains low-probability transitions such "German" - "Mahatma Gandhi"' The prompt formatting (e.g., choice of delimiter) can also be a source of mismatch. We aim to show that despite this mismatch, large LMs can infer the prompt concept from examples.
In-context predictor and task. For in-context learning, the output target y for each example is sampled according to Pprompt (v) (v)
Yress ~ Perompi (ess)) E, par Poproag (Aela (maa)) [P(y] A _ )]
(4)
where hegant denotes the hidden state corresponding to the first token of Trest. We analyze the in-context predictor fr (Frest) = argmax,, p,y() S,,,a)) which outputs the most likely prediction over the pre- training distribution conditioned on the prompt from the prompt distributior
We study the in-context
predictor and its expected (0- error with n examples (Jn) = E Mee PPyong (Zies)) } Ytea])
2.1 ASSUMPTIONS
We detail the assumptions in our framework, including the structure of delimiters and regularity assumptions. We first assume that there exists subset of delimiter hidden states D which generates the special delimiter token odelim deterministically.
Assumption 1 (Delimiter hidden states). Let the delimiter hidden states 'D be a subset H. For any f,delim E D and 0 e O, plodel d () = and for any h& D, peldei h =0.
Thus, observing the delimiter delim reveals that the corresponding hidden state is in D, but does not reveal which element of D it is. The delimiter is usually token that can appear in a broad range of contexts (c.g., newline). The delimiter ideally does not distract from the examples- - for example, an ad advessarial delimiter could look like part of the input æ. To mitigate these scenarios, we assume that no delimiter (e.g., newline) is significantly more likely under one concept rather than another.
The example length A is fixed for simplicity - we leave extending our analysis to variable i as future work. In practice, greedy decoding or nucleus sampling (Holtzman al. 2020) are used for likely completions.
3

---

Published as a conference paper at ICLR 2022
Assumption 2 (Bound on delimiter transitions). For any delimiter state hdelim D and any hidden state E H. the probability of transitioning to a delimiter hidden state under 0 is upper bounded pl p( hdellon h, C2 for any O Oe { and is lower bounded p p(hdelim (h,0*) >c > 0 for 8" Addition- ally, the start hidden state distribution for delimiter hidden states is bounded as p(haelim (0) E [ca,ca].
The prompt start distribution is source of distribution shift that is separate from the shift from concatenating independent examples. We make an assumption that limits how much distribution shift introduced by the prompt start distribution.
Assumption 3 (Distribution shift from prompt start distribution). We assume that the prompt start distribution Pprompr is close in TV distance to all hidden transition distributions (under 0*) starting from delimiter hidden state: max eD TV (Ppromp (Ppromp(h)]lpp (h)]lp(h ,0") < A/4. Here,
A = Pprompr Ymax (we)) - ma Pprompr (U [ress) is the margin between the most likely label Mmax argmax, Da pPprompr (y) (iess) and the second most likely label.
Even if the maximum TV distance is O, there is still distribution shift from concatenating independent examples. We also assume the prompt concept 6* is in the family θ a broad set of concepts.
Assumption 4 Well-specifiiation)) The prompt concept 0 is in θ.
Even though the pretraining distribution is broad, the prompt is still low probability under the pretraining distribution since it concatenates independent examples. Finally, if the prompt has zero probability under the prompt concept 8", then Bayesian inference will not be able to infer the prompt concept as in Section 3.1 The following are regularity assumptions which mainly ensure that the prompt not zero probability under 0*.
Assumption 5 (Regularity). The pretraining distribution p satisfies: 1) Lower bound on transition
probability for the prompt concept 6" for any pair of hidden states hih E H. p(h]h > cs > 2) Start hidden state is lower bounded: for any h h H. p(h(0") ca > 0. 3) All tokens can be emitted: for every symbol o, there is some hidden state h such that p(o) has support over the entire concept family O and is bounded above everywhere.
> 0, 4) The prior p(0)
3 THEORETICAL ANALYSIS
We prove that in the limit of infinite examples, the error of the in-context predictor is optimal if a distinguishaabiltyy condition holds - the prompeconcepl I is distinet enough from the other concepts in e (c.g.. when is a discrete set). When distinguishability does not hold (e.g. θ is continuous valuedd,, we show that the expected error still decreases with the length of each example, showing that information in both the inputs and the input-output mapping contribute to in-context learning.
3.1 HIGH-LEVEL APPROACH
Our goal is show that argmax, (Py)S,, (iss)))s) + argmax................. PpromppppPPpppp//////// as the number of examples n grows. In the following, assume that the prompt has non-zero probability under the pretraining distribution p given 8*, meaning that .S > O. We expand p(y) (iess) to analyze its limit:
p(y] (q Frea)) = plyl Trest . ()(( (ressd)
1
plyl S Freat 0 pp Sn wes 2w)p
(Bayes' rule, drop the constant
pl (rest)
/
(reat, p(Sni (0)
Σ
ply Mlyl A.stun 0 ()p(()) start
Freal (0)) 10)pp10))))))00 p(0)c0
(5)
lige EH
(Law of total prob, Markov property, divide by Su, meeal"))) (a constant))
Σ Aom EH
p(y (reea Moia ()pp())
start| Sn expppexxpppxrr))mme (x)))) (0))p(0))
(6)
where (x) 1 log P(S...zex.0 10) In Theorem we prove that under distinguishability condition.
12 200
exp(n-r..()) - for all concepts except the prompt concept 0*, where exp(n-r,, (0=))= 1. The
only nonzero term in the integral is when 0 = =0", and thus the prompt concept is "selected" as consequence of Bayesian inference") Lemma shows that the argmax after restricting to 8 is the
WWe can exchange limits and integrals since the probabilities are bounded (dominated convergence).
4

---

Published as a conference paper at ICLR 2022
same as the most likely label under Pprcemp (res) (using Assumption] 3). Putting these together with Equation 6, the in-context predictor infers the prompt concept 6*:
argmax p(u) (teat + argmax Pprompt (3/ (rest)
y
Thus, the in-context predictor is optimal as the number of in-context examples increases.
3.2 HRURISTIC DERIVATION
Recall from Section 3.1 that if exp(a- (()) -0)))000 - for all 0+00, then Bayesian inference "selects" the prompt concept through marginalization. To do this, we focus showing that r. (0), the average log-likelihood ratio between 8 and 0", converges to negative constant, and thus nrn goes to -00.
The main technical challenge is to handle the requence-of-exanee makes all the examples dependent with respect to the pretraining distribution. Our approach uses properties of delimiter tokens to approximately factorize the examples, with constant error per example. We let O | Oi] be the i-th input-output pair and the previous delimiter together for >>>>>>>> and define O = O1. Expanding the likelihood term inside r,, (0), our goal is to show
structure of the prompt, which
P(Snn - 1 ~ II [0(1)((0,10)
(8)
To show this, we expand p(S,, with the chain rule, and with Assumption 5 (to bound p( S) by ((1) can be shown that
We then marginalize 10 over the hidden state h corresponding to the delimiter in
0 = po Or}:
I
Σ IE-TED
p(O,)1 O) O - ((1)p(O,10))
==1
(10)
While summing over above would be trivial equality, we can replace H. with the set of delimiter
hidden states D since p(h) 1, for non non-delimiter hidden states h & 'D (Assumption 1.. We
used in the first equality that O -i--1 h.lelmm - O forms Markov chain and p(ogeim | I.dmm) (Assumption]] T to change O to O, Finally, we can show using properties delimiter hidden states
(Assumption 2) that p(h 10% 0) = (1) and Σ EmaRnED ED (O, .) (11)p(O,))))))))) ()))))))))))))))))))))))))) in the
second step. Therefore, we upper bound (f)
log p(0,0)) p(O,10*)
-0 Log P(O10) p(O))))
(0) O(n)+ Σ
(11)
The expectation term can be written K L (O)lp((00)) stant using Assumption 5 intuitively for one example, Porompt and p(-10*) are close. We break the second term into a sum of negative KL divergences over & tokens. There are O(k) KL terms and only (1)) other error terms, which come from the distribution mismatch between the prompt and pretraining distributions. If the KL terms are larger than the error terms, then r, (0) has negative limit. this holds for all 0 o 0", then we have exp(n-ra (0) ++0 for all 0 00 enabling in-context learning.
the difference of K LPproor (L][prooo(OOIlp(O)))))) (O]lp(()0)) We bound the first KL term by a con-
KI. divergences,
3.3 FORMAL RESULTS
3.3.1 IN-CONTEXT LEARNING UNDER DISTINGUISHIABILITY
We define distinguishability condition which formalizes when in-context learning occurs. Letting (o) = p(O[) = o/O[1 1) be the output distribution of the j-th token given the previous tokens and Pprompp (0) Ppremmp (Oli 0 [1: -1]) be the analogous distribution under the prompt

---

Published as a conference paper at ICLR 2022
000 low transtions between examples
M
Fintain
Gardhi
Curie
UU UUV
In-distriattinn transkions reveal coealiiformation.o about 9*
Figure 2: When the signal about the prompt concept within each example (green) is greater than the error from low-probability transitions between examples, in-context learning succeeds in our latent concept setting (Theorem I). Increasing the example length & increases the signal. The signal for in-context learning comes from tokens in both the inputs and the input-output mapping.
distribution, the distinguishability condition depends on the KL divergence between Pirompe (which represents 0*) and p well error terms & and delim coming from the distribution mismatch between the prompt and pretraining distributions at the start and delimiter token for each example:
L (0 (10) = Eo!i: ;- 1-1] K [P [P)]
(12)
2 = 20 -log (og((o)()))))) +log(c)))))))))) -og((c)))oo(cc)))) -lgg(//())
(13)
Condition (Distinguishabiiitt).. We define 8 be distinguishable if all 0
Σ KL, (K Enar e Edelii- 1=2
(14)
When the signal from KL divergence (LHS) is larger than the error terms, Equation 14 is satisfied (Figure[)). For larger example lengths the LHS increases, improving distinguishability Intuitively, larger example lengths increase the proportion of the prompt sampled from the pretraining distribution by providing more evidence for Bayesian inference. Under Condition T asymptotically achieves the optimal expected error.
the in-context predictor
Theorem 1. Assume the assumptions in Section 2. hold. If Condition Tholds, then as n - oo the prediction according to the pretraining distribution is
argmax ply)Sn - argmax Pprampe (y) (rese).
(15)
9
Thus, the in-context predictor fn achieves the optimal 0-1 risk: lim (an (J)
3.3.2 NON-DISTINGUISHABLI CASE
The distinguishability condition (Condition D fails when there is some 0 * 0* for which the KL divergence between 0 and is less than the error terms. However, this also means that the output distributions of 9 and 6* are close in KL. We leverage this to prove that the expected 0-1 error decreases with the example length under two different settings where distinguishability does not hold.
Continuity. Our first result relies continuity assumption between the concept parameter and its corresponding output distribution. Our assumption is based on prior works (Kleijn van der Vaart 2012), where the KL divergence is assumed to have a 2nd-order Taylor expansion.
Theorem 2. Let the set of which does not satisfy Equation 74 Condition be B. Assume that KI. diveeregss have 2nd-order Taylor expansion around 0":
Vj>1, K (L, 10-0000010000000000----00- (0000000000000000000000000000000
(16)
where 15.9- is the Fisher information matrix of the j-th token distribution with respect to 0*, Let
max, ), where AmaryAmin return the largest and smallest eigenvalues. Then for k 1 and
Ti- minj hanal as n. 0o, the 0-1 risk of the in-context learning predictor fn is bounded as
(0
"00-SU
( delim))
lim La n-200
(17)
k-1
(1+))) is a fellibrationfuncttioon Steinwart Pires & Szepesvári 2016) for the multiclass logistic loss for ó (0.1).
where g(8) = 3 (11
2007
Avila

---

Published as a conference paper at ICLR 2022
90
100-
80
k=5
70
k=
60
60
50
40
40 Num examples
20
0
20
40
60
Nur examples
Figure 3: In-context accuracy (95% intervals) of Transformers (left) and LSTMs (right) the GINC dataset. Accuracy increases with number of examples n and length of each example k.
13
111
12
10
20 hum cazmpics
40
Rum caumao
Num cuumplos
Figure 4: Ablation studies for layer Transformers on the GINC dataset with vocab size 50. (Lefft) When pretrained with only one concept, in-context learning fails. (Middle) When the pretraining data has random transitions, the model sees all token transitions but in-context learning fails. (Right) When prompts from random unseen concepts, in-context learning fails to extrapolate.
Since the inverse calibration function -ー is roughly linear in for e S 0.7, the excess risk roughly decreases as O(1 O(1/k). When the "worst-case condition number" "yp- of the Fisher information matrices is smaller (well-conditioned) the eπrоr decreases. Intuitively, this means that there is direction to vary 8* in which the output distribution will sharply change. As consequence, the concepts 0 that not distinguishable from the prompt concept l" parameterize distributions that produce similar outputs to the prompt concept and thus achieve a small error.
Varying-length test examples. In the setting where the length Vies random (uniformly from 2 to k), we can give similar error guarantec without continuity.
Theorem 3. Let the set of which does not satisfy Equation [4 Condition 7 to be B. Let the length of the test example Tus be uniformly distributed between 2 and k. for 2> 2. Then for k 2 and as n - 0c, the 0-1 risk of the in-context learning predictor fn is bounded as
(SupeeB( ( t delim))
lim La (o)( Linn
(18)
k-
Instead of measuring only the error at the k-th token, average the prediction error on the 2nd to k-th tokens. However, we leave bridging the mismatch between training examples, which are consistently length k. and test examples, which have random length, to future work.
SIMULATIONS
We generate the GINC dataset and show that Transformers (Vaswani al. 2017) and L.STMs (Hochre iter & Schmidhuber 1997) trained on GINC exhibit in-context learning. In the theory, we assumed that the pretrained LM fits the pretraining distribution exactly. Here, we pretrain LMs approximate the pretraining distribution and find that the in-context learning properties transfer to the LM.
GINC dataset. We construct the GINC dataset according to our theory (see Appendix F.J. For family θ of 5 concepts to generate 1000 pretraining documents with 10 million tokens total. For prompting, we generate prompts with 64 training examples and example lengths E (3,5,8,10) (2500 prompts for each setting). The target token Yeet is taken to be the most likely output instead of sampling so that the intrinsic error is O.
pretraining, define uniform mixture of HMMSs
Main result. We train GPT-2- based Transformers (Radford et al. 2019) and LSTMs on three versions of the GINC dataset with vocabulary sizes 50, 100, and 150, then evaluate the in-context
7

---

Published as a conference paper at ICLR 2022
100
Madel
Paaro
Ican qestririip
Valkos (peetiiing)
Invortion Ace
90
201
1.80
1.30
602
Iraetlomrr12 byerr
X034
1.31
1.35
81.2 2.34
Traedtmmrtt 16 byyr)
113
1.31
Vocabatt 108, 10 d4
Imaetoomer aoerr
2004
1.8
120
-50
Traastomen 112 byyen)
R0M
1.41
Vocab size
L.TMM
20M
1.43
1.44
$5.8 1.54
Vocab size #10
Vocalsito 150, 10
0.4
Num Par Parameeters
0.6
0.8
1.0
108
Traefommr lieer) 1.5TM
Tranffimrr (12 liyer)
115M
R5M
70M
1.20
1.27
1.78
1.31
59.7
984-04
Figure 5: In-context accuracy (95% intervals) of Transformers improves as model size increases on the GINC dataset for vocabulary sizes 50, 100, and 150.
Figure 6: In-context accuracies (95% intervals) on GINC with vocab sizes (50, 100, 150) for Transformers and LSTMs. Accuracy improves with scale even though the pretraining loss may be the same.
accuracy (see Appendix F.2 F.3 We average all results over 5 pretraining runs. Figure 3 shows that for both Transformer and Ind..TMM. in-context accuracy improves as the number of prompt examples and the example length l increase, verifying our theory.
Ablations on the latent concept structure. We ablate the role of the mixture-of-coneepe structure in GINC. In Figure (left), we pretrain 4 layer Transformer on data with only one concept (removing the prior) from θ, resulting in flat in-context learning curves. Figure (middle) (midde(( shows that pretraining on random pretraining data, which contains all possible token transitions, in-context learning also fails. Therefore, the nif-ure-of-concepts structure is important and simply seeing diverse token transitions does not enable in-context learning.
Extrapolation to unseen concepts. Full generative control of GINC allows for experimentation with latent variables in the pretraining distribution. For example, in large-scale datasets, it is difficult to test whether a concept or task is in the pretraining data. We test this in GINC by testing the in-context accu- racy of a 4 layer Transformer on prompts generated from random concepts that are not in the pretrain- ing family of concepts. Figure (right) shows that in-context learning also fails for these novel concepts.
Effeet of model size and architecture. FigureBjshoss that increasing the size of the Transformer (4, 12, 16 layers) steadily increases the in-context accuracy, corroborating the results of Brown et al. (2020). Table shows that even though larger Transformers may have the same pretraining loss (e.g., 12 and 16 layer Transformers both get 1.33 validation loss for vocab size 50), the in-context accuracy still improves (81% to 85% from 12 to 16 layers). suggesting that larger models can improve in-context learning beyond improving pretraining perplexity. This may be related to phenomena from overparameterizattio and overtraining (Zhang et al. 2017] |Pwerr et al.) (021). Finally, the model architecture also plays a role- Sidde-LSt LSTMs consistently outperform Transformers on GINC despite having fewer parameters, perhaps due to the similarity between HMMs and LSTMs. We leave analysis of the effect of model scaling and model architecture as open questions.
Sensitivity to example ordering. In Figure 7 (left), we test the sensitivity of in-context accuracy on GINC to the ordering of the prompt examples, followingZZhae et al. (2021). For this experiment, we consider prompts generated from a single concept and prompt start distribution. We sample 10 different sets (leading to 10 training set IDs) of 4 examples and generate all 24 possible permutations for each example set. We consider the in-context accuracy of the layer Transformer trained on GINC with vocabulary size 50. Similarly the behavior of GPT-3 (Zhao al. 2021), there is significant variation (10-0% difference) between permutations of the same set of examples.
Zero-shot is sometimes better than few-shot. In some settings in GINC, we find that zero-shot performance can be better than few-shot performance. This mirrors GPT-3 on some datasets g.. LAM- BADA, HellaSwag. PhysicalQA, RACE-m, CoQA/SAT analogies for smaller models (Brown et al. 2020).. This occurs especially when the transition probabilities GINC are lower entropy (controlled temperature parameter). For this experiment, ature parameter 0.01 (instcad of 0. I). 12 concepts, and vocabulary size 100. Figure 7 (right) shows that here, few-shot accuracy is initially worse than zero-shot accuracy, but can recover with more examples. We hypothesize that the distracting prompt structure initially decreases the accuracy in this setting.
consider GINC with transition matrix temper-
8

---

Published as a conference paper at ICLR 2022
70
Accuracy
60-
50-
40-
20
30
20 40 Num examples
60
Traiiing set
Figure 7: (Left) In-context accuracy varies widely with example ordering. Each training IDD refers to a set of training examples. Each dot refers to the in-context learning accuracy of one permutation of the training examples for that particular training ID. (Right) Zero-shot performance can be higher than one/few-shot performance in some settings in GINC, mirroring the behavior of GPT-3 on some datasets such as LAMBADA (Brown et al. 2020). The few-shot setting introduces the distracting prompt structure, which can initially lower accuracy.
DISCUSSION AND RELATED WORK
Learning via Bayesian inference and extrapolation. The canonical Bernstein-von Mises theo- rem (van der Vaart 1998) does not apply for in-context learning since the prompt examples are not independent under the pretraining distribution. Gunst& Shcherbakova (2008) show a Bernstein Mises-type result for observations from an HMM, but do not handle observations from a different dis- tribution. Future directions include more precise asymptotic results about the posterior distribution and results under cesspeccficatioonextaallleee
(Kleijn & van der Vaart 2012). A possible avenue for ex- trapolation to some types of unseen concepts to factorize the latent concept into semantics and syntax. While the pretraining data may contain only some semantics-syntax pairs, the language model could generalize to unseen pairs if it learns generalizable syntactical operations such as copying or reordering.
Topic models and HMMs. Topic models such as LDA (Blei et al. 2003) also have document-leve latent variables, but learning is typically relies on algorithms such as EM (Dempster et al.) 1977).
variational inference (Jordan et al.) 1999). ). or MCMC (Metropolis et al. 1 Hastings] 1970). We natural result of Bayesian inference without explicit inference algorithm.
focus on learning
Wei et al. (2021a) la also use an HMM model in their pretraining analysis. However, they analyze how
pre-trained representations learned with masked LMs (Devlin al. 2019 Liu et al.
2019 Lewis
ct al.
2020 Clark et al.] 2020) can improve optimization based downstream learning (Li & Liang
202
Lester al.
2021 rather than in-context learning.
Bridging the mismatch between pretraining and prompting. Prior works support our theoretical
intuitions that reducing the prompt distribution mismatch would improve in-context learning. Finetun-
ing LMs text with a prompting format improves its zero-shot performance (Wei et al.
[022 Sanh
cct al. 2021) and optimizing prompt templates improves few-shot finetuning (Jiang et al. 2020 Schick
& Schütze 2021 (Shin et al. 2020 Gao et al. 2021). Zhao al. (2021): Holtzman et al. (2021) improve
in-context accuracy via calibration or renormalization, form of adaptation to the prompt distribution.
Meta-learning, Meta-lcarning methods can also train a sequence model to learn from examples (Ravi) & Larochelle| 2017). However, meta-learning models are trained to learn, while in-context learning emerges from LM pretraining.
Studying large-scale phenomena at small scale. We study in-context learning, large scale phenomenon. at small scale in GINC because the complexity of the pretraining distribution (HMM hidden state size, number of latent latent concepts) is small, such that the data and models are relatively larger. Since GINC is synthetic, we can also control the latent data properties (e.g., unseen concepts) to make predictions about large LMs while working at a small scale.
CONCLUSION
We cast in-context learning as implicit Bayesian inference, where the pretrained LM implicitly infers concept when making prediction. We show that in-context learning occurs when the pre-training distribution is a mixture of HMMs. Our work provides a first step towards understanding in-context learning, which we hope will provide insight for improving pretraining and prompting.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDG
We thank Tianyi Zhang. Frieda Rong, Lisa Li, Colin Wei, Shibani Santurkar, Tri Dao, Ananya Kumar, and Shivam Garg for helpful discussions and feedback. SMX is supported by an NDSEG Fellowship. The work is partially supported by Open Philanthropy Project Award, SDSI, and SAIL at Stanford University. TM acknowledges support of Google Faculty Award, NSF IIS 2045685, the Sloan Fellowship, and JD.com. Toyota Research Institute provided funds to support this work.
REFERENCES
Leonard E Baum and Ted Petrie. Statistical inference for probabilistic functions of finite state markov chains. The annals of mathematical statistics, 3766):155-11563, 1966.
D. Blei, Andrew Ng, and M. I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research (JMLR), 3:923-1022, 2003.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Keererrt-Vsss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 240:2000..1165,, 2020.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations (ICLR), 2020.
A. Dempster, Laird N. M., and Rubin D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, (1): 1-33, 1977.
Jacob Devlin, Ming- Wiig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding, Association Computational Lingussiies (ACL), pp- 4171-4186. 2019.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv,, 2021.
Zoubin Ghahramani and Michael Jordan. Factorial hidden Markov models. Machine Learning, 29: 245-273, 1997.
Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. Hidden topic Markov models. In Artificial Intelligence and Statistics (AISTATS), 2007.
M. Gunst and O. Shcherbakova. Asymptotic behavior Bayes estimators for hidden Markov models with application to ion channels. Mathematical Methods of Statistics, 17, 2008
Keith W. Hastings. Monte Carlo sampling methods using Markov chains and their applications.
Biometrika, 57(1)977-109 1970.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't always right, 2021.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Association for Computational Linguistics (ACL), 2020.
Michael Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37:183-233, 1999.
10

---

Published as a conference paper at ICLR 2022
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics 2017) 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
B.J.K. Kleijn and A.W. van der Vaart. The Bernstein-von mises theorem under misspecification. Electronic Journal of Statistics, 6, 2012.
Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-efffcient prompt 2021.
tuning, arXiv preprint
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeecee pre- pre-training for natural language generation, translation, and comprehension. In Association for Computational Linguistics (ACI.), 2020.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Association for Computationall Linguistics Linguiisss (ACL), 2021.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation.
Technical report, A121 Labs, August 2021.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint /907. 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Eguation of state calculations by fast computing machines. The journal of chemical physics, 2166):0087-1092, 1953.
Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. Association for Computational Linguisties (ACL), 2016.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. In ICLR MATH A/ Workshop, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl Blog, 188), 2019.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Unmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak. Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2021.
Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. In European Association for Computational Linguistics 2021,
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Eliciting knowledge from language models using automatically generated prompts. In Empirical Methods in Natural I Language Processing (FMNI.P) 2020.

---

Published as a conference paper at ICLR 2022
Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation, 26, 2007.
A. W. van der Vaart. Asymptotic statistics. Cambridge University Press, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 7w0. (70.006276. 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https: /ttp::////ithunn www.oiigoooioiocc.eeooooooo.o..e www.ooo///meeetateeeeee wwwh-eeaassssroeeeeeeeeae
jax, May 2021.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. arXiv, 202
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. arXiv, 2021b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' emi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace's s transformers: State-of-the-art natural language processing. arXiv preprint 1rXiv:1910.0377,, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference Learming Representations (ICLR), 2017.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. International Conference on Machine Learning (ICML), 2021.
Bernardo Ávila Pires and Csaba Szepesvári. Multiclass classificaion calibration functions. arXiv. 2016.
12

---

Published as a conference paper at ICLR 2022
A FRAMEWORK DETAILS
Prompt distribution details. For in-context learning, we sample a prompt from a new distribution Pprompe. which consists of independent training examples and 1 test example. We first sample n hidden segments H of length k by sampling the first element haian [[ from a prompt start distribution Prooupt Then, sample the rest of the segment H = H [2: from the hidden transition distribution of the pretraining distribution corresponding to a particular concept
H1....Haa Has
H; = [Ms,1, (.s,,....,
(19)
1 start H1 [a[11~PPommpp
H H, (2: ~pp H Ih start ,8))
(20)
To end each example (except the test example), sample delimiters h dellm D from pariip hi Idelim
1/ delim ....,h? ....,i
~ PPproppp-
(21)
Conditioned on hidden variables H, and Aeelim, we sample the observed tokens O; = -00= (0.,1 and delim respectively from the pre-training distribution:
O O On, Op~p(O:(HH))
(22)
0)
delim
delim On
h; 0 ~ p(o, oelim Idelim (r)
(23)
The "input" for each example is I; = O, : k - 1] and the "output" is yı :==0, Ok[k) Taking S to be the sequence of training examples (without the test example), the resulting prompt sequence is
[San
=01 [0 ..... On.00 delim
delim
delim Freaa] (/1 delim ,22.22202 D.
delim [Riesl Pprompt
(24)
where Thea 54n+1 = On+1 [1 [1:k- k= 1] sampled via the same process but with k elements.
B PROPOSITIONS FOR THEOREM
The following propositions, which lower bound the probability of a delimiter token and probability of an example under 8*, are direct corollaries of the assumptions.
Proposition 1. For all i, have p(hçriim of ...,,,,8*)) eı and p phheelim Of ..... ,,.,) C2.
Proof. By Assumption
2
p(hdelim 01 01,0felim delim ,..,, 0,,0= -000=-pppe Ep (he.mmm)))))hll.l.00m
delim ,..,O
(25)
Cap(hi, 1 delogli
delim ,..,, =22
(26)
he.e
Similarly,
(h
Og.ofelim
dellim
....,,, Ep(heelian p (h ,.,00)
(27)
h
delim ...,O
(28)
hi.k
Proposition 2. The probability of an example is lower bounded for 0* there is some e7 > such that phO,l h,,,0') c7 for all and future hidden states h. any I and i.
Proof. By Assumption we have
(h start
1)) = Σ (HHH))H)))(HH111 11,100)) (co)*
(29)
H.
for some H.. We have
p (H (igtt ,,,,,0") p( H H, ,0 80)'p'((HH))))))) haaan A
(30)
p(hg,1 I Ih saart 0+)
which lower bounds the terms in the numerator by cs (marginalizing over previous hidden states), and upper bounding the denominator by 1. Setting C7 (ca)* of finishes the proof.
13

---

Published as a conference paper at ICLR 2022
C CONVERGENCE OF THE IN-CONTEXT PREDICTOR
Under Assumption we show that the in-context predictor fn (Tis)) = argmax, y ply Blu Ties) converges when abstracting away the Bayesian inference component (the selection of l from θ) of the in-context predictor. We will complete the argument for the convergence of the in-context predictor in the proof of Theorem
Suppose the prompt Sn and the test input Ies are given. Under Assumption 3 we show that the argmax of the averaged predictive distribution conditioned on O* and a prompt S, the same as the argmax of the prompl predictive distribution:
Lemma 1.
argmax
Σ
p(y) Mestt 0 }pp(b mor Sn "enn =argaaagapppaan (3) Frest).
(31)
Fi E.
Proof. First, we note by definition that
Pprompe (y) (Fest))
Σ
Fresttest haaer 0 Ppeompe (h (fust (rrest)
(32)
Expanding the last term, we have
Pprompe (hrest Frest)
X Trest Mest 0 ( Pprompa (Fiees) )-
(33)
which is proportional a constant in Trest-
On the other hand, analyzing one term inside the LHS of the lemma statement, we have
p( h Mart Sis 00) Xx p( Niest ,0" p p(ttssst (,,,8")
start
(34)
which is proportional a constant in Frest and Sn. The quantities differ in the last term, which we expand below and put in matrix form. Let T E [ be the matrix that represents the transition probabilities starting from a delimiter state: p( (h (n (ndelim) ) for hiuar E H. and dedlim E ED. As a result,
p( start (n,0") )²=)===> = Σ
p(hesn
start nna delim -0" (p))p(a (S,
(35)
Tu
(36)
where dellim is the delimiter hidden state before Aster.
Let W [ be the matrix that represents the probabilities p(yl N ( )p( Tress /hstrt (0*) for all the possible y E y and hatart H. Overall, we can write
Dees Nigant 0 (")p(hees Sn Frest 0*) WT
(37)
FAR EH
Preampe
(- (rest) = W w.
(38)
where E [K] is the vector of probabilities that corresponds the prompt start distribution Pprompt-
Bounding the difference between the two predictive distributions,
WT uu -0 < |TT. Wullz
(39)
|
W. (Tv-u)l
(40)
(Ty(To-u)j
(41)
P [2
WTl(Tu-u))
(Wi 20)
(42)
R
(o))(To-u)]
(43)
(44)
14

---

Published as a conference paper at ICLR 2022
Using Assumption 3 we can further bound this by A/2:
[Tu-uli u|| = 27V (Pprompl
cap( delim =i,
(45)
2 TV (P (ppoompe (1pp(11 pedpim -i, (convexity of TV distance)
(46)
i=1
2 max E
TV (Pprompt ) 1Ip(p11 delim 0 <A/2.
(47)
Since the probability of any output does not change by more than A and the margin between the most
likely label and the second most likely label A, the argmax's 's are the same, showing the result. f
D PROOF OF THEOREM
Proof. We analyze the most likely prediction over the pretraining distribution conditioned on the prompt argmax, (wa))
ply 0))
(48)
p(u (0 ntees (0))do
(49)
p(y) Sna p(S Freal (0)
x
p(S, 18-) p(0)do
(50)
start Sp Frest pl Thest (0)
Σ
ply) Wiest Mose O)p(( (1
pl Freel (00) 100 pl (00)d0
(51)
Mise EH
Defining the following quantity,
n' (0) log ( (0)
(52)
n reen (-))
we will show that under distinguishability for all # 0" In (0) converges to negative constant such that
pl piSnnitress (
p(Sa Tress (0r) exp(n-r, (0)) +
(53)
for 0 whereas this ratio is always for - This will then "select" the desired prompt concept through marginalization.
Supposing that Equation
53 holds, we show that the theorem statement holds. Let
A max TV (Pprompe (- (²)) <A/2,
(54)
and let < (A/2- A ('))(0'). Then for large enough (due to Equation 53
/ 9
p(S -Teat (8)
Σ ply h .0 (hapaa Drsat
pls Frest (-) p p(0)d0
(55)
Σ plyl Frrest Nytat 0 )p( (h Frest (p(f))+ I jopp- coly)p(O)d8
(56)
hean EH
1 p(0') Jega-
f co())p(0)d8
ply Frest, H
start _ )p(hee saar (Sn 0"
(57)
EH
where (o()) 5/2 for all y
By Lemmal [] the argmax of the first term of Equation 57 is the same as argmax, "ygmxx,pPpomppp (y) Rres), where the margin between the most likely label and the second most likely is at least A - A'. Since
1
/
ca())p(0)
2p(0~) <(A/2-
(58)
p(0') 000
15

---

Published as a conference paper at ICLR 2022
for all y y the argmax of Equation577 57 is also the same argmaxppompl (res))
= Now remains to show that r, (0) converges negative constant for 8 f0" Let = i ,0,] be the i-th observation segment and the previous delimiter together for <>>> and define 0 O₁. Expanding the numerator of the ratio in r' (0), we have
Pten (0) p( Fresst Sn, Sn, (S)(
(59)
Σ (rest I1 jest 0
0)p((ppp)))))) (eatt (
1:n ==1
0)
II p(O 10%
p
(60)
D (fres Mest p)p saat 0)
(61)
plonclim hggeiin (p(k (hgg
delim
1 Tmn 0)
II Σ
delimm delim
1.0-
Adiim DD
{= 1IIA-TED
(62)
prrss sear
O)p(( (aatt
S)
(63)
p(ha
delim
O 0)
II Σ
p(Os)
(64)
lan
Aellin
ED
pl Trest Mest 0) ()p(1
start
0n
II Σ
p(O:) EI
delim 10pll-0r00ppppp00 0p(( 1.
(65)
Note that in the last line, the inner sum is over the set of delimiter states D by using the assumption that observing a delimiter o delim implies that the corresponding hidden state h delim must be in D. We also see that Eneup p(h ( delimm O -1
We restrict our attention to 0 where a(Sa 10) > 0, since otherwise does not affect the prediction. Expanding (0), we have the following upper bound:
n (1og log ((
rr r0)
(66)
- (
Σ how pl Fress I heaart (0)(pp(hh)) hattrr S)
Σ Juann D (O, p)ppppp)ppppppp (h dellim -
log
log
n
Σ hov Freses hatar A= )p( hatart 0*)
Σ Cillinn ED O2 O. [ddeim 0 ) (Adelim 0 0+)
1=1
(67)
Σ haan 1 1p(h start S)
Σ Dill O,Indelm .0)
log
Σ +n(log(c2)-
log
C7 C7 p Mest Sn
Min eDp (O, O0 B+))
1=1
(68)
Σ Sading p (O OO .))
1 ( -ogg())) n
11 >log log Σ Em ((O f*)
(69)
In the above steps, we used both Propositions I and in the terms involving c2 (bounding the probability of delim hidden states) and ey (bounding the probability of Twa)). Note that in the second line, the sum can must be over the set of delimiter states D by using the assumption that observing delimiter delim implies that the corresponding hidden state dellim must be in D.
16

---

Published as a conference paper at ICLR 2022
Focusing on the numerator of the ratio term and summing over the start hidden state for the i-th example,
Σ
p(O,14,-T00) 0) = Σ
Ep(0,16, ()p(h 14,-1 (00-mm.0). .))
(70)
METED
h Ri-1ED
p p(h 1- .8)
p(Os 0)p(hqum
Σ
(71)
p(h (
Addinn ED
p plo In 0)
p(O,|ng ()p(h
star
(72)
p(
A
1 ED
where the last step applies Bayes' rule. We can lower and upper bound the following quantity for any i using Assumption 2
p p(h (1 (ngium 0)
p p( /1
0)
(73)
p(h (hoeelim (8)
Cg
p( (clclimm 11 Pugtur .)
p (1 Netter 0))
(74)
p ph ())
C4
This implies that
p(h ( Iggaa
-
(75)
p h (hdellm ())
C3
MARED
p p(ha-iim h .0)
1
(76)
p p( (1 (
C4
Adlim ED
Plugging in these bounds, we have
Tr, (0) - [ ( ( (ogf:c/--2ooloaee2)--aaaa)----(1 log((s)) D log
Σ Aoun p(O
0)p(1:=m00) (0)
M pl (Oihh:t
ppp (+)
(77)
=1 1 (llog(c))+22((ogg) (ogp log(C)))
log p(0,10)
(78)
p(O,10-))
log p(010) p(0)0+)
00 Eo Percupes
deelim
(79)
where set
delim 2(l0g(())))))))) (lggc) log(cs) log(a log(c))
(80)
Next, we convert the expectation in the bound into KL divergence. We have
log p(00))
Eo Eo-ppeeee log p(0 0) + log Prrompe ( (0)
Eorepparei
(81)
Pprompr (O)
p(O
K L' (Ppromps Ip(-10")) - L (pprompt) (p--
(82)
We will upper bound the first term:
log Preampp (0) p( (+))
K [ [|l((|(pp( =Bo))==o~ Parorge
(83)
Expanding the numerator and denominator of the ratio inside, we have
&
Pprompu (0)= = >
http:///w.iiaiioooooooo..iannnna Prommpp (F11)
(84)
H
J=2
p(010')= = Σ 2p(H))HH))))))) [ | -1)00)
(85)
H
17

---

Published as a conference paper at ICLR 2022
which differ in only the hidden start distribution. Using Assumption 5 any E H. which implies that
we have that p(hl0*) cy for
Pprompe(ft (h)
(86)
(())00)
C8
1 Prrompe (0)
p(0(0))
(87)
C8
Finally, this implies that the KL term is bounded
K (pppomp -log(c)))
(88)
This term is non-negative since eg SI
Aiming to decompose the second KL term into over the h tokens, write =(00)==)000]=0000:----- and pprompe (0) Prompe p (01]==00) We have
Prromp (O)
K L (p(-10) = Σ Proompe (0)
(89)
p(O Σ Prrompp (0)
Pprmpi 0b]]0[1:--11)
log
(90)
p [0]0
Pprompi (0]]]0[1:j-11)
Pprompt (0 log
(91)
p(00j]0[1;j-11.0))
jml 1
O
Eolly 1/
(p (1pb)
(92)
5
Then we have that
lim Tr (0) 100
Eo 1100----ppppe /=1
[K (pprompplll))) + c + delim
(93)
The second term (set 0 start = log( (1)) )) is an error term that depends on how different the starting prompt distribution Prrompt (which is part of Pprompp) is to the pretraining distribution. The third term is an error term that comes from the delimiter transitions. The bound is negative when the sum of KL terms is larger in magnitude than the error terms. Note that becomes larger, the number of observations of 0= "overpowers" the distracting transitions in the prompt distribution. This condition is equivalent to the disinguishability condition [Condition[].
By assumption, for 0 of 0* the Condition Condition[holdd. and thus
lim plS p((Sx=F1sst))) lim exp(n-r.(0)) =0 12 12 . (*)) n-o00
(94)
since r,, (0) has a negative, constant limit. Note that exp(n-r,, (0*)) =1 for 0"
E NON-DISTINGUISHABLE CASE
When Condition is unsatisfied, Equation 14), gives upper bound on the sum of KL divergences for the next token distributions given different-length histories. In contrast, the in-context task only measures the accuracy of the last (k-th) token. The main challenge is to relate the different-length histories to each other to give a more precise bound for the error on the in-context task (last token).
Before addressing this challenge, we give the following lemma, which leverages the result of Avila Pires & Szepesvári (2016): (teiiwarrt (2007) to relate bound on the KI divergence 0-1 loss.
18

---

Published as a conference paper at ICLR 2022
Lemma 2. Let the set of 0 which does not satisfy Condition 1 to be B. Assume that
K [PPrropp Wrest Frest) l[p(( Treat, 8) is bounded above for all and that 8 minimizes the multiclass logistic risk LCE(O) - Ppoaaye CacinaaeePPrrpppp Pprrmpr Yeest Fustt llgpl (oopp(yrellea Fuss 0}}. 1f
E Prencpp [K L (Pprompr (yrear (Zees)) (1p()ten Tren 0))] Seg for all 0e B,
(95)
then
lim Sin 2=000
supe@
(96)
where
g( 1 (10)=((16))00(100)--
(97)
is a calibration function for the multiclass logistic loss for & E [0,1].
Proof. First, that can study the 0-1 risk of the limiting predictor:
lim (fn( n=000
lim Em Bhe Prncaps. [[ (Fres) # Mtess]
(98)
200
Exs thet Pyonng
lim 1 [[ (Frest # yessc] (dominated convergence, boundedness of indicator)
(99)
Emps tiee Promm
lim fa (Frea) + yhesll
(100)
n1000
where in the last step we use that since the output space of fn discrete and the probabili- ties that the in-context predictor takes an argmax over converges, then for N large enough, JNv (f (Frea).
Note that for every input Tiest the limiting in-context learning predictor outputs the argmax of predictive distribution which can be mixture of predictive distributions B:
lim fa (Trea) (rg n
(101)
for some distribution 9 over B. The KL divergence between this mixture and the prompt concept is bounded by the KI. divergence of any one 0 E B, due the convexity of KL.
E Engs Be Proonee K L (Pprompl (y) (rest) Ea-in [p(y) (ieest O)]]
(102)
B Proeep [Egnug K L (Prromppl (y) Trest) [|p (eesst
(103)
BBang Beees Pansage
K (P (y) (hest) [|p (esss
(104)
SsuplEmarr Pronnus
( prompp (y) Trest (lpp)lppppppp ())]
(105)
oE
where exchange the order of expectations since the KL bounded doonnattedooovergeeeee
From the KL bound KL (Pproppe (Bresa Frest) (1eed Frest f). we thus have
Fress E E -o Preomas K [ Prompp (Busa Frest ) p1 / 0)) Lcu(O) ( LCu( $ suucco
(106)
8EB
where Lce(f)= = Er - Proaae Pprompe (Bees Tust llogo(((sss / [)) is the multiclass logistic risk, and
LcE(O*) is the optimal risk over + by assumption. Applying Theorem 2.2 and 5.11 of Avila
Pires & Szepesvárij (2016), g is calibration function for the multiclass logistic loss, and allows us
convert the surrogate risk bound bound on the 0-1 loss, giving the result. Note that we have zero approximation error here, since 6* e
Note that g is roughly linear in for smaller than 0.7, where the bound is non-vacuous.
19

---

Published as a conference paper at ICLR 2022
E.1
PROOF OF THEOREM
Proof. By the continuity assumption, we have for any in B that
k Σ
&
KL, 1L,
(10 2
Σ 0 -8) 1 (0- + (0-00)1100000000)00000000000--0-
(107)
2
1=2
)=2
2 2
(k- (k- 11-1))min
(108)
E + Sdelim
= |00 0 11
(109)
(k- (min, Amin (1,,8))
We use this bound the last KL term by plugging in below:
1
K La (0"110) (0-0") 0-) (0 -0--0))0 +0000000000000-00001
(110)
2
1 2
(max Amax -000
(111)
(c + Edetion (max, Amax +
(112)
(k- (k-1))m 1)min Amin (7500))
Rearranging and noting that K L Pasag [K L (pprompo (Hrest (rest A1p())les)))) (rest 0))] we have
(Estar + l Edllim (max, Amax (4,00))
Exerr Peneape
[K L (prrompt
(Bhest (Shest/ (Fest) (1pee Trest, 0))]
(I 1) 1ini Amin (1
(113)
Plugging into Lemma 2 gives the result.
E.2 PROOF OF THEOREM 3
Note that Condition I ensuree that the sum of KL divergences between positions within a k-length input is bounded. This means that have bound not only the last-position KL divergence, but also for all the intermediate tokens. Intuitively, the random length test example allows the in-context predictor "take credit" for fitting the intermediate tokens. The proof is immediate given the KI. bound and Lemma 2 given that the length of zestt is uniformly random between 2 to k.
Proof. Let the of 0 that does satisfy Condition be B. We have for any in B that
E E PProon-P
K T. Pppromp (Jhea Fress) (1peet Whest 0))]
(114)
b
1
Bo 1 11 K (1110:0 1000://01L0--100000000----------
j=2
(115)
supp (E
(116)
by Theorem and Condition
Plugging this into Lemma gives the result.
n
F
EXPERIMENTAL DETAILS
F.1 GINC DATASET
Pretraining distribution. We consider pretraining distribution from mixture of HMMs with an interpretable hidden state structure and emission distribution. The HMM hidden state h, = [sr,2]] at time is composed of an entity E {l.....\Vl] (e.g.. Einstein) and property 5, E (l. (e.g.. nationality, first name, last name, other grammatical tokens). We model the entities and properties as independent Markov chains (i.e., factorial HMM (Ghahramani & Jordan 1997)). ), while the emissions depend on both. In pretraining documents, we expect that the entities (c.g., Einstein) change slowly over time while and the properties of the entity (e.g., their nationality) change quickly with some pattern generate
20

---

Published as a conference paper at ICLR 2022
Pretraining document
In-context Prompt
f / h x o k au ap / a O u au ae
ao an ah
u as a k au j w ax 1
1 aw ac / ax aj ae / ac
aw ae au g
ap
u
aj ae d a h x af aj i
r j w as a yxsnni ap
Figure 8: Example pretraining document snippet (Left) and example prompt with 3 training examples, 1 test example, and example length 3 (Right). The delimiter token is the backslash.
Properties (s)
Memory matrix
M
Entttles (+)
\n
Albert
Einstein
German
was
\n
Mahatma
Gandhi
Indian
was
Emplrical HMM model (Prompt distribution)
0
Transition matrix for properties (s)
Hidden
states
1
1
2
2
4
1
1
4
4
Prompt
M M[P sc]
Albert
Einstein
German
Mahatma
Gandhi was
Indian
Figure 9: The GINC dataset generates sequences from a mixture of HMMs. The HMM hidden states consist of entities (r) and properties (s), which index into memory matrix produce the observed token. The entity and property sequences are sampled from independent Markov chains. The concept parameter / is the transition matrix for properties, which defines relations between properties. In this example, the sequence of properties [2,3,5,4] relates names to nationalities, defining the in-context task. The blue color represents hidden states/observations sampled from the prompt distribution, and the purple color represents hidden states/observations sampled from the pretraining distribution.
natural sentences. We implement this by ensuring that the probability of transitioning to the same entity index in the next step is at least 0.9. The emission distribution depends memory matrix M with [ rows and [S] columns (Figure At step I, we use the entity e; and property 8, to index into the memory matrix. In particular, the observed tokens deterministic with p(o, (h.) if o M [v),s.) This construction satisfies the structure on delimiter states (Assumption[]) We ensure that all the transitions have nonzero probability and uniform prior over concepts, satisfying Assumptions 2nd
Concept parameter. The concept parameter is the property transition matrix, while the entity transition matrix is fixed for all concepts. The prompt start distribution and the concept together determine the in-context task. We define uniform mixture of HMMMs family O of concepts to generate 1000 documents with ~~ million tokens total.
Vocabulary. The GINC dataset generated from a mixture of HMMs. These HMMs output tokens from vocabulary of size in (50, 10,,100,,1))) The vocabulary contains special delimiter token (backslash
21

---

Published as a conference paper at ICLR 2022
35
90-
60
30
K-10 550
K=10
25
60
20
30
Num examples
mcamples
Num examples
Figure 10: In-context accuracy curve of the 4 layer Transformer the GINC dataset when the entity transition matrix does not have an additional identity component, for vocabulary sizes 50 (left), 100 (middle), and 150 (right). In-context learning still generally successful.
see Figure designated to index The vocabulary generated as combinations of letters starting from a to then aa to az, and so on. All sequences are tokenized by splitting on whitespaces.
Memory matrix. The shared memory matrix has entities and 10 properties, totaling 100 entries (corresponding to 100 hidden states). The first column of the memory matrix is fixed to be the delimiter token, while each remaining entry of the shared memory matrix is populated with a token sampled uniformly from the vocabulary.
Transition matrix for properties. We generate 5 property transition matrices, for each component of the HMM mixture. We generate each transition matrix via a convex combination of 100 random permutation matrices. The weights of the convex combination are randomly generated as
softmax ((--0.))
(117)
where E 100 has uniform random entries in [0,1] and / is temperature parameter, set to 0.1.
Transition matrix for entities. The entity transition matrix shared between all the HMMs that consistute the mixture. The entity transition matrix is generated in the same way as the property transition matrices, except with one additional step. Letting T be a transition matrix sampled in the same way as property transition matrix,
In pretraining documents, expect that the entities (e.g., Einstein) change slowly over time while and the properties the entity (e.g., their occupation) change quickly with some pattern to generate natural sentences. We implement this by ensuring that the probability of transitioning to the same entity index in the next step is at least 0.9. The final entity transition matrix is then 0.17 +.997 where / is the identity matrix. Although we add the diagonal component for added realism, adding this component. Figure shows in-context learning curves for a small (4 layer) Transformer trained on data that does not add the diagonal component (we check this for vocabulary sizes 50, 100, and 150). In-context learning still works in this case, although not as well for the 50 vocab size case.
also consider not
Start distribution. The starting distribution for the hidden states in all HMMs in the mixture close to uniform. We generate the start distribution as softmax ((u-0.5) 0tt)/t) /t) for random vector u with entries uniformly from [0,1] and temperature f = 10. In the pretraining documents, we only sample from the start distribution in the beginning of the document.
Prompt distribution. To generate the prompts, we first sample concept 0 uniformly at random from well-sppecification. Assumption[4) 4). then use to generate all the prompt examples. The prompt start distribution is chosen to be uniform over entities but with fixed starting property that is chosen randomly for each prompt, for consistency in the task. This may not satisfy Assumption 3 but we found this to still work empirically and is simpler. Given the starting property, we sample & tokens from the HMM defined by the concept 0. Finally, we append the delimiter token for the example. We repeat this process for each example in the prompt, concatenating all examples. The label is generated as
argmaax Pprompo (y (rest)
(118)
under the prompt concept 6*. This differs from the theory, which samples Bees instead of taking it to be the most likely token. However, there can be large amount intrinsic error that sampling introduces. We define the label this way in the simulations to remove the intrinsic error from sampling.
22

---

Published as a conference paper at ICLR 2022
Example of prompt generation. In the example in (egurr@/rrihh)) the starting property is fixed to be 5 (for example). The first token (1) is generated by sampling a random entity index (3), and indexing into the memory matrix returns I. Rumning the hidden state chain of the HMM forward gives the next pair of property and entity. Since the entity Markov chain changes slowly, the entity is still 3 in the next step however, the property has changed to 4, and indexing into the memory matrix outputs the next token (aw). Following this same process to generate the third token (the output for the first example), we finish generating one example. end the example, we append a delimiter (backslash). We repeat this example generation process for all the examples, except for the test example at the end, where we do not generate the last token. We condition the HMM on the generated prompt to compute the posterior distribution over the next token Pprompe (1) (rea) ). We take the argmax of this distribution to be the ground truth label.
Dataset details. The dataset contains 1000 training documents and 100 validation documents, where training documents have 10240 tokens and validation documents have 1024 tokens. Each document is generated by first selecting one of the HMMs from the mixture uniformly at random, then generating 10240 tokens from the HMM.
We also generate 2500 in-context prompts for each (example length.number of examples) pair, for example lengths == [3,5,8, 10) and number of examples n = [0,1,2,4,8, 16,32,64, Each prompt generated using a random HMM in the mixture.
F.2 TRANSFORMER DETAILS
Our Transformer models are based on the GPT-2 architectures with 4. 12, and 16 layers respectively, with 12 attention heads, 768 dimensional embeddings, residuallembeeddinngateeeitt and context window of 1024. Other than the number of layers, the other parameters are the default settings from the Hugging library (Wolf et al., 2019). We train for epochs using the Adam optimizer (Loshchilov & Hutter 2019 Kingma& & Ba 2015 with a batch size of 8 and a linear learning rate schedule (with 1000 step warmup) up to a learning rate of 8e-4 for the 4 layer and 12 layer model, while for the 16 layer model we start with a constant learning rate of 8e-4 and reduce by factor of 0.25 whenever the best validation loss does not improve. We tried both learning rate strategies for all models and take the most consistent. We tuned these models so that the training loss curves between seeds have smaller variability between the runs in terms of the curve shape and when the loss decreases we found that this is an important indication of stable results. The models took 50 minutes, 2 hours, 3 hours to train respectively. The hardware was mainly Titan Xp GPUs, trained and evaluated using 1 16-bit precision. All the results are reported with pretraining runs (5 different seeds).
dropout set to 0.I,
F.3 LSTM DETAILS
We train an LSTM language model with embedding size 768, hidden layer size 768, and layers. We use dropout 0.2 and weight decay 1e-5. The optimizer is Adam starting with a learning rate of 1e-3, then reducing by a factor of 0.25 whenever the best validation loss does not go down. We train for a total of 10 epochs, with gradient clipping at norm 1.0. We use batch size of 8 and packproa through time for 1024 steps (each pretraining data segment is also 1024 tokens). Each model takes roughly 2 hours to train Titan Xp GPUs.
F.4 VARYING THE VOCABULARY SIZE
To do well on the in-context learning task, the model must both infer the prompt concept and the last HMM hidden state. In general, increasing the number of observable symbols makes the in-context task casier by making the inference of the HMM hidden state casier. With more symbols, each hidden state is more likely to output a different symbol, making the inference problem easier. This improvement comes despite the number of output classes in the problem (same as the vocabulary size) increasing. Figures II 12 13 14 show in-context learning curves for vocabulary sizes 50, 100, and 150, keeping other hyperparmeters of the dataset the same.
F.5 EXPERIMENT ON GPT-3
We conduct additional experiment which shows that longer examples improve in-context learning in GPT-3 on the LAMBADA (Paperno et al. 2016) completion task.
Data. In this experiment, we define a short version of the LAMBADA test dataset (LAMBADA test-short) which contains only test examples with up to 200-300 characters in length. We also define two "training" datasets from which to sample examples for the in-context prompts from. The short training dataset (LAMBADA train-short) contains examples from the training set that
23

---

Published as a conference paper at ICLR 2022
70
500
K-10
50
K=10
70
30
Num examples
mecmples
Num examples
Figure 11: In-context accuracy of the 4 layer Transformer on the GINC dataset for vocabulary sizes 50 (left), 100 (middle) and 150 (right). Accuracies generally improve as the vocabulary size increases.
90
100
K=5
ke5
k0
80
50
40
60
20
40
60
20 sccampls
40
60
20
40
60
examples
Num exemplis
Figure 12: In-context accuracy of the 12 layer Transformer on the GINC dataset for vocabulary sizes
50 (left), 100 (middle) and 150 (right). Accuracies generally improve as the vocabulary size increases.
km3
90
100
ke5
70
K-10
70
40
20
40
6b
20 Num excmppes
40
60
20
40
Num examples
Num excmpples
Figure 13: In-context accuracy of the 16 layer Transformer on the GINC dataset for vocabulary sizes
50 (left), 100 (middle) and 150 (right). Accuracies generally improve the vocabulary size increases.
20
40
20
40
60
20
40
20
40
60
Num wormppes
Nurm wourmphp
Num eocemppe
Figure 14: In-context accuracy of the L.TT on the GINC dataset for vocabulary sizes 50 (left), 100 (middle) and 150 (right). Accuracies generally improve the vocabulary size increases.
200-300 characters in length, which matches the distribution of test-short. The long training dataset (LAMBADA train-long) contains training examples that are 500-600 characters long. We cut the number of examples in the larger of the training datasets that the two training datasets equally sized (47 examples). For each test example, we sample random training examples (5-shot learning).
also consider equalizing the total length of the prompts two ways. First, we consider duplicating the 5 short examples (if the examples are [1,2,3,4.5], duplicating refers to This allows for equalizing the total length without increasing the number examples. As skyline comparison, also consider sampling 10 independent short examples, which contains more input-output pairs for the task.
Table shows that when evaluating only on LAMBADA test-short, 5-shot in-context learning using LAMBADA train-long improves the test accuracy by almost 1% compared to LAMBADA train-short, despite the long/short distribution mismatch between train and test. This supports intuitions from our theory.
Result.
24

---

Published as a conference paper at ICLR 2022
Prompt example length
Test Ace (200-300 chars)
5 examples
Short (200-300 chars)
69.8
Long (500-600 chars)
70.7
10 examples
Short, duplicated examples
69.6
Short, independent examples
71.4
Table 1: Accuracies for 5-shot in-context learning of GPT-3 on filtered LAMBADA test set with short examples (200-300 characters). Even though there distribution mismatch with the test set, having longer examples improves the accuracy, supporting theoretical intuitions. The first two rows use 5 train- ing examples the prompt, while the last two rows use training examples to equalize the total length.
In comparison, simply increasing the total prompt length by duplicating the short examples does improve the accuracy. Intuitively, the longer examples have additional information that is not directly related to mapping between the input and output, but can be leveraged to improve in-context learning by helping the model infer the latent concept. Using 5 long examples (as opposed to 5 short examples) closes about 56% of the gap between using short examples and 10 independent short examples despite not adding additional examples or task-related information.
25


Reviews:
[{"review_id": "lyfl48tFfSI", "reviewer": "Reviewer_j2L6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Main Review\n\n#### Strengths\n\n1. Significance: The paper studies a timely question: Why does in-context learning (in particular in LLMs) take place? The small-scale reproduction and theoretical model of this phenomenon may lead to new understanding.\n\n2. Novel empirical phenomenon: The existence proof of in-context learning in small-scale models is new and surprising, to my knowledge.\n\n#### Weaknesses\n\n1. I am not sure of the novelty of the theoretical results. The theoretical model considered in the paper seems to be a mixture Hidden Markov Model (HMM). Helske & Helske (2019) demonstrate that such a model can be expressed as a standard HMM, for which inference is known to be possible and moreover efficient. Given this, it is not clear to me what Theorem 1 adds, which shows that the \"in-context predictor\" (ie. Bayes) infers the latent mixture variable. Theorems 2 & 3 seem novel to my knowledge, as they tie this inference to the expected 0-1 loss, but further clarification on the significance of these results would be helpful to understand the contribution.\n\n1. Broadly, I think the paper should more clearly distinguish ideal performance (ie. Bayes) from models picked out by their behavior (eg. \"in-context predictor,\" Transformer, LSTM) from datasets (ie. pretraining distribution, downstream task). As examples:\n    - In Theorem 1, 2, & 3, Bayes is shown to recover latent variables and make accurate predictions in the mixture HMM; i.e., these theorems describe ideal performance. However, in some cases, this ideal performer is described an \"in-context predictor\" (\"....we have that the in-context predictor infers prompt concept...\"). Elsewhere in the paper, \"in-context predictor\" is used to describe a behavior (ie. learning a task from a small context). Using \"in-context predictor\" widely in this way is problematic, because stating that the ideal model is an in-context predictor is circular with respect to the claim that in-context predictors can be modeled as a Bayesian ideal (which is one of the central claims of the paper).\n    - In Section 4, the pretraining distribution is modelled as a mixture HMM, and data generated from this model is used as training data for scaled-down versions of LLMs . This is a distinct use of the mixture HMM formalism (ie. as a data-generating process) from its use to derive the ideal performance described in Theorems 1-3. This distinction could be made clearer in the paper.\n\n1. The results in Section 4 are claimed to demonstrate that in-context learning is taking place in this small-scale setup. However, if in-context learning just refers to improvement in the number of test examples (ie. the behavioral definition), then these results are quite distinct from the theoretical results. Can the asymptotic performance also be plotted to demonstrate that empirical performance approaches the ideal performance described in Section 3? This would more closely tie the \"in-context\" phenomenon to the theoretical model.\n\n1. The paper should more clearly discuss scope & limitations, and, in particular, which assumptions in the theoretical results and data model may not match reality. For example, the topic-model/mixture assumption; the enumerable token set; the assumption that the \"language model fits this pretraining distribution exactly with enough data and expressivity.\"\n\n1. The paper is a little light on take-aways for LLMs. It has reproduced the in-context learning phenomenon in smaller models, but there lacks a strong connection to argue that the same mechanisms are at play for larger-scale models.\n\n### Minor comments\n\n#### Minor, specific sections:\n\n- Intro & Figure 1: I think the example in the introduction is better served in the caption of Figure 1, and the introduction should describe just the abstract problem and reference Figure 1.\n- \"Thus, in-context learning can be thought of as implicit Bayesian inference.\" I think this should be more explicitly explained as \"we model in-context learning as Bayesian inference with this particular HMM.\"\n- \"Ideally, the posterior predictive distribution should concentrate\" Is this referencing prior result? Or an assumption that would be beneficial for the framework? Explain further.\n- \"During in-context learning, the prompt is presented as a contiguous sequence rather than IID examples and are thus “out-of-distribution” to the language model.\" I'm not sure I follow why this fact makes the test prompt OoD.\n- \"The canonical asymptotic tool in Bayesian methods...\" The reason for introducing this should be much more elaborated. What is the purpose of looking for a \"tool\" like this? What does the tool do?\n- \"We take the prompt concept $\\theta^*$ to be unique in this paper to simplify the notation of the prompt distribution $p_\\text{prompt}$.\" I'm not sure why this would be just to \"simplify the notation;\" is it not in place to ensure that the test prompt does not occur in the pre-training set? Perhaps \"unique\" is unclear.\n- \"The target distribution differs from the pretraining distribution p on the distribution of $h_\\text{start}^\\text{test}$.\" And not $\\theta^*$ as well?\n- In Eq. (4), $p_\\text{prompt}$ is overloaded---it represents the joint, whereas before, it represented only the conditional. \n\n#### Minor, general comments:\n- The paper should discuss work on overparameterization (e.g., Zhang et al. (2017)) and overtraining (e.g., Power et al. (2021)), which do not change the train error significantly but do affect downstream performance. These lines of work are relevant to the observation that \"the inductive bias of scale may improve in-context learning beyond memorizing the training data better.\"\n\n### References\n\nHelske S, Helske J (2019). “Mixture Hidden Markov Models for Sequence Data: The seqHMM Package in R.” Journal of Statistical Software, 88(3), 1–32. doi:10.18637/jss.v088.i03\n\nPower, Alethea, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. \"Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets.\" In ICLR MATH-AI Workshop. 2021.\n\nZhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. \"Understanding deep learning requires rethinking generalization (2016).\" arXiv preprint arXiv:1611.03530 (2017).", "summary_of_the_paper": "The submission studies the phenomenon of the \"in-context learning\" behavior of large language models (LLMs). Two results are presented: First, the submission argues that in-context learning can be formalized as Bayesian inference in a mixture of Hidden Markov Models (HMMs). Second, the submission introduces small-scale settings in which the in-context learning behavior of LLMs is reproduced empirically.", "summary_of_the_review": "The question (S1) is highly relevant, and I think the small-scale reproduction (S2) could be meaningful. However, the implications of what is shown in the paper for LLMs is not sufficiently clear (W5), so the significance of the work in answering the question (S1) is not clear. The paper also has a few other weaknesses that prevent me from recommending acceptance in its current form (W1-4).\n\n### Update after discussion\n\nThanks to the authors for their detailed response. My concerns have been largely addressed and I appreciate the greater empirical connection to LLMs via the additional experiments; I have updated my score to reflect this. I do have recommendations to the authors to increase the clarity of the paper, which I do think should be improved for the next revision:\n\n- I agree with Reviewer BHn6 that the title overclaims and does not reflect the contributions of the paper. The current version is suggestive of something like \"An Explanation of In-context Learning [in Large Language Models like GPT-3] as Implicit Bayesian Inference,\" which is *not* the contribution of the paper, because the paper explains in-context learning in a simplified theoretical model, and provides some  empirical evidence that this captures important aspects of the same in LLMs, but does not strictly \"explain\" the same in LLMs.\n- Related to the above, everywhere in the text where the term \"language model\" is used should have greater precision on the distinction between the idealized setting and the practical setting, because this term is used interchangeably for both the theoretical model and a large language model. \n- I find the terms \"in-context predictor\" (theoretical notion) and \"in-context learning\" (behavioral notion) confusable. It would be worth writing \"*ideal* in-context predictor\" and \"*behavior of* in-context learning\" to make this less so.\n-  OOD = low probability is an imprecise notion and one that does not seem to be connected to how the structure of the prompts is used in the analysis. I think the authors can improve on this point by clarifying what they mean by \"we can no longer factorize the examples under the pretraining distribution.\"", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "c-7zHSmhgK", "reviewer": "Reviewer_BHn6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths__: The main strength of the paper is in the connection (and differences) it raises between the ability of ICL to exploit prompts consisting of a sequence of examples of the same type and the Bernstein-Von Mises theorem in Bayesian theory that characterizes the convergence of the posterior to the true underlying parameter, given enough examples.\n\n__Weaknesses__: While the paper goes into mathematical technicalities about certain formal details, it does not explain clearly certain high-level, core, aspects of the approach, which make it difficult for the reader to really understand what is actually claimed. My first question below raises this point in more detail. \n\n__Questions__:\n\n* It is crucial for the reader to clearly understand how the \"pretraining distribution\" is actually constructed. Although I could not find it detailed in the paper (and actually found some statements that seemed to contradict each other, see below), my understanding is that, after a concept $\\theta$ is drawn, a __sequence__ of examples of the concept is produced, separated by a special symbol. The total number of symbols produced is $T$ (is this correct?), a parameter which is not detailed. If this interpretation is correct, am I right to assume that it means that the pretraining distribution, for a given $\\theta$, generates essentially the same sequences as the \"prompt distribution\" for that $\\theta$ ?\n\n\t* Note: Another interpretation, slightly different from the one I just wrote, would also seem to be supported by the paper, namely that while the pretrained distribution is not _exactly_ the same as the prompt distribution, it does ensure that sequences of examples are generated in almost the same way, but I think the nuance is subtle.\n\n    Assuming my interpretation is correct, the formal part of the paper essentially says that, (under some technical conditions), if we let the pretraining distribution generate a long sequence of examples (but without disclosing which concept $\\theta^*$ was employed), and then rerun the (full) pretraining distribution conditionally from that prefix, then the next example generated will tend to be an example \"associated\" with $\\theta^*$.\n\t\n    If this is the main observation that the paper makes, it should be stated more openly, even if there is a danger that some readers may dismiss it as a bit \"obvious\" (which it is not really at the formal level).\n\n    * Concerning statements that look to go in different directions relative to this point: the caption in Fig. 1 says \"... and are thus 'out of distribution' \", but the example in Fig. 2 seems to concatenate several examples. The paragraph after assumption 3 on p. 4 says: \"the prompt is still out of distribution\", while the next \"regularity\" assumption seems to be there to avoid such a possibility... \n\t* Related to these points, a clearer description of the way the GINC dataset is produced, in terms of whether and how it produces sequences of examples, would be very welcome. Also, the definition of the GINC should be related to the formal assumptions in section 2.1, which it is not.\n\n* Other questions:\n\n    * Please explain why you need a special treatment for the first symbol of each example in the description of the prompt distribution in the first half of p. 3. Is this related with the different color used for \"Albert\" and \"Einstein\" in Figure 1 ?\n\t* In your simulations, the fact that neural LMs such as LSTMs or Transformers are able to recover the completion of examples should be sufficiently explained by their ability to well approximate the pretraining distribution underlying the GINC dataset, without having to invoke additional ICL abilities. However, your experiments indicate that their ability to do completions goes beyond their ability to approximate the underlying distribution. Do you think your theory could be extended to explain that phenomenon, or is it orthogonal to it?\n\n__Comment on the title of the paper__: as a last comment, I feel that the title, claiming to provide an _Explanation_ of iCL in terms of Bayesian learning, exaggerates the current contribution. A true explanation would require to go into how _the hidden states of a pretrained Transformer model_ focus more and more onto a characterization of the task being illustrated by the sequence of examples in the prompt (and typically do so based on only a few examples) and are often able to solve that task even if these examples appear to be remotely related to the training distribution. This is not what the paper does, so I think the title could be reformulated in a more cautious way.", "summary_of_the_paper": "The paper attempts to bring clarification to In-Context Learning (ICL) by relating it to Bayesian Learning over mixtures of HMMs. Each component of the mixture generates examples of facts of the same conceptual type, such as \"Albert Einstein was German\", \"Marie Curie was Polish\", etc (here the conceptual type is \"nationality\"). The first part of the paper addresses a theoretical question, namely whether the mixture model, on being given a prompt, that is, a prefix such as \"Albert Einstein was German. Marie Curie was Polish. Isaac Newton was \", is able to implicitly identify the relevant component of the mixture (i.e. \"nationality\"), and consequently, whether it is able to correctly complete the prompt, here with \"British\".\nThe second part of the paper consists in simulations, where the mixture model is used to produce data on which a neural language model (LSTM or Transformer) is trained, and on evaluations of how well the neural LM is able to perform the completions, under different configurations and ablations.", "summary_of_the_review": "The paper raises interesting connections between Bayesian learning and certain aspects of in-context learning, based on a simplified formal model exploiting mixtures of HMMs. While the formal developments are extensive, the take-home message is somewhat obscured in the description, making it difficult to assess the true impact of the paper.\n\n\n*After reading authors' responses*: Thank you for your detailed answers and for the clarifications in the paper. I am not fully convinced by the large significance you put on the OOD nature of the prompts relative to the pretrained distribution --- technically true, but in a narrow and not very illuminating sense IMO ---, and still think your title is an exaggeration. However, I feel that your paper deserves to be discussed by the community and therefore I am raising my score, leaning towards acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YEdG6dhyfYj", "reviewer": "Reviewer_qavU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper does a good job in explaining why concatenating independent examples as a sequence and prompting a LM trained on HMMs gives good results, considering the prompt sequence can be OOD. I have some questions about practicality of the approach and the observation generation process for the prompting.\n\n1- I think there is a significant gap between simulated experiments and real sequential data. Showing that the theory holds for the simulated data as the initial step is useful, but HMMs are far from capturing real data and results with existing LMs would be needed to understand the gap. For example, GPT-3 zero-shot results are actually better than GPT-3 one-shot results on LAMBADA and HelloSwag datasets (Table 3.2) [1] and few-shot results are also lower for smaller networks (Figure 3.2). \n\n- There is an additional result in the appendix related to GPT-3 but it is not clear if using longer examples gives better results or because the overall prompt sequence is longer. Can you also present results where you use more short examples that gives the same prompt length as that of longer examples (such as using 10 short vs 5 long examples)?\n\n- GPT-3 uses an additional task description, which would be similar to the concept in your paper, could you explain how this can be incorporated?\n\n2- The example sequence (O_i) is generated using the same pre-trained language model. In this case, OODness of the prompt sequence only comes from the fact that examples are concatenated. I think having prompt distribution not necessarily sharing the same pre-training data distribution could be important.\n- Could you train an additional language model on your corpus and generate prompt observations from that?\n\n\n3- How sensitive is your results to different orders of the same set of examples?\n\n[1] Language Models are Few-Shot Learners. OpenAI.\n", "summary_of_the_paper": "This paper studies the question of why few-shot prompting works for large language models (LM) such as GPT? The paper is mainly focused on understanding why prompting works given that concatenating multiple examples separated by a delimiter introduces out-of-distribution prompts that the LM would most likely not have seen during its pre-training. The paper shows that when the pre-training distribution is a mixture of HMMs, prompting works as a result of Bayesian inference. The authors show the validity of their theoretical results by training transformers and LSTM on synthetic datasets sampled from some HMMs that they devised.", "summary_of_the_review": "I think the theory behind the paper is really interesting to understand in what conditions few-shot prompting might work. I also think that there needs to be more experiments with real data to understand the gap between their simulated experiments and real language.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N5AVz0GZ8W", "reviewer": "Reviewer_VX98", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes and analyzes a framework under which \"in-context learning\" can be studied from a statistical perspective. Despite the potential limitations of the framework and assumptions, this is an original perspective that allows for theoretical investigation of a complex empirical phenomenon. My main comments are focused on three topics:\n\n1. It would be useful for the authors to give a more robust account of why we need a specific account of \"in-context learning\" in the first place. Put differently, why should we not just consider these examples to be relatively straightforward cases of next-word prediction for a model that has learned an accurate approximation to the distribution over sequences of words? In the first paragraph of $\\S$1, the authors write, \"Given that the distribution of prompts are quite different from real text...\" - but is this really true, particularly given the enormous breadth of modern training corpuses? The prompt that precedes the quote seems reasonably close to natural (if somewhat simple) language in this case, certainly enough for a reader to reasonably ask whether there is anything particularly special here that needs explaining. How do we distinguish between \"recognition and continuation of a specific concept / pattern / construction\" and just \"good next-word prediction conditioned on a relatively plausible sequence?\"\n\n2. The overall strategy of the analysis is clear, particularly the high-level walkthrough that precedes the theorem statements and proofs. It is also clear that the analysis depends at several points on specific, technical aspects of the problem formalization, including the mixture-of-HMMs model for the pretraining distribution (Eq. 5) and the specific structure of the HMM transitions around delimiter tokens (Eqs. 9-10). The paper could be improved by some discussion of these assumptions. In particular, what is critical to this framework, and what is required merely for technical purposes in some proofs? Why does the mixture-of-HMMs model seem to be the right choice for this analysis?\n\n3. The experiments are carefully documented and results include estimates of uncertainty where appropriate. The prediction performance as a function of $n$ and $k$ generally supports the theoretical results, and the ablations indicate the importance of the specific mixture-of-HMMs structure of the pretraining distribution for this task. However, some aspects of the experimental approach could use more discussion. One main question is: how does this toy setting help us understand the emergence of \"in-context learning\" in models trained on vast corpuses whose contents are very unlikely to be well-modeled by a mixture of HMMs corresponding to well-separated \"concepts?\" We see that this phenomenon disappears when the LSTM and transformer models are trained on \"random\" data, but the real-world data on which language models are trained clearly lives between these extremes of rigid concept structure and meaningless sequences. Some lower-level questions: Does the theory explain why performance seems to be relatively more sensitive to $k$ than $n$? Why do we observe \"in-context learning\" in these relatively small LSTM and transformer models, when previously it was observed that even models as large as GPT-2 fail to demonstrate this phenomenon? \n\nOther comments:\n- The first sentence of $\\S$3, particularly \"... conditioning the pretraining distribution on the prompt infers the prompt concept to enable in-context learning...\" is unclear.\n- What are $c_1$ and $c_2$ in Eq. (13)? What is $k$ in Eq. (14)?\n- To avoid ambiguity, I'd suggest using some other symbol for the footnote in Thm. 2. \n- Typo in first paragraph of $\\S$1: \"Intuigingly\"", "summary_of_the_paper": "The authors propose to study the phenomenon of \"in-context learning\" through the lens of Bayesian prediction. They introduce a framework in which the data generating distribution is given as a mixture of HMMs parameterized by a latent \"concept,\" and they show that under suitable conditions the posterior predictive distribution over a completion given a structured prompt asymptotically selects the \"concept\" behind this particular structure. They illustrate this framework and reproduce the phenomenon in a simplified language modeling example.", "summary_of_the_review": "This paper contributes a framework for theoretical exposition of a complex empirical behavior observed in language models. This framework and its results offer useful insights, and these are corroborated by a clear set of experiments. There remain some questions as to the motivation for \"in-context learning\" as a specific phenomenon of interest, the potential limitations of some assumptions that seem critical to the proofs, and the gap between the toy setting and \"in-context learning' as it arises in large language models. In my view the strengths outweigh the weaknesses, and my score could potentially be improved with some further discussion of the points above.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission introduces a theoretical model to explain how "in-context learning" (i.e. the ability to output a correct prediction based on inputs for a task that the model was not explicitly trained on) is possible. The model uses a mixture of HMMs and shows that in-context learning is a natural consequence of Bayesian inference under that model. Overall, reviewers agreed that the contribution was useful and timely, and were somewhat convinced by the theoretical arguments. However, there was some broad concern with the framing of the paper. Namely,
1) The paper claims that prompted data is OOD w.r.t. the pre-training distribution. In fact, this is almost certainly not the case for many tasks and datasets. Indeed, it is highly plausible that data very similar to the example given by the paper (identifying the nationality of different celebrities) appears in the pre-training dataset of large LMs. Other examples include the popular "tldr;" task format for summarization which is incredibly common on the internet, etc.
2) The paper does not sufficiently distinguish between insights gained in the toy setting considered by the theoretical model and insights that can be applied to large LMs. Most reviewers were concerned that there might not be any reason to think that the insights gained from the theoretical model would apply to large LMs. The paper, however, very much frames itself as developing insight into the behavior of large LMs.

I will recommend acceptance of this paper, but will stipulate that the above two issues should be fixed in the camera-ready version. Namely, I would suggest that the authors do not refer to prompted forms of tasks/datasets as "OOD", and I would suggest that any claims about different insights are not applied to large LMs.
Decision: Accept (Poster)