Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
Year: 2023

Abstract:
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without parameter updates. Despite the great success in performance, its working mechanism still remains an open question. In this paper, we explain language models as metaoptimizers and understand in-context learning as implicit finetuning. Theoretically, we figure out that Transformer attention has a dual form of gradient descent. On top of it, we understand ICL as follows: GPT first produces meta-gradients according to the demonstration examples, and then these meta-gradients are applied to the original GPT to build an ICL model. We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding. Experimental results show that in-context learning behaves similarly to explicit finetuning from multiple perspectives. Inspired by the dual form between Transformer attention and gradient descent, we design a momentum-based attention by analogy with gradient descent with momentum. The improved performance over vanilla attention further supports our understanding from another perspective, and more importantly, shows the potential to utilize our understanding for future model design. The code is available at https://aka.ms/icl.

heading: 1 Introduction
text: In recent years, large pretrained language models, especially in Transformer-based architectures (e.g., GPT; Brown et al. 2020), have shown strong emergent in-context learning (ICL) ability (Wei et al., 2022; Dong et al., 2023). Different from finetuning which needs additional parameter updates, ICL just needs several demonstration examples prepended
∗Contribution during internship at Microsoft Research.
Finetuning
before the query input, and then the model can predict labels for unseen inputs. On numerous downstream tasks, large GPT models can achieve surprising performance, which even exceeds smaller models with supervised finetuning. However, although ICL has achieved great performance, its working mechanism is still an open question to be investigated.
In this paper, we explain in-context learning as a process of meta-optimization and analyze connections between GPT-based in-context learning and finetuning. Concentrating on the attention modules, we figure out that the Transformer attention has a dual form of gradient descent. On top of it, we propose a novel perspective to explain in-
4005
context learning: (1) a pretrained GPT serves as a meta-optimizer; (2) it produces meta-gradients according to the demonstration examples through forward computation; (3) the meta-gradients are applied to the original language model through attention to build an ICL model. As illustrated in Figure 1, in-context learning and explicit finetuning share a dual view of gradient descent, where ICL produces meta-gradients through forward computation, while finetuning computes gradients by back-propagation. Therefore, it is reasonable to understand in-context learning as implicit finetuning.
In order to provide empirical evidence to support our understanding, we conduct comprehensive experiments based on real tasks. On six classification tasks, we compare the model predictions, attention outputs, attention weights to query tokens, and attention weights to training tokens between in-context learning and finetuning. Experimental results validate that the behavior of in-context learning is similar to explicit finetuning from multiple perspectives. These results are strong evidence to prove the reasonability of our understanding of in-context learning as implicit finetuning.
Further, inspired by the dual form between Transformer attention and gradient descent, we design a momentum-based attention, which regards the attention values as meta-gradients and applies the momentum mechanism (Polyak, 1964; Sutskever et al., 2013) to them. Experiments on both language modeling and in-context learning show that our momentum-based attention consistently outperforms vanilla attention, which supports our understanding of meta-optimization again from another perspective. We note that beyond this preliminary attempt, our understanding may have more potential to enlighten model design, which is worth investigating in the future.
Our contributions are summarized as follows:
• We figure out a dual form between Transformer attention and gradient descent, and explain ICL as a process of meta-optimization.
• We analyze connections between in-context learning and explicit finetuning and propose to understand ICL as implicit finetuning.
• We provide several lines of empirical evidence to prove that ICL and explicit finetuning behave similarly from multiple perspectives.
• We design a momentum-based attention and validate its effectiveness, which supports our
understanding of meta-optimization again and shows the potential of our understanding to enlighten future model design.

heading: 2 Background

heading: 2.1 In-Context Learning with GPT
text: In this paper, we focus on ICL for classification tasks using GPT (Brown et al., 2020). A GPT model is stacked with L identical Transformer (Vaswani et al., 2017) decoder layers where each layer consists of an attention module and a feed-forward network. For a classification task, given a query input text x and a candidate answer set Y = {y1, y2, . . . , ym}, we need to predict a label ŷ conditional on n demonstration examples C = {(x′1, y′1), (x′2, y′2), . . . , (x′n, y′n)}, where (x′i, y ′ i) is an input-label pair different from the query one. Formally, given a GPT model M, we first compute the probability of each answer yj :
PM(yj | C, x). (1)
Since the label space is restricted for classification, we predict the final answer ŷ by selecting the answer with the highest probability from the candidate answer set Y :
ŷ = argmax yj
PM(yj | C, x). (2)
In practice, we usually use a pre-defined template to format the demonstrations and prepend them before the query input. Let T (·) be the function that formats an example, e.g.:
T (x, y) = Sentence: x. Sentiment: y. (3)
The contextual model input I is organized like
T (x′1, y′1) T (x′2, y′2) ... T (x′n, y′n) T (x, _). (4)
Feeding this contextual input into M, the probability of an answer yj is computed as
lj = M(I) · eyj , (5) PM(yj | C, x) = softmax(lj), (6)
where M(I) denotes the output hidden state at the last token position; eyj denotes the output word embedding of yj ; and lj is the logit corresponding to the j-th answer.

heading: 2.2 Dual Form Between Attention and Linear Layers Optimized by Gradient Descent
text: The idea in this paper to explain language models as meta-optimizers is inspired by Aizerman et al. (1964); Irie et al. (2022). They present that linear layers optimized by gradient descent have a dual form of linear attention. Let W0,∆W ∈ Rdout×din be the initialized parameter matrix and the update matrix, respectively, and x ∈ Rdin be the input representation. A linear layer optimized by gradient descent can be formulated as
F(x) = (W0 +∆W )x. (7)
In the back-propagation algorithm, ∆W is computed by accumulating the outer products of historic input representations x′Ti ∈ Rdin and the error signals ei ∈ Rdout of their corresponding outputs:
∆W = ∑
i
ei ⊗ x′i, (8)
where ei is derived from the historic output gradients by multiplying −γ, the negative learning rate. Combing Equation (7) and Equation (8), we can derive the dual form of linear layers optimized by gradient descent:
F(x) = (W0 +∆W )x =W0x+∆Wx
=W0x+ ∑
i
( ei ⊗ x′i ) x
=W0x+ ∑
i
ei ( x′Ti x )
=W0x+ LinearAttn ( E,X ′,x ) ,
(9)
where LinearAttn(V,K,q) denotes the linear attention operation, in which we regard the historic output error signals E as values, the historic inputs X ′ as keys, and the current input x as the query.

heading: 3 Understanding In-Context Learning (ICL) as Implicit Finetuning
text: We first qualitatively analyze the Transformer attention under a relaxed linear attention form to figure out a dual form between it and gradient descent. Then, we compare in-context learning with explicit finetuning to analyze connections between these two optimization forms. Based on these theoretical findings, we propose to understand in-context learning as implicit finetuning.

heading: 3.1 Understanding Transformer Attention as Meta-Optimization
text: Let x ∈ Rd be the input representation of a query token t, and q = WQx ∈ Rd ′ be the attention query vector. In the ICL setting, the attention result of a head is formulated as FICL(q) = Attn(V,K,q)
=WV [X ′;X] softmax
( (WK [X
′;X])T q√ d
) , (10)
where WQ,WK ,WV ∈ Rd ′×d are the projection matrices for computing the attention queries, keys, and values, respectively; √ d denotes the scaling factor; X denotes the input representations of query tokens before t; X ′ denotes the input representations of the demonstration tokens; and [X ′;X] denotes the matrix concatenation. For ease of qualitative analysis, we approximate the standard attention to relaxed linear attention by removing the softmax operation and the scaling factor:
FICL(q) ≈ WV [X ′;X] ( WK [X ′;X] )T q
= WV X (WKX) T q+WV X ′ (WKX ′ )T q
= F̃ICL(q). (11)
We define WZSL = WV X (WKX) T as the initialized parameters to be updated since WZSLq is the attention result in the zero-shot learning (ZSL) setting, where no demonstrations are given. Following the reverse direction of Equation (9), we derive a dual form of the Transformer attention: F̃ICL(q) = WZSLq+WV X ′ ( WKX
′)T q =WZSLq+ LinearAttn ( WV X ′,WKX ′,q ) =WZSLq+ ∑
i
WV x′i (( WKx′i )T q )
=WZSLq+ ∑
i
( (WV x′i)⊗ ( WKx′i )) q
=WZSLq+∆WICLq =(WZSL +∆WICL)q.
(12)
As shown in the above equations, the attention to the demonstration tokens is equivalent to parameter updates ∆WICL that take effect on WZSL. In addition, by analogy with E in Equation (9), we regard WV X ′ as meta-gradients, which are used to compute the update matrix ∆WICL.
In summary, we explain in-context learning as a process of meta-optimization: (1) a pretrained GPT model serves as a meta-optimizer; (2) it produces meta-gradients according to the demonstration examples through forward computation; (3) through attention, the meta-gradients are applied to the original language model to build an ICL model.

heading: 3.2 Comparing ICL with Finetuning
text: Based on the above understanding of in-context learning, we further compare the meta-optimization of in-context learning with the explicit optimization of finetuning to analyze connections between them. Considering that ICL directly takes effect on only the attention keys and values, we design a specific finetuning setting as the compared baseline, which also updates only the parameters for the key and value projection. Also in the relaxed linear attention form, the attention result of a finetuned head is formulated as
F̃FT(q) = (WV +∆WV )XXT (WK +∆WK)Tq = (WZSL +∆WFT)q, (13)
where ∆WK and ∆WV denote the parameter updates to WK and WV , respectively, which are acquired by back-propagation from task-specific training objectives; and ∆WFT is the updates to WZSL introduced by finetuning.
For a more fair comparison with in-context learning, we further restrict the finetuning setting as follows: (1) we specify the training examples as the demonstration examples for in-context learning; (2) we train each example for only one step in the same order as demonstrated for in-context learning; (3) we format each training example with the same template used for ICL T (x′i, y′i) and use the causal language modeling objective for finetuning.
Comparing in-context learning and this finetuning setting, we find that ICL has many properties in common with finetuning. We organize these common properties into the following four aspects.
Both Perform Gradient Descent Comparing Equation (12) and Equation (13), we find that both in-context learning and finetuning introduce updates (∆WICL v.s. ∆WFT) to WZSL, which drive from implicit and explicit gradient descent, respectively. The main difference is that ICL produces meta-gradients by forward computation while finetuning acquires real gradients by back-propagation.
Same Training Information The meta-gradients of ICL are produced according to the demonstration examples. The gradients of finetuning are also derived from the same training examples. That is to say, in-context learning and finetuning share the same source of training information.
Same Causal Order of Training Examples Incontext learning and our finetuning setting share the same causal order of training examples. ICL
uses decoder-only Transformers so the subsequent tokens in the demonstrations will not affect the preceding ones. For our finetuning setting, we use the same order of training examples and train only one epoch, so we can also guarantee that the subsequent examples have no effect on the preceding ones.
Both Aim at Attention Compared with zeroshot learning, the direct effect of in-context learning and our finetuning are both restricted to the computation of attention keys and values. For ICL, the model parameters are unchanged and it encodes demonstration information into additional keys and values to change the attention behavior. For finetuning, due to our restriction, the training information can be introduced to only the projection matrices for attention keys and values as well.
Considering the above common properties between in-context learning and finetuning, we show that it is reasonable to understand in-context learning as implicit finetuning. In the rest of this paper, we compare ICL and explicit finetuning empirically from multiple perspectives to provide quantitative results to support this understanding.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: We analyze two off-the-shelf pretrained GPT models with 1.3 billion and 2.7 billion model parameters, respectively, which are released by fairseq1. In the rest of this paper, we call them GPT 1.3B and GPT 2.7B for short. All experiments are conducted on NVIDIA V100 GPUs with 32 GB memory.
For each task, we use the same template to format examples for zero-shot learning (ZSL), finetuning (FT), and in-context learning (ICL). Details of the templates used for each task are provided in Appendix A. The answer prediction processes for ZSL and finetuning are the same with ICL as described in Section 2.1, except that they do not have demonstration examples.
For in-context learning, we fix the max number of demonstration examples to 32 and tune the random seed for each task to find a set of demonstration examples that achieves the best validation performance. For explicit finetuning, we use the same demonstration examples for in-context learning as the training examples and use SGD as the optimizer. For a fair comparison, we fine-tune the
1https://github.com/facebookresearch/fairseq
model for only one epoch and the training examples are provided in the same order as demonstrated for in-context learning. We tune the learning rate for finetuning and select the one that achieves the best validation performance. Details of the search range and selected value for the random seeds and learning rates are shown in Appendix B.

heading: 4.2 Evaluation Datasets
text: We compare in-context learning and finetuning based on six datasets spanning three sorts of classification tasks. SST2 (Socher et al., 2013), SST5 (Socher et al., 2013), MR (Pang and Lee, 2005) and Subj (Pang and Lee, 2004) are four datasets for sentiment classification; AGNews (Zhang et al., 2015) is a topic classification dataset; and CB (De Marneffe et al., 2019) is used for natural language inference. Statistics of the number of validation examples and label types are summarized in Table 1.
For reference, we present the validation accuracy in the ZSL, finetuning, and ICL settings on six classification datasets in Table 1. Compared with ZSL, ICL and finetuning both achieve considerable improvements, which means the optimizations they make are both helpful to these downstream tasks.

heading: 4.3 ICL Covers Most of Correct Predictions of Finetuning
text: We compute a recall to finetuning prediction (Rec2FTP) to measure ICL can cover how much behavior of finetuning from the perspective of the model prediction. We first count NFT>ZSL, the number of query examples that finetuning can predict correctly but ZSL cannot. Then, among these examples, we count N(FT>ZSL)∧(ICL>ZSL), the number that ICL can also predict correctly. Finally, we compute the Rec2FTP score as
N(FT>ZSL)∧(ICL>ZSL) NFT>ZSL
. A higher Rec2FTP score suggests that ICL covers more correct behavior of finetuning from the perspective of the model prediction.
We show the Rec2FTP scores for two GPT models on six datasets in Table 2. As shown in the table, on average, ICL can correctly predict more than 85% of the examples that finetuning can correct from ZSL. These results indicate that from the perspective of model prediction, ICL can cover most of the correct behavior of finetuning.

heading: 4.4 ICL Tends to Change Attention Outputs in the Same Direction as Finetuning
text: From the perspective of representation, we compute a similarity of the attention output updates (SimAOU) to measure the similarity between the updates that ICL and finetuning make. For a query example, let h(l)X denote the normalized output rep-
resentation of the last token at the l-th attention layer in setting X. The updates of ICL and finetuning compared with ZSL are h(l)ICL − h (l) ZSL and h (l) FT − h (l) ZSL, respectively. We compute the cosine between these two updates to get SimAOU (∆FT) at the l-th layer. A higher SimAOU (∆FT) means ICL is more inclined to update the attention output in the same direction as finetuning. For comparison, we also compute a baseline metric called SimAOU (Random ∆) that computes the similarity between ICL updates and randomly generated updates.
We present the SimAOU scores averaged across examples and layers for two GPT models on six datasets in Table 3. From the table, we find that SimAOU (Random ∆) is always around zero, while SimAOU (∆FT) remains much more positive. These results indicate that ICL updates are much more similar to finetuning updates than to random updates. From the perspective of representation, we prove that ICL tends to change the attention outputs in the same direction as finetuning.

heading: 4.5 ICL Is Inclined to Generate Similar Attention Weights to Finetuning
text: From the perspective of attention behavior, we compute a similarity of the attention map (SimAM) to measure the similarity of the attention map to query tokens for ICL and finetuning. For a query example, let m(l,h)X denote the attention weights before softmax of the last token at the h-th attention
head in the l-th attention layer in setting X. For ICL, we omit the attention to the demonstration tokens and only monitor the attention weights to the query tokens. First, before finetuning, we compute the cosine between m(l,h)ICL and m (l,h) ZSL and then average the similarity across attention heads to get SimAM (Before Finetuning) at each layer. Similarly, after finetuning, we compute the cosine between m(l,h)ICL and m(l,h)FT to get SimAM (After Finetuning). A higher SimAM (After Finetuning) over SimAM (Before Finetuning) indicates that the attention behavior of ICL is more similar to a finetuned model than a non-finetuned one.
Table 4 demonstrates the SimAM scores averaged across examples and layers for two GPT models on six datasets. We observe that compared with attention weights before finetuning, ICL is more inclined to generate similar attention weights to attention weights after finetuning. Again, from the perspective of attention behavior, we prove that ICL behaves similarly to finetuning.

heading: 4.6 ICL and Finetuning Tend to Pay Similar Attention to Training Tokens
text: Since we understand ICL as a process of metaoptimization, we also compare the attention to training tokens for ICL and finetuning with the Kendall rank correlation coefficient (Kendall, 1948). For a query example, let m(l)ICL denote the ICL attention weights to the demonstration tokens
of the last query token in the l-th attention layer, which is summed across attention heads. For finetuning, we first record all the attention queries Q′(l,h) ∈ Rd′×N of the training tokens, and then use the inner product between them and the attention query q(l,h) ∈ Rd′ of the last token in the query example as the finetuning attention weights to the training tokens: m(l)FT = ∑ hQ ′(l,h)Tq(l,h), which is also summed across attention heads. The Kendall coefficient between m(l)ICL and m (l) FT is computed as Kendall (ICL, FT) = Pc−PdN(N−1)/2 , where N denotes the number of training tokens, Pc denotes the number of concordant pairs, and Pd denotes the number of discordant pairs. A higher Kendall coefficient means that the orders of attention weights to training tokens of ICL and finetuning are more similar. For comparison, we also compute the Kendall coefficient between m(l)ICL and randomly generated attention weights m(l)Random, which we call Kendall (ICL, Random).
Table 5 shows the Kendall correlation coefficients averaged across examples and layers for two GPT models on six datasets. We find that Kendall (ICL, Random) is always near zero, while Kendall (ICL, FT) always maintains a distinctly positive value. These results suggest that ICL and finetuning tend to pay similar attention to training tokens.

heading: 5 Momentum-Based Attention Inspired by Dual Form of Transformer Attention
text: We have figured out the dual form between Transformer attention and gradient descent. As illustrated in Figure 2, inspired by this dual view, we investigate whether we can utilize momentum (Polyak, 1964; Sutskever et al., 2013), a widely used technique for optimization algorithms, to improve Transformer attention.
Gradient descent with momentum averages gra-
dients among timestamps:
Θt = Θt−1 − γ t−1∑
i=1
ηt−i∇fΘi , (14)
where γ is the learning rate and η is a scalar between 0 and 1. As stated in Section 3.1, the attention values serve as meta-gradients. By analogy with gradient descent with momentum, we try to use Exponential Moving Average (EMA; Hunter 1986) to average the attention values to build the momentum-based attention:
MoAttn(V,K,qt) = Attn(V,K,qt) + EMA(V )
= V softmax( KTqt√
d ) +
t−1∑
i=1
ηt−ivi,
where vi is the i-th attention value vector. The momentum of attention value vectors explicitly strengthens the recency bias of attention, which has been shown helpful for language modeling (Press et al., 2022). Therefore, we assume that introducing momentum into attention will contribute to faster convergence and better performance.
Experiments on Language Modeling First, we evaluate the effect of momentum-based attention on language modeling. We train two GPT models with 350M parameters from scratch, where one is the vanilla Transformer, and another applies momentum to attention. More training details are provided in Appendix C. We evaluate the perplexity
of these two models on the training set and three validation sets with input lengths of 256, 512, and 1024, respectively. The results are shown in Table 6. On all of the validation sets, applying momentum to attention introduces a consistent perplexity improvement compared with the vanilla Transformer.
Experiments on In-Context Learning We also evaluate the in-context learning ability of the above language models to verify the effectiveness of momentum-based attention on downstream tasks. We consider six datasets for sentiment analysis (SST5 (Socher et al., 2013), IMDB (Maas et al., 2011), and MR (Pang and Lee, 2005)), natural language inference (CB (De Marneffe et al., 2019)), and multi-choice selection (ARC-E (Clark et al., 2018) and PIQA (Bisk et al., 2020)). For all of these datasets, we use up to 32 examples as demonstrations. As shown in Table 7, compared with vanilla Transformer, using momentum-based attention achieves consistently higher accuracy on all of these datasets.
The performance improvements on both language modeling and in-context learning prove our deduction that introducing momentum will improve Transformer attention. From another perspective, these results further support our understanding of Transformer attention as meta-optimization.

heading: 6 Related Work
text: Recently, some pieces of work have attempted to understand the inference mechanism of in-context learning. Xie et al. (2022) explain in-context learning as implicit Bayesian inference. They state that in-context learning emerges when language models can infer the shared latent concept among the demonstration examples, which is learned during
pretraining. On another aspect, Olsson et al. (2022) focus on specific modules in Transformers. They find some induction heads in Transformers that refer to abstract patterns in previous sequences to help predict the next token. They indicate that the induction heads drive the ability of in-context learning. Different from them, we concentrate on the learning algorithm of ICL and explain it as a process of meta-optimization.
Some other work also studies the learning algorithm of ICL. As a case study, Garg et al. (2022) show that Transformers can be trained to in-context learn a class of linear functions and the performance is comparable to the least squares estimator. Based on linear regression, Akyürek et al. (2022) prove that they can construct parameters of Transformers to implement gradient-descent-based learning algorithms. Further, they show that models trained with an in-context learning objective tend to match the behavior of models computed by explicit learning algorithms. Also based on regression tasks, von Oswald et al. (2022) show that linear attention-only Transformers with constructed parameters that implement gradient descent and models learned by an in-context learning objective are highly related. Compared with them, we are the first ones to explain in-context learning in real scenarios. To be specific, (1) we analyze in-context learning for off-the-shelf GPT models, instead of models trained from scratch by an ICL objective; (2) our experiments are based on real NLP tasks, instead of toy ones like linear regression.

heading: 7 Conclusion
text: In this paper, we aim to explain the working mechanism of GPT-based ICL. Theoretically, we figure
out a dual form between Transformer attention and gradient descent, and propose to understand ICL as a process of meta-optimization. Further, we analyze connections between ICL and explicit finetuning and show the reasonability to regard ICL as implicit finetuning. Empirically, we comprehensively compare ICL and finetuning based on six real NLP tasks. The results prove that ICL behaves similarly to explicit finetuning from multiple perspectives. Further, inspired by our understanding of meta-optimization, we design a momentum-based attention that achieves consistent performance improvements over vanilla attention. We believe our understanding will have more potential to enlighten ICL applications and model design in the future.
Limitations
Although the ability of in-context learning has been found for different architectures (e.g., Transformer and LSTM), we consider only Transformer-based in-context learning in this paper because Transformer is the current mainstream architecture of NLP. However, as for in-context learning itself, figuring out how it works for other architectures is also a meaningful problem, which we encourage to study in the future.
As for the dual form we point out between Transformer attention and gradient descent, we consider a relaxed form of linear attention for qualitative analysis. Although the experimental results support our understanding well, the mechanism of standard Transformer attention without approximation may be more complex and should be studied more clearly in the future.
As for empirical experiments, our analysis needs to record a large number of intermediate results (e.g., attention output representations, and attention weights to query tokens and demonstration tokens) for thousands of validation examples. Considering the storage space and computational cost of analysis, we only analyze GPT models with up to 2.7B parameters and leave larger models such as GPT 13B for future work. In addition, for the clarity of the problem definition and the convenience of experiments, our analysis is based on only classification tasks. Although classification is a representative application of in-context learning, other tasks like multiple choice and open-ended generation are not considered in this paper and could be investigated in the future.

heading: Acknowledgement
text: Damai Dai and Zhifang Sui are supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.

heading: A Templates for In-Context Learning
text: We demonstrate the templates used to format examples and the candidate answer sets for six classification datasets used in our experiments in Table 8.

heading: B Hyper-Parameters for In-Context Learning and Finetuning
text: We perform grid search to find the best random seed for ICL and the best learning rate for finetuning. The search range for all the datasets is the same. For random seeds, we search in {1, 2, 3, 4, 5, 6, 7}. For learning rates, the search base values are {1, 2, 3, 4, 5, 6, 7, 8, 9} and we scale them to 0.1, 0.01, 0.001, and 0.0001 times, i.e., we have 9×4 = 36 values to search. As an exception, for GPT 1.3B finetuned on SST5, we perform a more fine-grained search and finally set its learning rate to 0.00016 since the finetuned model cannot outperform the zero-shot learning with the above 36 learning rates.
In Table 9, we present the details of the selected random seeds and learning rates for two GPT models on six classification datasets.

heading: C Hyper-Parameters for Training Language Models from Scratch
text: The hyper-parameters for training two language models from scratch are summarized in Table 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations"
A2. Did you discuss any potential risks of your work? Not applicable. This paper focuses on the interpretability of models, so there are no potential risks for our work.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section "Abstract" and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sections 4 & 5
3 B1. Did you cite the creators of artifacts you used? Section 4.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Fairseq is a widely-acceptable open-source library for NLP tasks, and we do not violate its license.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Our use of fairseq is consistent with their intended use.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use publicly available datasets that had been checked when they were created.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. We provide the website of fairseq, and the mentioned information is included in it.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.2
C 3 Did you run computational experiments? Sections 4 & 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Sections 4 & 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Sections 4 & 5, Appendices B & C
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Our analytical experiments do not focus on performance, but analysis of off-the-shelf models
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We did not use existing packages for preprocessing, normalization, or evaluation
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We believe our understanding will have more potential to enlighten ICL applications and model design in the future. Limitations
Although the ability of in-context learning has been found for different architectures (e. g. , Transformer and LSTM), we consider only Transformer-based in-context learning in this paper because Transformer is the current mainstream architecture of NLP. However, as for in-context learning itself, figuring out how it works for other architectures is also a meaningful problem, which we encourage to study in the future. As for the dual form we point out between Transformer attention and gradient descent, we consider a relaxed form of linear attention for qualitative analysis. Although the experimental results support our understanding well, the mechanism of standard Transformer attention without approximation may be more complex and should be studied more clearly in the future. As for empirical experiments, our analysis needs to record a large number of intermediate results (e. g. , attention output representations, and attention weights to query tokens and demonstration tokens) for thousands of validation examples. Considering the storage space and computational cost of analysis, we only analyze GPT models with up to 2. 7B parameters and leave larger models such as GPT 13B for future work. In addition, for the clarity of the problem definition and the convenience of experiments, our analysis is based on only classification tasks. Although classification is a representative application of in-context learning, other tasks like multiple choice and open-ended generation are not considered in this paper and could be investigated in the future