What In-Context Learning “Learns” In-Context: Disentangling Task Recognition and Task Learning
Year: 2023

Abstract:
Large language models (LLMs) exploit incontext learning (ICL) to solve tasks with only a few demonstrations, but its mechanisms are not yet well-understood. Some works suggest that LLMs only recall already learned concepts from pre-training, while others hint that ICL performs implicit learning over demonstrations. We characterize two ways through which ICL leverages demonstrations. Task recognition (TR) captures the extent to which LLMs can recognize a task through demonstrations – even without ground-truth labels – and apply their pre-trained priors, whereas task learning (TL) is the ability to capture new input-label mappings unseen in pre-training. Using a wide range of classification datasets and three LLM families (GPT-3, LLaMA and OPT), we design controlled experiments to disentangle the roles of TR and TL in ICL. We show that (1) models can achieve non-trivial performance with only TR, and TR does not further improve with larger models or more demonstrations; (2) LLMs acquire TL as the model scales, and TL’s performance consistently improves with more demonstrations in context. Our findings unravel two different forces behind ICL and we advocate for discriminating them in future ICL research due to their distinct nature.1

heading: Future Work
text: Our findings unravel two different forces behind ICL and we advocate for discriminating them in future ICL research due to their distinct nature. 1
1 Introduction
Large language models (LLMs) have demonstrated the ability to perform in-context learning (ICL), i. e. , “learning” to perform a task purely from examples in the context without any parameter updates (Brown et al. , 2020). This powerful and flexible phenomenon enables LLMs to be used as general-purpose models that can perform any task with a small set of labeled examples. However, there is still no consensus on how incontext learning works. Some previous work hy-
1Our code is publicly available at https://github. com/ princeton-nlp/WhatICLLearns. pothesizes that during pre-training, LLMs implicitly learn tasks required for downstream applications, and the in-context demonstrations merely provide information that allow the model to recognize which task is required (Xie et al. , 2022). Min et al. (2022) show empirical evidence of this hypothesis by demonstrating that ICL performance is insensitive to the usage of ground-truth labels. On the other hand, Akyürek et al. (2023); von Oswald et al. (2022) construct theories that Transformer-based models may perform implicit gradient descent to update an “inner-model”, and Dai et al. (2023) demonstrate similarities between in-context learning and explicit fine-tuning through a series of metrics on real-world datasets. Such hypotheses assume the correct input-output mappings are important and ICL actually performs implicit learning over demonstrations. In this paper, we disentangle ICL into task
8298
recognition (TR), which recognizes the task from demonstrations and applies LLMs’ pre-trained priors, and task learning (TL), which learns a new input-label mapping from demonstrations. In common ICL scenarios where ground-truth labels are provided, TR and TL take effect simultaneously. We propose two settings to tease them apart: 1) RANDOM, where the labels are uniformly sampled from the label space (Min et al. , 2022), in order to restrict LLMs to only apply TR; 2) ABSTRACT, where the labels are replaced with abstract symbols (e. g. , numbers or letters) that never co-occurred with the inputs in pre-training. We focus on how the two abilities in ICL evolve with two factors – model sizes and numbers of demonstrations, which have been neglected in related literature. Through extensive experiments with a series of classification datasets on GPT-3 (Brown et al. , 2020), LLaMA (Touvron et al. , 2023), and OPT (Zhang et al. , 2022), we find:
• The gap between GOLD and RANDOM is small with smaller models, corroborating with Min et al. (2022). However, with larger models and more examples, the gap becomes larger. This suggests TR plays a significant role in ICL, but it does not scale with increasing parameters or examples. • LLMs also perform TL, which emerges with larger models and more demonstrations. With the largest model and more than 16 examples, ABSTRACT outperforms RANDOM, pointing to a paradigm shift in in-context learning at scale. Together, our findings provide a better way to understand ICL behaviors. 2
2 Task Recognition and Task Learning
An LLM (parameterized by θ) performs ICL by conditioning on the input-label pair demonstrations Ddemo = (x1, y1, x2, y2,. , xK , yK) and the test input xtest to predict the label ytest ∼ pθ(y | Ddemo , xtest), where the demonstrations elicit a mapping f : X → Y, x ∈ X , y ∈ Y. We delineate two ways an LLM can leverage in-context demonstrations: task recognition and task learning. Task recognition (TR) represents models’ ability to recognize the mapping f purely by observing the input distribution {xi}Ki=1 and the label distribution {yi}Ki=1, without the provided (xi, yi) pairs. 2We discuss the differences between our work and Min et al. (2022); Yoo et al. (2022) in Section 5, detailing how our findings deviate and converge with existing results. The LLM then applies its pre-trained priors to the recognized f. Formally, when only TR is enabled,
pθ(y | xtest, {xi, yi}Ki=1) =pθ(y | xtest, {xi}Ki=1, {yi}Ki=1),
which suggests TR does not rely on the pair information. For example, an input distribution of movie reviews and a label distribution of “The sentiment is positive/negative” can be easily recognized as a sentiment classification task due to their prevalence during pre-training, and LLMs can make reasonable predictions without explicitly “learning” the task via ground-truth demonstrations. This leads to observations that the model can still perform well even when we provide wrong input-label mappings, e. g. , “The movie is great. The sentiment is negative” (Min et al. , 2022). Task learning (TL), on the other hand, characterizes how the model learns a new mapping from the input-label pairs through demonstrations. Unlike TR, TL allows models to learn novel mappings and thus correct input-label pairs will be crucial. We posit that the two mechanisms occur under separate conditions, as recognizing an already learned task is easier than learning a new mapping. Models are able to perform TR at a small scale, but this ability does not drastically improve with increasing model sizes and demonstrations; on the other hand, TL improves significantly when model sizes and numbers of demonstrations increase. To show the above phenomenon, we disentangle TR and TL through label space manipulation, including three different setups (examples in Figure 1):
• GOLD: the standard ICL setting where we use natural prompts and gold input-label pairs. This setup reflects both TR and TL abilities. • RANDOM: similar to Min et al. (2022), we use the same natural prompts as GOLD and sample demonstration labels uniformly at random from the label space. This setup reflects TR only. • ABSTRACT: we use minimal prompts (which provide no task information) and characters with no clear semantic meanings (e. g. numbers, letters, and random symbols) as the label for each class. We found that even abstract labels may have biases in pre-training, e. g. , “0” is biased towards negative. Hence, for each prompt x1, y1,. , xK , yK , we randomly sample a 1-1 mapping φ : Y → Y∗ to avoid any bias, and no task-specific information is leaked in either the prompt template or the label
OPTLLaMAGPT-3
space. To evaluate the model’s ABSTRACT performance, we measure its accuracy using φ(ytest) as target labels. Since these input-label mappings are never seen in pre-training, it reflects the TL ability. In the following sections, we conduct comprehensive experiments with the above three different settings under two axes – model sizes and numbers of demonstrations – and show how TR and TL manifest under different conditions. 3 Experimental Setup
3. 1 Datasets We experiment on 16 classification datasets across 4 type of tasks: sentiment analysis, toxicity detection, natural language inference/paraphrase detection, and topic/stance classification. All datasets and references are in Appendix A. Our dataset selection largely follows Min et al. (2022), but we exclude multi-choice datasets since it is difficult to apply our ABSTRACT experiments on them. 3. 2 Models We use three state-of-the-art LLM families: GPT3 (Brown et al. , 2020), LLaMA (Touvron et al. , 2023), and OPT (Zhang et al. , 2022). We use
GPT-3 ada (350M), babbage (1. 3B), curie (6. 7B), and davinci (175B) via the OpenAI API. For OPT, we use checkpoints from the Transformers library (Wolf et al. , 2020), with model sizes of 350M, 2. 7B, 6. 7B, 13B, 30B, and 66B parameters. For LLaMA, we use model sizes of 7B, 13B, 33B, and 65B parameters. 3
3. 3 Task Setup
We adopt the sample-based evaluation protocol: for each test example, we sample a different set of demonstrations from the training set. We manually design 3 prompt templates for each type of classification tasks in a similar style to the prompts from Min et al. (2022). We report the mean by averaging across datasets and prompts, and standard variation across different prompts for each datapoint. For GPT-3, we sample 150 examples for each dataset. We use fewer examples due to budget constraints, and GPT-3 presents lower variance than other model families. For OPT and LLaMA, we sample 1,350 examples for all datasets. 3For GPT-3, we use the non-instruction legacy models for fair comparison to OPT and LLaMA models. We did not run experiments on the largest OPT-175B model due to computational constraints. We design two kinds of prompts: natural language prompts (Table 1), which are similar to the manual prompts in Min et al. (2022), and minimal prompts (Table 3), which remove any natural language instructions for the task. For ABSTRACT, we tested three types of label choices: numbers (0,. , N − 1, where N is the number of classes), letters (N letters from A, B, C,. ), and symbols (first N symbols of “@”, “#”, “$”, ”%”, “*”, and “∧”). For each test example, we randomly sample a new mapping between labels and abstract characters. We report the number abstract labels in all the main results and compare the three forms in §4. 2. 4 Results
Figure 2 shows our main results with GPT-3, LLaMA, and OPT with our 3 settings: GOLD, RANDOM, and ABSTRACT. Below we summarize the trends of TR and TL across different conditions. 4. 1 Main Results
Summary of overall trends. We first verify that GOLD consistently performs the best across model families and number of demonstrations, which is expected given that the GOLD setting provides the model with all information. Overall, the RANDOM curves do not increase with either model sizes or number of demonstrations, remaining largely flat; considering the scenario with small model sizes and few examples (K = 8), there is an insignificant gap between RANDOM and GOLD. Meanwhile, the ABSTRACT curves demonstrate an increasingly steep slope as the model sizes and the number of demonstrations grow; with small models or small K, ABSTRACT mostly underperforms RANDOM, whereas ABSTRACT with largest models and K = 32 performs well above RANDOM (and may even be competitive with GOLD). We note that the OPT curves demonstrate significant variance, which we hypothesize to be a result of the models potentially being under-trained. We elaborate the takeaways on TR and TL below. Task recognition is a broader capability across scales. For all model families, the RANDOM setting shows similar performance at all sizes and numbers of demonstrations. Moreover, TR performance is significantly stronger than the random baseline, even with small models and few examples. For instance, even the smallest 350M parameter models are able to recognize the task using
just 8 examples, drawing around 10 points of average performance lead against the random baseline for GPT-3 ada and 5 points for OPT-350M. This shows that task recognition from in-context examples does not drastically scale with model sizes or numbers of examples. Task learning is enabled with scale. We observe that TL is dependent on model sizes: smaller models perform roughly the same across all numbers of demonstrations (see Figure 6). On the other hand, larger models can utilize the provided mapping information and perform TL, as ABSTRACT (TL) performance increases drastically with larger sizes (first row of Figure 2). When using a larger model, the results also improve as the number of demonstration increases (second row of Figure 2). With only 16 examples, OPT-66B and davinci are able to match the performance of GOLD while using a new label mapping. While LLaMA-65B’s ABSTRACT is not as competitive as its GOLD, the trend of improving ABSTRACT performance with larger size s or larger K is clear. This suggests that TL is only enabled by scales and further improves with more demonstrations. 4. 2 Further Analysis
The trends for task learning generalize across different types of abstract labels. In Figure 3, we show ABSTRACT results with number, letter, and symbol labels respectively. We observe that all three versions show a similar trend and coincide with our main results. Numbers and letters perform consistently better than symbols. This may be because letters and numbers appear more frequently in the pre-training corpus, and therefore make for a more "natural" label space. Task difficulty affects the trends. We notice that ABSTRACT scales better with sizes and examples when the task is simpler. In Figure 4 we compare two types of tasks: sentiment analysis and natural language inference (NLI). Since NLI is more difficult, we observe that it produces a flatter ABSTRACT curve, suggesting that the model relies more on the natural prompts and pre-training priors to solve those tasks. We demonstrate the full task-type breakdown results in §C. 5 Related Work
Many works have attempted to deepen empirical or theoretical understanding of ICL since its emergence in Brown et al. (2020). For instance, Xie et al. (2022) present a theoretical framework where latent “concepts" parameterize each document in pre-training. They posit that all concepts have been learned in pre-training; thus, ICL is the result of implicit Bayesian inference, where the LM uses incontext demonstrations as evidence to identify the correct concept. Min et al. (2022) present empirical evidence for this framework by showing that only limited information, rather than true input-label mappings, is needed to perform ICL. Other works investigate the impact of the pretraining corpus on ICL. Chan et al. (2022) identify properties of the pre-training distribution that enable ICL behavior, including burstiness, label multiplicity, and a long-tailed class distribution – all of which are satisfied by natural language. Razeghi et al. (2022) show that the frequencies of terms in the pre-training corpora is positively correlated with model performance. Kirsch et al. (2022) show that both a rich training distribution and a sufficiently large model are critical to the development of in-context learning abilities. More recently, several works have explored theoretical frameworks in which ICL can be seen as im-
plicit gradient descent, treating a forward pass over the in-context demonstrations as an “update” to an implicit internal model. (Akyürek et al. , 2023; von Oswald et al. , 2022; Dai et al. , 2023). For mechanistic perspectives on ICL, Olsson et al. (2022) and Bansal et al. (2022) identify induction heads (subnetworks that perform in-context pattern recognition) in small and large models, respectively