What Do Language Models Learn in Context? The Structured Task Hypothesis (2024)

Abstract: Large language models (LLMs) exhibit an intriguing ability to learn a novel task from incontext examples presented in a demonstration, termed in-context learning (ICL). Understandably, a swath of research has been dedicated to uncovering the theories underpinning ICL. One popular hypothesis explains ICL by task selection. LLMs identify the task based on the demonstration and generalize it to the prompt. Another popular hypothesis is that ICL is a form of meta-learning, i.e., the models learn a learning algorithm at pre-training time and apply it to the demonstration. Finally, a third hypothesis argues that LLMs use the demonstration to select a composition of tasks learned during pre-training to perform ICL. In this paper, we empirically explore these three hypotheses that explain LLMs’ ability to learn in context with a suite of experiments derived from common text classification tasks. We invalidate the first two hypotheses with counterexamples and provide evidence in support of the last hypothesis. Our results suggest an LLM could learn a novel task in context via composing tasks learned during pre-training. https://github.com/eth-lre/LLM_ ICL

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12365–12379 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/eth-lre/LLM_ ICL

heading: 1 Introduction
text: In-context learning (ICL) is a learning paradigm where a pre-trained large language model (LLM) learns to perform a certain task by extrapolating beyond a demonstration of the task in the form of example prompt–response pairs, given to the model as input. In-context learning does not require an update to the model’s parameters (Radford et al., 2019; Brown et al., 2020). Conditioned on the demonstration, the LLM is then tasked with generating responses to additional related prompts. Pre-trained large language models have exhibited an impressive ability to learn in context across various domains, e.g., code generation (Chen et al., 2021), education (Kasneci et al., 2023), and
* Equal contribution
medicine (Thirunavukarasu et al., 2023). However, there is still no consensus on when or how ICL works. We taxonomize existing candidate theories into three competing hypotheses (Fig. 1), which we summarize below.
Hypothesis 1 (Informal; Task Selection). During pre-training, an LLM learns a set of tasks T . At inference time, the LLM identifies the task τ ∈ T given the user-provided demonstration of the task and generalizes to the prompt.
Under Hypothesis 1, the demonstration merely allows the model to recognize a task, and no actual learning takes place. Min et al. (2022) offers empirical support for Hypothesis 1; they show that randomly shuffling the responses in the demonstration hardly has any effect on ICL performance, suggesting the demonstration of the task only serves to enable the LLM to look up a task. In other words, Hypothesis 1 asserts that no learning takes place during ICL. Some authors (Xie et al., 2022; Wang et al., 2023; Wies et al., 2023) have also argued for Hypothesis 1 from a theoretical angle, contending that if an LLM is pre-trained on a corpus that is generated from a mixture model over tasks, it will be able to infer the task that generated the demonstrations to be able to generalize to a prompt not in the demonstration.
The next hypothesis revolves around metalearning (Schmidhuber, 1987; Thrun and Pratt, 1998; Vilalta and Drissi, 2001; Finn et al., 2017).
Hypothesis 2 (Informal; Meta-Learning). During pre-training, an LLM learns certain learning algorithms. During ICL, the LLM learns a task τ directly from the demonstration using one of the
12365
learned learning algorithms.
Hypothesis 2 suggests that the pre-training stage prepares the parameters in an LLM in such a way that various learning algorithms, e.g., gradient descent and least squares regression, can be implicitly deployed during ICL (Von Oswald et al., 2023; Akyürek et al., 2023; Dai et al., 2023) to learn a new task from a demonstration. However, the setting assumed in the above-cited theoretical development is usually over-simplified, e.g., the assumption that the attention mechanism is linear. Moreover, the empirical evidence for Hypothesis 2 is mostly derived from experiments on small transformers trained from scratch on synthetic data (Garg et al., 2022; Raventos et al., 2023; Akyürek et al., 2024).
The final hypothesis may be viewed as a mixture of Hypothesis 1 and Hypothesis 2.
Hypothesis 3 (Informal; Structured Task Selection). During pre-training, an LLM learns a set of tasks T . At inference time, the LLM uses the demonstration to compose a sequence of learned tasks τ1, τ2, . . . ∈ T and uses this composition for prediction. The composition itself may result in a novel task not seen during pre-training.
Hypothesis 3 extends Hypothesis 1 by allowing ICL to not only index into the pre-learned task set T , but also compose tasks to obtain a novel task τ = τ1 ◦τ2 ◦· · · /∈ T .1 Hahn and Goyal (2023) lay the theoretical groundwork for Hypothesis 3; under their framework, they argue that the compositional structure of linguistic pre-training data gives rise to the task composition ability of ICL.
In this paper, we conduct a comprehensive battery of experiments to examine the above-stated three hypotheses. Given a demonstration of a task, i.e., a sequence of prompt–response pairs from which an LLM can successfully learn the task in context, we create a response-altered (RA) task by altering the responses such that the new task is unlikely to have occurred during pre-training. Our results confirm that LLMs can learn such an RA task in context, which rejects Hypothesis 1. We then create prompt-altered (PA) tasks, which instead alter the prompts. If Hypothesis 2 is true, this modification should not affect the performance of ICL. However, we find that the LLMs yield a substantially worse performance on PA tasks than on RA tasks, which contradicts Hypothesis 2. Lastly, in support of Hypothesis 3, we identify a sequence of simple tasks that the LLMs can compose
1The definition of task composition ◦ will be given in § 2.2.
to obtain unobserved RA tasks. It offers a possible explanation for ICL’s ability to perform novel tasks.

heading: 2 Preliminaries

heading: 2.1 Language Models
text: A language model p is a distribution over Σ∗ where Σ is an alphabet. The elements of Σ are tokens. A string w = w1 · · ·wN of length N is a finite sequence of tokens wn ∈ Σ. Most modern LLMs are defined in an autoregressive manner, i.e.,
p(w) = p(EOS | w) N∏
n=1
p(wn | w<n), (1)
and each local conditional distribution p(· | w<n) is defined over Σ def= Σ∪{EOS}. We denote w<n def= w1 · · ·wn−1 and w<1 def= ε. Note that not all models defined as in (1) are distributions over Σ∗. However, in the context of our paper, we assume p is.

heading: 2.2 A Restatement of the Hypotheses
text: In this section, we offer a more formal version of each of the three hypotheses discussed in § 1. Note that our treatment is decidedly not a formalization. Nevertheless, we do find it useful to build up some level of formal notation to discuss the three hypotheses more precisely; we save a true formalization for future work.
We start with a concrete definition of a task. In this paper, a task τ ∈ T is taken to be a pair of language models ⟨πτ , ρτ ⟩ where both πτ and ρτ are distributions over Σ∗. We interpret πτ as a distribution over prompts for task τ and ρτ as a distribution over responses conditioned on a prompt. Let T be a countable set of tasks. A demonstration of a task τ ∈ T is an interwoven sequence of prompt–response pairs. We denote a demonstration of length L as d = p1r1♮ · · · ♮pLrL, where ♮ ∈ Σ is a distinguished delimiter. We assume pℓ ∼ πτ (·) and rℓ ∼ ρτ (· | pℓ) for 1 ≤ ℓ ≤ L. With T ⊂ T , we denote the set of tasks observed at the pre-training time, i.e., those tasks where there exist demonstrations p1r1♮ · · · ♮pLrL of the task in the pre-training data.
At inference time, given a demonstration d = p1r1 · · ·pLrL, we say that a language model p has (approximately) learned a task if
p(r | d♮p) ≈ ∑
τ∈T ρτ (r | p)p(τ | d♮p), (2)
and p(τ | d♮p) is sufficiently low entropy for large L. In words, (2) says that the task to be performed
by the in-context learning may fruitfully be viewed as a latent variable (Xie et al., 2022). And, moreover, when the task-selection distribution p(τ | d♮p) is low entropy for large enough L, i.e., when we observe a large enough demonstration, the sum is dominated by a single summand. This means the language model has succeeded at identifying the task with high probability. Xie et al. (2022) present a more detailed theoretical framework to explore this scenario in a more precise manner.
We now restate the hypotheses in our notation.
Hypothesis 1 (Task Selection). The task-selection distribution p(τ | d♮p) is only well-calibrated for tasks in T , i.e., the finite set of tasks observed at pre-training time.
Hypothesis 2 (Meta-Learning). The task-selection distribution p(τ | d♮p) generalizes to some tasks in T \ T , i.e., tasks not observed at pre-training time.
To explain our third hypothesis, let T be a set of primitive tasks. We define the notion of task composition as follows. Given two tasks τ1 = ⟨ρτ1 , πτ1⟩ and τ2 = ⟨ρτ2 , πτ2⟩, we define
τ1 ◦ τ2 def= ⟨ρτ1◦τ2 , πτ2⟩, (3)
where we further define
ρτ1◦τ2(r | p) def =
∑
r̃∈Σ∗ ρτ1(r | r̃)ρτ2(r̃ | p), (4)
It is easy to see that ◦ is associative.2 Then, consider the semigroup3 (T ∗, ◦) where ◦ is as defined above. In other words, any composition of primitive tasks τ1 ◦ · · · ◦ τK ∈ T ∗ results in a new task. This semigroup structure encodes a primitive (non-hierarchical) notion of task composition. We then take T = T ∗. Finally, let T be the set of observed primitive tasks. Consider (T ∗, ◦), a
2See App. B. 3A semigroup is a set endowed with an associative operator,
under which the set is closed.
subsemigroup of (T ∗, ◦, ). Note that T ∗ is larger than T , as it includes tasks not observed during the pre-training time, but whose composite primitive tasks were. With this notation, we now present the third hypothesis.
Hypothesis 3 (Structured Task Selection). The task-selection distribution p(τ | d) is only wellcalibrated for (T ∗, ◦), i.e., compositions of primitive tasks observed at pre-training time.

heading: 3 Testing Hypothesis 1
text: According to Hypothesis 1, ICL selects the task it needs to perform from the demonstration. However, it is only able to select among the finite set of tasks observed at the pre-training time, denoted as T . It follows from Hypothesis 1, then, that if a novel task that has never been seen during pretraining is presented to a pre-trained model as a demonstration, ICL should not be able to perform it. We construct such a novel task as follows. We first create a string-to-string function g : Σ∗ → Σ∗. Note that such a function g is a special case of a response distribution that places probability 1 on a specific output string for every input string. Then, given a task ⟨πτ , ρτ ⟩, we obtain a new task τRA by applying g to the responses of τ , i.e.,
πτRA(p) def = πτ (p) (5a)
ρτRA(r | p) def = ρτ
( g−1(r) | p ) . (5b)
Note that the definition in (5) is no more than task composition, i.e., ⟨g, •⟩ ◦ ⟨ρ, π⟩ where • is a stand-in for an arbitrary prompt distribution. We define τ g def = ⟨g, •⟩ for the remainder of the paper, i.e., a task induced by the string-to-string function g with a stand-in prompt distribution; we also write τRA = τ g ◦ τ . The new task τRA is almost certainly not observed in the pre-training data, i.e., τRA /∈ T . We call τRA a response-altered task (RA) and the ICL setting with RA tasks τRA-ICL.
This setting resembles the semantically unrelated label ICL setting of Wei et al. (2023) and the abstract formalization of Pan et al. (2023).
We examine the following logical consequence of Hypothesis 1.
Prediction 1. If Hypothesis 1 is true, then an LLM’s performance in the τRA-ICL setting should be similar to random guessing.

heading: 3.1 Experimental Setup
text: Tasks. As shown in Tab. 3 (App. C.1), we select 3 commonly used text classification datasets for ICL: Customer Reviews (CR; Hu and Liu, 2004), Standford Sentiment Treebank with binary sentiments (SST-2; Socher et al., 2013), and AG News (Zhang et al., 2015); see App. D for the license details.
Experimental Setup. Each text classification dataset contains a set of pairs {(x(ℓ), y(ℓ))}Lℓ=1 where x(ℓ) ∈ Σ∗ is an input string and y(ℓ) ∈ Y is x(ℓ)’s classification label drawn from Y , a finite, task-dependent label set. To encode a classification problem as ICL, we map each element of Y to a string in Σ∗ by means of a function o : Y → Σ∗. Additionally, we convert the input x into a prompt p through a templating function t : Σ∗ → Σ∗. The exact templating function we use for each dataset is listed in Tab. 3. Then, we construct a delimiter-separated demonstration of size L: d = t(x(1))o(y(1))♮ · · · ♮t(x(L))o(y(L)). To perform classification on a test prompt t(x), we select the highest-probability class as follows
y⋆ = argmax ỹ∈Y p(o(ỹ) | d♮t(x)). (6)
Settings. We consider three settings: (1) Chance: random guessing uniformly across different classes;4 (2) Vanilla ICL: the standard ICL setting; (3) τRA-ICL: The responses r in the
4All the datasets are largely class-balanced.
demonstrations are replaced by g(r). The prompts p are left unchanged.
Implementation Details. We conduct experiments on a publicly available LLM: LLaMA2 (Touvron et al., 2023) with three sizes: 7B, 13B, and 70B. Each experiment is repeated 20 times with different random seeds and the average F1-Macro score is reported. In each experiment, we construct a distinct test set consisting of 256 prompt–response pairs, and for each element of this test set, we sample L prompt–response pairs from the training set, which serve as its demonstration.

heading: 3.2 Results
text: ICL Example Number. We present results on LLaMA2-70B with varying demonstration lengths (labeled as L) in Fig. 3. The τRA-ICL’s performance is near chance when the number of prompt–response pairs in the demonstration is small, but it quickly grows to above 80% as L increases and matches the performance of vanilla ICL on SST-2 and AG News when L is large.
Model Size. In Fig. 4, we report the ICL’s performance (L = 32) of models of different sizes. The performance of both vanilla ICL and τRA-ICL generally improves as the model size increases, but even the smallest model (LLaMA2-7B) yields a performance well above chance in τRA-ICL setting.
Summary. These results contradict Prediction 1 and demonstrate that an LLM can learn RA tasks τRA in context, which are highly unlikely to belong to the set of observed tasks T during pre-training. These experiments speak against Hypothesis 1.

heading: 4 Testing Hypothesis 2
text: We have shown in the previous section that an LLM can learn a novel task in context, but it is still unclear what type of tasks can be learned in context.
If we accept Hypothesis 2 as true, certain learning algorithms are learned by the pre-trained LLM, so learning from a demonstration in context happens on the fly at inference time. This hypothesis implies that there need not be knowledge of the prompted task in the pre-training data. And indeed, on this view, the role of the pre-training is merely to prepare the parameters in the LLM in such a way that the LLM architecture encodes various learning algorithms. For instance, some authors (Von Oswald et al., 2023; Akyürek et al., 2023; Dai et al., 2023) argue that training a linear model with gradient descent can be encoded as in-context learning under certain simplifying assumptions. This leads to the following prediction.
Prediction 2. If Hypothesis 2 is true, then ICL should behave similarly to a model trained with a certain learning algorithm, e.g., gradient descent.
A caveat of Prediction 2 is, of course, that we do not know a priori which learning algorithm the LLMs learn at pre-training time.

heading: 4.1 Experiment 1
text: We first propose a prompt-altered (PA) task, where instead of transforming the responses as τRA does, we transform the prompts, i.e., we create a stringto-string function h : Σ∗ → Σ∗ and apply it to the prompts of τ . This results in the following novel task τPA = ⟨πτPA , ρτPA⟩ where distributions are defined as
πτPA(p) = πτ (h −1(p)) (7a) ρτPA(r | p) = ρτ (r | h−1(p)). (7b)
The ICL setting with PA tasks is named τPA-ICL. We choose a h such that the PA task can be learned given the demonstration. If the RA task τRA in the τRA-ICL setting is indeed learned through a metalearning-esque procedure, τPA-ICL should be just
as easily learnable in the τPA-ICL setting because τRA and τPA are essentially the same task. And, moreover, performing both tasks in context should exhibit similar performance to a logistic regression classifier if the LLM encodes the ability to learn a linear model implicitly in its parameters.

heading: 4.1.1 Experimental Setup
text: Our experimental setup follows § 3.1. However, we introduce two more settings in addition to the ones we consider in § 3.1:
• τPA-ICL: As defined in (7), the prompts p (including the template tokens such as “Review” and “Sentiment”) in the demonstrations are replaced with h(p). The responses r are left unchanged.
• τPA-LR: To make sure a PA task τPA is learnable from a demonstration, we fit a logistic regression (LR) model as a baseline. Because a variablelength string cannot be easily fed into a logistic regressor, we first tokenize a string using the LLaMA2 tokenizer and then convert it into bagof-words (BoW) representations.5 The classifier is then trained using a BoW representation of exactly the same L prompt–response pairs as used in a demonstration of τPA-ICL. We use this baseline to gauge the performance of a model with minimum learning ability.

heading: 4.1.2 Results
text: The average accuracy of LLaMA2-70B across 3 datasets are in Fig. 5. More detailed results can be found in App. C.3.
τPA-LR. We find that the logistic regressor is able to learn the tasks to some degree, achieving an F1Macro score of around 50%. In contrast, a random
5More accurately, a bag of tokens.
guesser achieves an F1-Macro score of 41%. This experiment shows the task is indeed learnable to a certain extent given the demonstration as training data. And, as expected, the performance improves steadily as L increases.
τPA-ICL. However, in the τPA-ICL setting, the LLM always performs near or below chance regardless of the size of the demonstration. Indeed, the performance does not improve even when the demonstration length L reaches the maximum number of tokens allowed for LLaMA2 (Fig. 9).
τRA-ICL. In stark contrast to the τPA-ICL setting, as shown in Fig. 5, the LLM has an average score above 80% in the τRA-ICL setting.
Summary. The enormous performance gap between τRA-ICL and τPA-ICL does not concord with Prediction 2, and, thereby gives us evidence against Hypothesis 2. Specifically, our results imply that even though the RA tasks τRA are novel, they are not learned with some learning algorithm on the fly at inference time. This is in line with the observations of Kossen et al. (2024) and Shen et al. (2024).

heading: 4.2 Experiment 2
text: In our second experiment, we focus on one specific theoretical claim—specifically, that LLMs may implicitly learn a linear regression using gradient descent during ICL (Von Oswald et al., 2023; Akyürek et al., 2023; Dai et al., 2023).

heading: 4.2.1 Experimental Setup
text: Tasks. In addition to CR, SST-2, and AG News, we also consider DBpedia (Zhang et al., 2015), which has many more classes (Tab. 3).
Experimental Setup. We consider the task τ g induced by the string-to-string function g, as introduced in § 3. Given string– label pairs {(x(ℓ), y(ℓ))}Lℓ=1, we construct a demonstration for task τ g as follows: o(y(1))g(o(y(1)))♮ · · · ♮o(y(L))g(o(y(L))), i.e., pℓ = o(y (ℓ)) and rℓ = g(o(y(ℓ))). In this
formulation, both prompts and responses consist of a single token, i.e., p, r ∈ Σ. We denote the set of distinct responses as R ⊂ Σ.
The single-token construction allows us to model the task using linear regression. The embedding layer (0th layer) of an LLM is a matrix E ∈ R|Σ|×D, where D is the dimensionality of embeddings. We retrieve the row vector pℓ def = Epℓ,: ∈ R1×D for each prompt pℓ, where Epℓ,: denotes the row vector in E that corresponds to the token pℓ. We vertically stack the row vectors to create the embedding matrix P ∈ RL×D of the prompts in the demonstration.6 Also, let R ∈ RL×D be a similarly constructed embedding matrix of the responses, i.e., each response rℓ is embedded as a row vector rℓ ∈ R1×D. Additionally, for a test prompt– response pair ⟨p, r⟩ = ⟨o(y), g(o(y))⟩, we embed the prompt p as a row vector p ∈ R1×D. Thus, learning τ g is reduced to performing a multiple linear regression, i.e., learning a parameter matrix W⋆ ∈ RD×D that minimizes the following (nonstrictly) convex objective
W⋆ ∈ argmax W ||R−PW||2. (8)
Moreover, the data are constructed such that the test prompt–response pair ⟨p, r⟩ has appeared at least once in {⟨pℓ, rℓ⟩}Lℓ=1, so the task does not require generalization at all.
Settings. We compare the following two settings:
• τ g-ICL: We construct a demonstration d = p1r1♮ · · · ♮pLrL and perform classification as follows:
r⋆ = argmax r̃∈R p(r̃ | d♮p). (9)
The classification is correct if r⋆ = r.
• τ g-Linear: We train a linear regression with gradient descent. At inference time, we use a minimzer W⋆ to compute the predicted embedding
6The rows of P are linearly independent.
vector r⋆ for r as follows
r⋆ = pW⋆ ∈ R1×D. (10)
In order to evaluate its performance against ICL, we utilize the transposed embedding layer E⊤ ∈ RD×|Σ| to project r⋆ into R1×|Σ| and perform classification as follows:
r⋆ = argmax r̃∈R
( r⋆E⊤ ) r̃ . (11)
where (r⋆E⊤)r̃ denotes the entry of the vector (r⋆E⊤) that corresponds to the token r̃. We judge classification correct if r⋆ = r. Note that the embedding layer E is taken directly from the LLM and not trained with the linear model.
Implementation Details. For all the following experiments in § 4.2 and § 5, we experiment on the largest model (LLaMA2-70B) and set the demonstration length L = 32. We randomly sample 500 functions g for τ g. Each mapping g is constructed by randomly selecting a token from Σ as the image g(o(y)) of a o(y). We train τ g-Linear with also 32 examples for 80 epochs which corresponds to the 80 layers of LLaMA2-70B. We choose a learning rate of 1000, which we find yields the best performance. The correlation between the F1-Macro scores of τ g-ICL and τ g-Linear is computed.

heading: 4.2.2 Results
text: As shown in Tab. 1, τ g-Linear can learn most of the functions perfectly. On the other hand, τ g-ICL has a much lower average and higher variance. In the same table, we also list the correlation between the performance of τ g-Linear and τ g-ICL. In contrast to Prediction 2, there does not exist any significant correlation, which again gives us evidence against Hypothesis 2. It is worth mentioning that the high
variance of τ g-ICL’s performance also goes against the claim of Olsson et al. (2022) that there exists a special kind of heads called induction heads that copy any abstract pattern.

heading: 5 Testing Hypothesis 3
text: Because neither Hypothesis 1 nor Hypothesis 2 matches our experimental findings, we now turn to Hypothesis 3 to explain the empirical facts.

heading: 5.1 Experiment 1
text: We first examine the following prediction.
Prediction 3. Consider τRA = τ g ◦ τ . Then, an LLM’s ability to learn τRA in context correlates with its ability to learn τ g in context.
We verify it by comparing the performance of τRA-ICL and τ g-ICL.

heading: 5.1.1 Experimental Setup
text: The experimental setup of τRA-ICL and τ g-ICL follow that of § 3.1 and § 4.2, respectively. The correlation between the F1-Macro scores of τRA-ICL and τ g-ICL is computed.

heading: 5.1.2 Results
text: For each dataset in CR, SST-2, and AG News, we bucket the 500 data points for 500 functions into 8 bins to better visualize the results. The first group contains all functions g with an F1-Macro of 100% on τ g-ICL. The rest of the data points are put into 8− 1 = 7 bins evenly distributed and by increasing τ g performance. The mean and the standard deviation of τRA-ICL’s performance of each group are reported in Fig. 6. We compute the Pearson correlation coefficient (denoted as γp) and Spearman’s rank correlation coefficient (denoted as γs) between the τ g-ICL scores and the τRA-ICL scores of all the data points. There exists a modest
positive correlation with 0.34 ≤ γp ≤ 0.42 and 0.35 ≤ γs ≤ 0.51, but it is statistically significant with p-value smaller than 0.01 under Student’s t-test. We take it as evidence supporting that τRA is learned via composing τ g and τ .

heading: 5.2 Experiment 2
text: Next, rather than creating string-to-string functions g randomly, we construct hand-crafted functions that are intuitive and likely to have been in the pretraining data. We call such functions natural. One example of such a natural function is gsyn, where the prompts are mapped to synonyms. We compare ICL performance on these hand-crafted functions against that on random functions. If Hypothesis 3 were true, τ g-ICL should have a significantly higher performance on such natural functions.

heading: 5.2.1 Experimental Setup
text: We consider three types of natural functions: (1) synonym: Each prompt o(y) is mapped to one of its synonyms, (2) antonym: Each prompt o(y) is mapped to one of its antonyms, and (3) keyword: Each prompt o(y) is mapped to a keyword in its genre. Synonyms and antonyms are selected using PyMultiDictionary library.7 Keywords are obtained for each genre by querying GPT-4 (OpenAI, 2023). In contrast to random, we cannot create a large number of natural functions as easily. Therefore, we manually choose a candidate set of 10 possible synonyms (resp. antonyms and keywords) for each prompt. Thus, we can create a natural function by sampling a synonym (resp. antonym and keyword), from the candidate sets for every input. We create 500 such functions.

heading: 5.2.2 Results
text: We plot the mean F1 Macro scores as well as the standard deviations of τ g-ICL across different functions in Tab. 2. We observe that the LLM can learn synonym and keyword in context almost perfectly. The function antonym appears to be more difficult to learn, but still clearly much easier than random, which is most evident in DBPedia where the LLM has an F1-Macro score of 84.5% on antonym and only 59.8% on random. We also perform a onesided Welch’s t-test between each of antonym, synonym, keyword and random. The t-values are all greater than 2 and the p-values are all smaller
7https://github.com/ppizarror/ PyMultiDictionary. The library aggregates information from educalingo.com, synonym.com, and WordNet (Miller, 1994)
than 0.01. In other words, the natural functions are indeed significantly easier to learn in context, which is in line with our prediction.

heading: 5.3 Experiment 3
text: For Hypothesis 3 to be a good hypothesis, however, we need to additionally show why seemingly arbitrary tasks τ g would likely be elements of T ∗. We offer a tentative explanation. While we cannot show how to construct an arbitrary string-to-string function g out of natural functions, we can exhibit compositions of natural functions that appear arbitrary. We do so by composing natural functions, e.g., gsyn, repeatedly as follows
gmsyn(·) = gsyn(gsyn(· · · gsyn(·)))︸ ︷︷ ︸ ×m . (12)
We call gmsyn the m th power of gsyn.

heading: 5.3.1 Experiment Setup
text: The powers of the synonym functions are created the same way as in § 5.2.1. Because the candidate sets of synonyms are small (of size 10), to demonstrate how it is possible to create seemingly arbitrary functions, we adversarially sample functions such that for every p, gmsyn(p) ∈ Sm(p) \ Sm−1(p), where Sm(p) denotes the set of all the mth order synonyms of p, i.e., we only keep those that are not synonyms of lower orders. As an example of this strategy, we may have the following mapping: “company” is synonymous with “firm”, which is synonymous with “association” (as a noun) and “adamant” (as an adjective). Thus, we would map “adamant” to “company”, a seemingly unrelated word.

heading: 5.3.2 Results
text: We visualize a small subset of the synonyms of the two o(y) of CR/SST in Fig. 7. The concentric circles are the candidate sets, with radii representing their synonym orders. As can be seen, taking a natural function to higher power results in functions that appear as if they were arbitrary string-to-string mappings, e.g.,
“positive” → “tang” “negative” → “or”.
Yet, as our results demonstrate, ICL is still able to learn these functions well—above 90% F1-score— as shown in Fig. 8b. The same also holds true for the AG News and DBPedia text classification datasets. With a large task alphabet T , we expect an LLM to be able to learn many seemingly arbitrary string-to-string functions. More evidence demonstrating that these functions g are learned via composition comes from the noticeable performance decrease as the task composition becomes more complex. As shown in Fig. 8, there exists a
negative correlation (p-value < 0.01) between the performance of τ g-ICL and the order of synonym.

heading: 6 Conclusion
text: In this paper, we examine the ability of pre-trained LLMs to learn various tasks in context. We find that LLMs can learn text classification tasks with corrupted responses (τRA-ICL), but when the prompts are corrupted in the same manner (τPA-ICL), ICL performs significantly worse than a logistic regression model. A closer look suggests the tasks in the τRA-ICL may have been learned via composing primitive tasks learned during pre-training. Overall, our paper provides insights into the nature, abilities, and limitations of ICL.

heading: Acknowledgements
text: This publication was made possible by an ETH AI Center doctoral fellowship to Jiaoda Li. Yifan Hou is supported by the Swiss Data Science Center PhD Grant (P22-05).
Limitations
One limitation is that we only experiment on one family of LLMs—LLaMA2. Our results might not hold on larger models, e.g., GPT-4. Another limitation is that our formalization is not complete and is, sadly, a bit more vibes-based than we would have liked. We hope to achieve a more concrete formalization in future work.
Ethical Considerations
The datasets and pre-trained LLMs that we use are all publicly available. Our paper focuses on model interpretation, and we thereby do not foresee any ethical issues originating from this work.

heading: A Related Work
text: Many papers attempt to provide a theoretical grounding for ICL’s emergence, hypothesizing that ICL emerges from pre-training on documents that are drawn from a mixture of latent concepts (Xie et al., 2022; Wang et al., 2023; Wies et al., 2023) or a compositional attribute grammar (Hahn and Goyal, 2023). After pre-training, the LLM can recognize the concept or the generative process that the prompt is sampled from and use it for next-token prediction. Another stream of research endeavors to show how LLMs can learn new tasks in context. Von Oswald et al. (2023); Akyürek et al. (2023); Dai et al. (2023) show by construction that the transformer can implicitly implement various learning algorithms, such as gradient descent and least squares regression. Nevertheless, there is a gap between the settings assumed in the papers and the ones used in practice, e.g. the attention mechanism needs to be linear, or only linear regression problems are considered. Empirically, it has been shown that transformers can learn various function classes in context, e.g., linear regression (Raventos et al., 2023), multi-layer perceptrons and decision trees (Garg et al., 2022), and regular languages (Akyürek et al., 2024), but the models are trained from scratch on synthetic data derived from functions in the same class, which do not conform with how LLMs are pre-trained in practice. It is argued that their performance might not be extrapolated to models with a larger size (Wei et al., 2022) or longer training time (Singh et al., 2023). Thus, we fix our attention on LLMs that are pre-trained on large natural text corpus and are practically used. Most related to our work are Kossen et al. (2024); Shen et al. (2024), which also find that ICL’s behavior is different from models trained with conventional learning algorithms such as gradient descent. Our work is distinct in the investigation on the possibility of ICL learning via task composition.

heading: B Associativity of Task Composition
text: Proposition 1. Task composition is associative, i.e., (τ1 ◦ τ2) ◦ τ3 = τ1 ◦ (τ2 ◦ τ3). Proof. The proof follows by simple manipulation:
ρτ1◦(τ2◦τ3)(r | p) = ∑
r̃1∈Σ∗ ρτ1(r | r̃1)
  ∑
r̃2∈Σ∗ ρτ2(r̃1 | r̃2)ρτ3(r̃2 | p)
 
= ∑
r̃2∈Σ∗
  ∑
r̃1∈Σ∗ ρτ1(r | r̃1)ρτ2(r̃1 | r̃2)
  ρτ3(r̃2 | p)
= ρ(τ1◦τ2)◦τ3(r | p).
(13)
πτ1◦(τ2◦τ3) = πτ2◦τ3 = πτ3
= π(τ1◦τ2)◦τ3
(14)
■

heading: C Supplementary Results
text: C.1 Dataset Details The prompt templating functions t(x) and the responses (o(y) for CR, SST-2, and AG News are in Tab. 3. The altered responses are tokens from the Lorem Ipsum generator.
C.2 (First-Order) Synonym The 10 candidate synonyms for each prompt are in Tab. 4.
C.3 Detailed Results We provide detailed results on CR, SST-2, and AG News in Fig. 9. We report LLaMA2 with three model sizes. The results on LLaMA2-7B and LLaMA2-13B are consistent with those on LLaMA2-70B that we report in the main text.

heading: D Experimental Setup
text: We implement our experiments on A100-80G. Each experiment takes around 1-2 GPU hours. For LLaMA2, We use the implementation and pre-trained weights provided by HuggingFace (Wolf et al., 2020). CR (Hu and Liu, 2004) and SST (Socher et al., 2013) are under the CC-BY license. AG News (Zhang et al., 2015) and DBpedia (Zhang et al., 2015) are under the BSD-3-Clause license. We use these datasets consistently with their intended use.

heading: Future Work
text: We hope to achieve a more concrete formalization in future work. Ethical Considerations
The datasets and pre-trained LLMs that we use are all publicly available. Our paper focuses on model interpretation, and we thereby do not foresee any ethical issues originating from this work