Title: From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When
Venue: NeurIPS 2024
Accepted: yes
Keywords: in-context learning

large language models

unstructured data

continuous bag of words

co-occurrence
Abstract: Large language models (LLMs) like transformers demonstrate impressive in-context learning (ICL) capabilities, allowing them to make
predictions for new tasks based on prompt exemplars without parameter updates. While existing ICL theories often assume structured training data resembling ICL tasks (e.g., x-y pairs for linear regression), LLMs are typically trained unsupervised on unstructured text, such as web content, which lacks clear parallels to tasks like word analogy. To address this gap, we examine what enables ICL in models trained on unstructured data, focusing on critical sequence model requirements and training data structure. We find that many ICL capabilities can
emerge simply from co-occurrence of semantically related word pairs in unstructured data; word analogy completion, for example, can provably arise purely through co-occurrence modeling, using classical language models like continuous bag of words (CBOW), without needing positional information or attention mechanisms. However, positional information becomes crucial for logic reasoning tasks requiring generalization to unseen tokens. Finally, we identify two cases where ICL fails: one in logic reasoning tasks that require generalizing to new, unseen patterns, and another in analogy completion where relevant word pairs appear only in fixed training positions. These findings suggest that LLMs' ICL abilities depend heavily on the structural elements within their training data.

From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When

Abstract

Large language models (LLMs) like transformers demonstrate impressive incontext learning (ICL) capabilities, allowing them to make predictions for new tasks based on prompt exemplars without parameter updates. While existing ICL theories often assume structured training data resembling ICL tasks (e.g., x-y pairs for linear regression), LLMs are typically trained unsupervised on unstructured text, such as web content, which lacks clear parallels to tasks like word analogy. To address this gap, we examine what enables ICL in models trained on unstructured data, focusing on critical sequence model requirements and training data structure. We find that many ICL capabilities can emerge simply from co-occurrence of semantically related word pairs in unstructured data; word analogy completion, for example, can provably arise purely through co-occurrence modeling, using classical language models like continuous bag of words (CBOW), without needing positional information or attention mechanisms. However, positional information becomes crucial for logic reasoning tasks requiring generalization to unseen tokens. Finally, we identify two cases where ICL fails: one in logic reasoning tasks that require generalizing to new, unseen patterns, and another in analogy completion where relevant word pairs appear only in fixed training positions. These findings suggest that LLMs' ICL abilities depend heavily on the structural elements within their training data. 1

1 Introduction

Large language models (LLMs) such as transformers demonstrate remarkable in-context learning (ICL) abilities [10]: without any parameter updates, they can recognize tasks and generate predictions from prompt examples. For instance, given the prompt dog anjing, cat kucing, lion singa, elephant , a well-trained LLM should detect the English-to-Indonesian pattern in the prompt and predict gajah -the Indonesian translation for elephant -as the most likely next token. The ICL capabilities of LLMs are surprising for two main reasons. First, these models are trained in an unsupervised manner on unstructured natural language data through next-token prediction, without any loss function specifically designed for ICL. Second, the training data for LLMs likely lacks sequences resembling typical ICL prompts, i.e., of the form c 1 d 1 · · · c K d K , where ( c k , d k ) 's represent word pairs with specific semantic relationships.

Many efforts have sought to understand ICL from theoretical and empirical perspectives, e.g., gradient descent in regression and Bayesian inference. While insightful, these analyses often rely on structured training data that mirrors ICL tasks. For instance, they train on sequences of x-y pairs from various linear regression tasks and test on similar data. In practice, however, LLMs are trained in an unsupervised manner on unstructured text data, such as web content, which bears little resemblance

Section 2: ICL from CBOW

Section 3: ICL from modeling positions

Section 4: Failures ofICL

Example:

Training data:

Example:

Training data:

Example:

Trainingdata:

The capital city of

pqcrWxZd

5,2,3,5.Predict the next

abca pqrs defd

PEK is the airport

C1pqrwsd

Indonesia is Jakarta.

wXdc2d2rYm

number: 8,4,6,8.

code for Beijing.

C2mxtspd2

capital of China.

Gas prices in India are

Beijing functions as the

C4e4wr0AqB

stC3xye3CD

next? 2,1,3,2,5,7,6,5.

Guess the next number

What numbercomes

yzrt aded bcac

deaebfcftqzn

airport serving Chicago.

SINdenotes theprimary

ORDuniquelyidentifies the

C4polqwd4

around10oRupeesperliter.

Cedi,the currency of Ghana,

ICLprompts:

2,9,7,9,1,3,6,3.Extend

1,2,3,2,4,6,7,6.

ICL prompts:

airport for Singapore.

DEN is an airport located in

ICL prompt:

is facing depreciation.

ICL prompts:

CdC2d2c3d3

the pattern please.

abdb cded bdad

abdacdecbdab

Denver.

Cd1C2d2C3d3

GhanaAccraNigeriaLagos

ICL prompts:

ICL prompt:

JapanYen

China Beijing

IndiaRupee ChinaYuan

Patterns:

(Ci, di),(Ci,ei)

2,4,6,2,8,9,1,8,5,7,4,5

6,9,2,9,1,2,3,2,5,1,21

Repeat 1st or 2nd token

Patterns:

PEK Beijing ORD

Chicago SIN Singapore

(Ci, di)

Pattern:

to typical ICL tasks like word analogy. Consequently, these analyses may only partially capture the complexities of ICL.

This work. We investigate common ICL tasks to identify what tasks can be learned in context by a model trained on unstructured data. Specifically, we examine essential components of sequence modeling that enable in-context learning, along with requirements on the unstructured training data.

The first set of (theoretical and empirical) results focuses on ICL for word analogy completion using frequently co-occurring tokens [10, 50]. This task involves identifying relationships between word pairs, such as (country)-(capital) and (English word)-(Indonesian translation) , then applying the same relationship to complete a sequence. For this task (see left of Figure 1), we explore cases where training sentences contain one or two types of word pairs with distinct semantic relationships. We prove that, in most cases, ICL can arise by simply modeling word co-occurrence using classical (pre-transformer) language models like continuous bag of words (CBOW) [36], without needing positional information or attention mechanisms.

The second set of results involves ICL for logic reasoning tasks that require recognizing patterns that do not commonly co-occur in a sentence, such as (word)-(first letter) [13, 62]. For this task (see middle of Figure 1), we investigate scenarios where training sentences contain one or two distinct patterns, as well as a more realistic scenario where nuisance tokens are present. We prove that positional information and blocked nuisance structure (e.g., pqrs in Figure 1) are crucial for the success of ICL in these tasks . This finding aligns with Chen et al.'s [2024] observation that parallel structures in pre-training data support ICL. We also find that learned positional embeddings generally perform better, except in scenarios where the nuisance tokens are not clustered in blocks.

Finally, we present two scenarios where ICL fails regardless of model architectures (see right of Figure 1). In the first scenario (left example), we consider a logic reasoning task that involves identifying and completing meta-patterns within sequences. Here, each training sequence repeats the pattern established by its starting tokens; the ICL task sequence then requires the model to recognize this meta-pattern of repetition and generalize it to a novel, unseen starting pattern. In the second scenario (right example), we examine a word analogy completion task in which relevant word pairs appear in the unstructured training sentences but are restricted to specific fixed positions. These findings, along with their empirical and theoretical explanations, underscore that LLMs require specific structures in the pre-training data to exhibit ICL ability .

Summary of contributions. We (1) theoretically and empirically show that ICL for word analogy tasks with semantically related word pairs can arise from modeling co-occurrence patterns using CBOW, (2) prove that, to recognize token patterns and generalize them to novel tokens, ICL requires modeling positional information and blocked nuisance structure , and (3) present scenarios where ICL fails, highlighting the crucial role of training data structure for ICL.

Related work. Below, we highlight some of these studies and explain how our research aligns with, yet differs from, these approaches. We include a detailed discussion of related work in Appendix A. Numerous studies have connected ICL to classical methods, including gradient descent [3, 5, 17, 55, 66], Bayesian inference [14, 56, 67], and Newton's method [20]. In contrast, our work links ICL to the continuous bag of words (CBOW) model , showing that ICL for word analogy tasks can be achieved by learning co-occurrence patterns. Several studies have examined the pre-training aspects of ICL, such as data distribution [11, 30, 37] and task diversity [42, 63]. By comparison, our work emphasizes the importance of co-occurrence, positional information, and training data structure for ICL to arise. Other research has explored ICL in specific data-generating processes, such as discrete functions [8] and autoregressive processes [45]. In contrast, our work centers on data characterized by semantically related word pairs and repeating token patterns .

2 In-context learning can arise by modeling co-occurrence via CBOW

In this section, we focus on in-context learning (ICL) for word analogy tasks involving word pairs that frequently co-occur in training sentences; see Figure 1 (left). To motivate the discussion, we present two experiments using the LLaMA 2 model [51] involving countries (or US states) and their capital cities (see Appendix K for data sources). The prompts follow the format c 1 d 1 , c 2 d 2 , · · · , c 6 d 6 , c 7 , where c i is a country (or US state) and d i is its capital city. In this scenario, we consider ICL successful if the model outputs d 7 -the capital city of c 7 -as the most likely token.

Experiment 1. We consider all 160 countries with a population exceeding one million in 2022. Among these countries, 31 have capital cities that are not their most populous cities, denoted by type A . The remaining 129 countries fall under type B . Each ICL prompt includes three type A countries among c 1 , · · · , c 6 to emphasize that the desired relationship is (country)-(capital) rather than (country)-(largest city) . Subsequently, we randomly generate 1,000 prompts, with 500 having a c 7 being a type A country and 500 having a c 7 being a type B country. The model's ICL accuracy is 0 . 58 for type A and 0 . 96 for type B.

Experiment 2. We consider all 50 US states, among which 33 are of type A and 17 are of type B , similarly defined. Following the setup in Experiment 1, we generate prompts for these states. The ICL accuracy is 0 . 69 for type A and 0 . 84 for type B.

In both experiments, LLaMA 2 performs better on type B prompts (i.e., the capital city as the largest city). Since larger cities tend to appear more frequently as compared to smaller ones in the model's pre-training data, this naturally raises the question: Can/does ICL with frequently co-occuring word pairs arise purely from modeling co-occurrence patterns?

̸

ICL via classical non-transformer-based language models. We prove that, for word analogy tasks with frequently co-occurring word pairs, ICL can be achieved by modeling token co-occurrencewithout needing positional encoding or attention mechanisms-using classical, non-transformer language models such as the continuous bag of words (CBOW) model [36]. (It does not imply that ICL in transformer-based models arises through learning co-occurrence patterns.) We utilize CBOW variant where each center word is modeled conditional on all other words in a sentence, not just neighboring words. Specifically, each word w has center and context embeddings u w and v w of the same dimension. Given a sentence x 1 x 2 · · · x I , the i -th word ( x i ) is distributed as p ( x i = k | x -i ) ∝ exp(( u ⊤ k ∑ j = i v x j ) / ( I -1)) , with u w 's and v w 's learned by minimizing cross-entropy losses across all positions.

Roadmap of Section 2. In Section 2.1, we begin by considering a simple ICL task of the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 , where ( c i , d i ) represents a frequently co-occuring word pair (e.g., a country and its capital city) and i 1 , i 2 , · · · , i ℓ +1 are all distinct. The focus is to investigate whether a trained CBOW model can correctly output d i ℓ . We also explore two other scenarios: ICL tasks of the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 and c i 1 e i 1 · · · c i ℓ e i ℓ c i ℓ +1 in Section 2.2 (two connected word relationships), as well as c i 1 d i 1 · · · c i ℓ -1 d i ℓ -1 c i ℓ and e i 1 f i 1 · · · e i ℓ f i ℓ e i ℓ +1 (two disjoint word relationships) in Section 2.3. Section 2.4 concludes with synthetic experiments supporting the theory.

2.1 In-context learning on single-relationship word analogy tasks

We investigate ICL in single-relationship word analogy tasks, where the training data contains only one type of relationship between frequently co-occurring word pairs. This task takes the form of c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 , where ( c i , d i ) pair represents a frequent co-occurrence, such as a country and its capital city. The vocabulary consists of c 1: K , d 1: K , r 1: L , where r ′ i s represent other words (e.g.,

stop words). We first introduce Theorem 1, which states that ICL can arise if each sentence consists of exactly one ( c i , d i ) pair, as long as the number of in-context examples ( ℓ ) is not too large. To simplify calculations, we replace the cross-entropy loss with squared loss by removing the softmax activation and comparing outputs against the one-hot encoding of target words. The proof is in Appendix B.

Theorem 1 (ICL on single-relationship word analogy tasks) . Let K,L ≥ S ≥ 3 . Suppose each training sentence of length S is generated by selecting one ( c i , d i ) pair and S -2 distinct r i 's uniformly at random. We train a CBOW model with the squared loss and a sufficiently large embedding dimension on these sentences. Given a prompt c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 with distinct i k 's, the model correctly predicts d i ℓ +1 if and only if 2 ℓ +1 < KL ( S -1) 3 ( K + L )( S -2) 2 ( S -1)+ K ( S -2)( S -1) 2 -2( S -2) 4 .

As an example, when each training sentence contains exactly one country-capital pair (i.e., ( c i , d i ) ), Theorem 1 says that a trained CBOW model will correctly predict d i ℓ +1 (i.e., the capital city of c i ℓ +1 ) given an ICL prompt of the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 , provided that the prompt length is not too large. Intuitively, this behavior is due to the presence of c i ℓ +1 in the ICL prompt, leading the model to correctly predict d i ℓ +1 given the frequent occurrences of the pair ( c i ℓ +1 , d i ℓ +1 ) in the training data. However, when the prompt length is too large, the model will instead predict one of the r i 's (see Theorem 1's proof in Appendix B for more details). Moreover, if we let L →∞ and fix K and S , the condition in Theorem 1 becomes 2 ℓ +1 < K ( S -1) 2 / ( S -2) 2 . This inequality trivially holds if the prompt length is set to be S -1 to match the length of the training sentences.

Furthermore, it is possible to adapt the proof of Theorem 1 to handle the case when each sentence comprises exactly two (not one) different ( c i , d i ) pairs. In this case, letting L → ∞ and fixing K and S , the model correctly predicts d i ℓ +1 given the same ICL prompt if and only if 2 ℓ + 1 <

K ( K -2)( S -1) 2 ( K -2)( S -2)( S -4) -K . This upper bound is strictly larger than K ( S -1) 2 / ( S -2) 2 : when each sentence contains exactly two ( c i , d i ) pairs, ICL under the squared loss holds for longer prompts.

Experiments. To empirically verify Theorem 1 and its generalizations, we conduct experiments using the cross-entropy loss with S = 8 , K = 10 , L = 20 , and ℓ = 3 . We explore multiple ( p 0 , p 1 , p 2 ) values, where p k denotes the probability of having exactly k pairs of ( c i , d i ) in the sentence. For each ( p 0 , p 1 , p 2 ) triple, we also introduce a more realistic setting where c i and d i do not always appear together by considering its corrupted version. In this setup, each ( c i , d i ) pair has a 25% chance of being replaced with ( c i , r j ) and a 25% chance of being replaced with ( d i , r j ) for some j ∈ [ L ] . More details are provided in Appendix K.

Results. Table 1 displays the average accuracy for each scenario, calculated over 10 repetitions. Notably, when ( p 0 , p 1 , p 2 ) is (0 , 1 , 0) or (0 , 0 , 1) , ICL under the cross-entropy loss achieves zero accuracy, in contrast to perfect accuracy with the squared loss as shown in Theorem 1. We believe this difference in accuracy is an artifact of the loss functions used, although its relevance is limited by the fact that it is unlikely for every sentence to contain at least one ( c i , d i )

Table 1: ICL on different single-relationship word analogy tasks, averaged over 10 repetitions, demonstrates stable, good performance across embedding dimensions ( d E ), as Theorem 1 suggests. The corrupted setting also demonstrates excellent ICL ability under certain scenarios.

pair, in reality. On the other hand, perfect ICL performance is observed in other settings (e.g., when the training sentences contain either zero, one, or two ( c i , d i ) pairs) in both the clean and corrupted scenarios. For an in-depth comparison of ICL performance using both the squared and cross-entropy loss across various numbers of demonstration examples, see Appendix C.

2.2 In-context learning on dual-connected-relationship word analogy tasks

Building on the scenario that contains only a single type of relationship between frequently cooccurring word pairs, namely ( c i , d i ) , we now explore ICL on dual-connected-relationship word

analogy tasks. Here, some words frequently co-occur with two different types of words in the training data, represented by the relationships ( c i , d i ) and ( c i , e i ) . For instance, c i might represent a country, d i its capital city, and e i its currency. The vocabulary is comprised of c 1: K , d 1: K , e 1: K , r 1: L , where r i 's represent other words. Moreover, the corresponding ICL tasks take the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 and c i 1 e i 1 · · · c i ℓ e i ℓ c i ℓ +1 , where the model is expected to output d i ℓ +1 and e i ℓ +1 , respectively. These can be regarded as task selection since the model should use the in-context examples to infer the tasks. We present Theorem 2, stating that a trained CBOW model can perform task selection if each sentence contains exactly two distinct ( c i , d i ) or two distinct ( c i , e i ) pairs with uniform probability. 2

Theorem 2 (Task selection in CBOW) . Let K,L ≥ 2 and S ≥ 5 . Suppose each training sentence of length S is generated by selecting two distinct ( c i , d i ) pairs or ( c i , e i ) pairs, and S -4 distinct r i 's uniformly at random. We train a CBOW model with the squared loss and a large enough embedding dimension. Given a prompt c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 ( c i 1 e i 1 · · · c i ℓ e i ℓ c i ℓ +1 ) with distinct i k 's, the model is more likely to predict d i ℓ +1 ( e i ℓ +1 ) than e i ℓ +1 ( d i ℓ +1 ). (The proof is in Appendix D.)

According to Theorem 2, when each training sentence includes two ( c i , d i ) pairs or two ( c i , e i ) pairs, a trained CBOW model is capable of performing task selection. To intuitively understand this result, consider the ICL prompt of the first type, i.e., c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 . Here, the output is more likely to be d i ℓ +1 than e i ℓ +1 since d i ℓ +1 co-occurs with the other d i j 's in the training data (and e i ℓ +1 does not). Note that in Theorem 2, we unrealistically require each sentence to contain either two distinct ( c i , d i ) pairs or ( c i , e i ) pairs. However, this condition is not necessary as we empirically show next.

Experiments. We use the cross-entropy loss with S = 8 , K = 10 , L = 60 , and ℓ = 3 . Each training sentence is equally likely to be a cd sentence (i.e., containing ( c i , d i ) pairs) or a ce sentence (i.e., containing ( c i , e i ) pairs), but not both. We explore multiple ( p 0 , p 1 , p 2 ) 's, where p k is the probability of having exactly k pairs of ( c i , d i ) for a cd sentence, or k pairs of ( c i , e i ) for a ce sentence. Additionally, we introduce three different scenarios: balanced , where all L random words are equally likely to occur in both cd and ce sentences; imbalanced , where L/ 3 words are more likely to occur in cd ( ce ) sentences; and extreme , where L/ 3 of the words can only occur in cd ( ce ) sentences. More details are provided in Appendix K.

Results. Table 2 shows the accuracies of both tasks for each scenario, averaged over 10 repetitions. We observe a perfect accuracy when ( p 0 , p 1 , p 2 ) ∈ { (1 / 2 , 0 / 1 , 2) , (0 , 1 / 2 , 1 / 2) , (1 / 3 , 1 / 3 , 1 / 3) } across all embedding dimensions and scenario types. The near-zero accuracy when ( p 0 , p 1 , p 2 ) or (0 , 1 , 0) or (0 , 0 , 1) is again an artifact of the cross-entropy loss, as discussed in Section 2.1.

Interestingly, ICL works in the imbalanced and extreme scenarios when ( p 0 , p 1 , p 2 ) = (1 / 2 , 1 / 2 , 0) , where sentences do not contain more than one ( c i , d i ) or ( c i , e i ) pair. To see this, consider the balanced scenario where each r i is equally probable to appear in both types of sentences. Given a prompt of the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 , it is easy to see that the model should output d i ℓ +1 or e i ℓ +1 with equal probability. On the other hand, in the imbalanced and extreme scenarios, the information from the r i 's can allow for task selection, thus contributing to the success of ICL.

2.3 In-context learning on dual-disjoint-relationship tasks

We next replicate the experiments in Section 2.2, but with disjoint word pair relationships of two distinct types with no overlapping tokens, i.e., ( c i , d i ) and ( e i , f i ) . For example, ( c i , d i ) represents a country and its capital city, and ( e i , f i ) represents a company and its CEO. Our vocabulary consists of c 1: K , d 1: K , e 1: K , f 1: K , r 1: L , where r i 's represent other words; see Appendix K for details.

Results. Table 3 summarizes the accuracies of the ICL tasks c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 and e i 1 f i 1 · · · e i ℓ f i ℓ e i ℓ +1 for each scenario, averaged over 10 repetitions. Similar to the connected setting in Section 2.2, we observe a perfect accuracy when ( p 0 , p 1 , p 2 ) ∈ { (1 / 2 , 0 / 1 , 2) , (0 , 1 / 2 , 1 / 2) , (1 / 3 , 1 / 3 , 1 / 3) } across all embedding dimensions and scenario types. However, when ( p 0 , p 1 , p 2 ) = (1 / 2 , 1 / 2 , 0) , ICL already works well in the balanced scenario. Intuitively, this is because the two relationships are disjoint, thus making task selection easier.

In addition, we consider a contaminated version of the training data where cd ( ef ) sentences can contain some e i 's and f i 's ( c i 's and d i 's). We also obtain a perfect accuracy when ( p 0 , p 1 , p 2 ) ∈ { (1 / 2 , 0 / 1 , 2) , (0 , 1 / 2 , 1 / 2) , (1 / 3 , 1 / 3 , 1 / 3) } across all embedding dimensions and scenario types.

2.4 Experiments on a synthetic corpus

We conduct experiments on a synthetic corpus consisting of (country)-(capital) and (country)-(IOC code) relationships. Each sentence in the corpus is categorized into exactly one of six possible categories: (1) exactly one country-capital pair; (2) exactly two country-capital pairs; (3) exactly one country-IOC pair; (4) exactly two country-IOC pairs; (5) exactly one country without any pair; and (6) no country. In sentences with country-capital pairs, each capital city can appear in any position relative to the country. Conversely, in sentences with country-IOC pairs, each IOC code must directly follow the country. The data source and corpus generation process are detailed in Appendix K.

Two models are trained on this corpus: a CBOW and a five-layer two-head autoregressive transformer. Both models have an embedding dimension of 100 . We then compare the ICL accuracies for both relationships given one to five in-context examples. For the CBOW model, the country-capital accuracies are (0 . 81 , 0 . 82 , 0 . 78 , 0 . 73 , 0 . 65) and the country-IOC accuracies are (0 . 15 , 0 . 38 , 0 . 59 , 0 . 71 , 0 . 79) . Here, the i -th number corresponds to the accuracy given i in-context examples. For the transformer, the accuracies are (0 . 00 , 0 . 15 , 0 . 34 , 0 . 22 , 0 . 07) and (1 . 00 , 0 . 77 , 0 . 78 , 0 . 97 , 0 . 99) , respectively.

When using the transformer, we find that the accuracies for the country-IOC task are significantly higher compared to those for the country-capital task . This is likely because each IOC code consistently follows the corresponding country in the corpus, similar to ICL prompts. On the other hand, ICL fails to work on the country-capital task, where there is no consistent pattern in how each pair occurs in the corpus. Meanwhile, ICL works decently well on both tasks under the CBOW model .

3 The essential role of positional information in enabling in-context learning

Weexamine another common example of in-context learning (ICL), where the task involves predicting the first (or second) token in a sequence. This task resembles general logic reasoning tasks that require recognizing patterns that do not typically co-occur in a sentence, such as (word)-(first letter) [13, 62]. While Section 2 shows that positional encoding is irrelevant for ICL in word analogy tasks, positional

information proves essential for such logic reasoning tasks. Specifically, we consider a simpler task of modeling x i 1 x i 2 x i 3 x i 1 . Theorem 3 underscores the importance of positional information to correctly predict x i 1 from x i 1 x i 2 x i 3 in a single-layer model, and provides a construction of an attention-based model achieving zero loss and perfect accuracy on this task. Its proof is in Appendix E.

̸

Theorem 3 (Necessity of modeling positions) . Let the vocabulary be V = { 1 , 2 , · · · , | V |} and the training sequences take the form x i 1 x i 2 x i 3 x i 1 , where x i 1 = x i 2 = x i 3 = x i 1 are chosen uniformly at random from V . Consider a one-layer model that predicts the last x i 1 via a learned function f ( { x i 1 , x i 2 } , x i 3 ) using the cross-entropy loss. In this case, it is not possible to achieve pefect accuracy or zero loss. On the other hand, we can achieve zero loss (and thus perfect accuracy) by incorporating positional information, i.e., via a learned function ˜ f ( { ( x i 1 , 1) , ( x i 2 , 2) } , ( x i 3 , 3)) .

̸

̸

Here, f ( { x i 1 , x i 2 } , x i 3 ) represents a scenario where the model lacks positional information (e.g., f is a one-layer autoregressive transformer without positional embeddings). Note that the output of this function is identical for inputs x i 1 x i 2 x i 3 and x i 2 x i 1 x i 3 , which leads to the impossibility of attaining zero loss. In contrast, ˜ f ( { ( x i 1 , 1) , ( x i 2 , 2) } , ( x i 3 , 3)) refers to a scenario where the model has access to positional information. We provide a construction of ˜ f that achieves zero loss in Appendix E.

Experiments. We validate Theorem 3 by training transformers with causal masking to autoregressively learn sequences of the form x i 1 x i 2 x i 3 x i 1 , and assessing their accuracy in predicting the last token on a separate test data of the same pattern. We use | V | = 20 and an embedding dimension of 10 . We consider these settings: (i) number of layers : 1, 5; (ii) positional embeddings : learned, sinusoidal, no positional embeddings; and (iii) train-test split : each token in the vocabulary is the first token in both the training and test sets ( Both ), each token in the vocabulary is the first token in either set, but not both ( Either ). More details are provided in Appendix K.

Results. Table 4 summarizes the results. Two main findings emerge: (1) for the model to generalize to unseen sentences, each token in V should be present as the first token in both the training and test sets; (2) positional embeddings are crucial when using only one attention layer. Note that in practice, the condition in (1) is likely met due to the vast size of LLMs' pre-training data.

Table 4: Prediction accuracy with single/multi-layer models. For ICL to occur, the first tokens of training sentences should cover the entire vocabulary ( Both ). Also, positional embeddings are essential, especially in one-layer models.

Multiple layers. Proposition 1 shows that multi-layer models can encode positional information without explicit positional embeddings.

Proposition 1 (Multi-layer models can encode positions) . Consider the sentence x i 1 x i 2 x i 3 x i 1 . Using a two-layer autoregressive model, the model's final output for predicting the last x i 1 is given by t ( x i 1 x i 2 x i 3 ) := g 3 ( { f 1 ( { x i 1 } ) , f 2 ( { x i 1 } , x i 2 ) } , f 3 ( { x i 1 , x i 2 } , x i 3 )) for some f 1 , f 2 , f 3 , and g 3 .

̸

The proof is in Appendix F. Proposition 1 shows that we generally have t ( x i 1 x i 2 x i 3 ) = t ( x i 2 x i 1 x i 3 ) , unlike in the one-layer case. Consequently, high accuracy is achievable without positional embeddings, as shown in Table 4. This result parallels findings in Haviv et al. [26] that autoregressive transformers implicitly encode positions, even without positional embeddings.

̸

Roadmap of Section 3. In the rest of this section, we consider settings where each sentence contains repeating patterns. Section 3.1 focuses on a simple scenario where training sentences follow the form abacdc , where a = b and c = d , or a noisy variation of it. The ICL prompts maintain the same pattern but use different combinations of ab and cd from those in the training data. Our goal is to understand what types of training data facilitate ICL in clean or noisy scenarios. Section 3.2 explores a more realistic case where two possible patterns are present: repeating the first letter ( abca ) and repeating the second letter ( abcb ).

̸

3.1 In-context learning on single-pattern tasks

In this section, we examine the case where the training sentences follow the pattern abacdc . To replicate real-world training scenarios, we also analyze how incorporating nuisance tokens into the training sentences affects the ICL capability of autoregressive models. To formalize the discussion, let our vocabulary be V ∪N , where N represents the nuisance tokens. We define S = { ( a, b ) | a, b ∈

̸

V , a = b } and partition S into S 1 (for training sentences) and S 2 (for ICL prompts). This is to ensure that training sentences are distinct from ICL prompts. Furthermore, we assume { c [1] | c ∈ S 1 } = { c [1] | c ∈ S 2 } = V , where c [ i ] is the i -th element of c . In other words, each token in V can be the first token in both the training sentences and ICL prompts. We consider three scenarios:

Clean : Training data follow the form abacdc where ab, cd ∈ S 1 . ICL prompts follow the form abacd where ab , cd ∈ S 2 .

One-noisy : Training data follow the form abacdc where ab, cd ∈ S 1 , with one nuisance token n ∈ N randomly inserted anywhere except the last position (to ensure ICL prompts do not resemble the training data). ICL prompts follow the form abacd where ab , cd ∈ S 2 .

Block-noisy : Training data follow the form abacdc where ab, cd ∈ S 1 , with three consecutive nuisance tokens n 1 , n 2 , n 3 ∈ N randomly inserted while preserving the aba and cdc blocks. ICL prompts follow the form abacdcef where ab , cd , ef ∈ S 2 .

We set the vocabulary size | V | = 20 , the number of nuisance tokens N = 20 , and use only one attention layer as additional layers do not improve performance. See Appendix K for more details.

Results. Table 5 reveals interesting phenomena. First, under the clean data scenario, ICL performs exceptionally well, with an observed performance increase with learned positional embeddings and a larger embedding dimension. However, ICL is notably challenging under the one-noisy scenario. In the block-noisy scenario, learned positional embeddings are crucial for satisfactory ICL performance. Theorem 4 formalizes these findings.

Theorem 4 (Blocked nuisance token structure facilitates ICL) . Consider a sufficiently large autoregressive position-aware model that can achieve the minimum possible theoretical loss. Training this model in the one-noisy (block-noisy) scenario results in zero (perfect) ICL accuracy.

The proof is in Appendix G. Theorem 4 says that ICL works perfectly under the block-noisy scenario, yet fails to work under the one-noisy scenario. However, as shown in Table 5, the use of sinusoidal positional embeddings significantly enhances prediction accuracy in the one-noisy scenario. This may be due to the fact that sinusoidal embeddings can encode relative positional information [53]. For example, training sentences of the form nabacdc , where n ∈ N , may help in predicting the most likely token following the ICL prompt abacd .

3.2 In-context learning on dual-pattern tasks

We next examine the case where training data and ICL prompts contain two different patterns occurring equally likely: abcadefd and abcbdefe , where a, b, c and d, e, f are distinct. We consider the clean and block-noisy scenarios as in Section 3.1, and set | V | = N = 20 (details in Appendix K).

Results. Table 6 outlines the ICL performance for both scenario types across different model configurations. Unlike the single-pattern scenario, there is an improvement in performance with five layers compared to one layer, particularly with learned positional embeddings.

This phenomenon is related to the notion of induction heads , where at least two layers may be necessary to distinguish the two patterns [38]. This is reflected in Figure 2, which compares the accuracy trajectories of one-layer and five-layer models. While the five-layer setup effectively differentiates the two patterns, the one-layer configuration fails to do so. Meanwhile, in both clean and block-noisy scenarios, learned positional embeddings lead to notably higher accuracies as compared to sinusoidal ones, similar to the single-pattern case.

0.8

0.6

0.4

0.2

Patterm 1

Pattem 2

0.0

0

50

100

150

200

250

300

350

400

Epoch

Accuracy over epochs(1-layer)

0.600

Pattem 1

Pattem 2

0.575

0.550

2

0.525

ccurac

R

0.500

0.475

0.450

0.425

0

50

100

150

200

250

300

350

400

Epoch

4 Scenarios where in-context learning fails

In this section, we consider two scenarios where in-context learning (ICL) fails, irrespective of architectures. In Section 4.1, we consider a logic reasoning task requiring identification and generalization of a repetition meta-pattern within sequences. In Section 4.2, we explore a word analogy task where relevant word pairs appear in unstructured training sentences but are limited to fixed positions. Section 4.3 concludes with a synthetic data experiment supporting the theory.

4.1 Failed scenario 1: Sentences with repeating patterns

̸

̸

In this meta-pattern recognition and generalization task, each training sequence follows a repeating pattern based on its starting tokens, and the ICL task sequence requires the model to identify this repetition and extend it to a new, unseen starting pattern. Specifically, our training data comprises sentences in the form of abacdcefe , where a = b , c = d , and e = f . Note that each sentence is structured into three blocks, each consisting of three tokens with the same pattern. For the ICL task, we consider predicting f from the prompt abbcddef , where a = b , c = d , and e = f . Given the repeated pattern within each training sequence, a well-trained model might be expected to output f to continue the pattern established in the in-context examples: abb and cdd . However, as seen in Table 7, all models fail to recognize and apply the pattern, resulting in incorrect predictions.

̸

̸

̸

̸

Formalization. We now formalize a generalization of this scenario. Let the vocabulary be V = { 1 , 2 , · · · , | V |} , and define S = { ( a, b ) | a, b ∈ V , a = b } . To ensure training sentences are distinct from the ICL prompts, we first partition S into S 1 and S 2 , where { c [1] | c ∈ S 1 } = { c [1] | c ∈ S 2 } = V . Here, c [ i ] denotes the i -th element of c . Suppose we autoregressively train a sufficiently large position-aware model so that it is possible to achieve the minimum possible theoretical loss. The training sentences take the form x 11 x 12 x 11 x 21 x 22 x 21 · · · x N 1 x N 2 x N 1 , where x i 1 = x i 2 and ( x i 1 , x i 2 ) is independently selected from S 1 for every i ∈ [ N ] . Theorem 5, whose proof is in Appendix H, states that ICL fails regardless of the number of in-context examples.

̸

̸

Theorem 5 (Failure of ICL: Different repeated patterns) . Consider the generalized scenario in Section 4.1. For any 1 ≤ ℓ ≤ N , given an in-context prompt of the form x 11 x 12 x 12 x 21 x 22 x 22 · · · x ℓ 1 x ℓ 2 where x i 1 = x i 2 and ( x i 1 , x i 2 ) ∈ S 2 for every i ∈ [ ℓ ] , the model predicts x ℓ 1 instead of x ℓ 2 .

̸

Results. Theorem 5 and Table 7 demonstrate that ICL achieves zero accuracy irrespective of the number of in-context examples ( ℓ -1 ). This insight sheds light on the ICL capacity of autoregressive models. Simply put, if the pattern in the in-context examples differs significantly from any pattern in the training data, ICL may not occur. These results align with the findings of Raventós et al. [42] and Yadlowsky et al. [63] on the importance of data diversity for ICL.

4.2 Failed scenario 2: Sentences with co-occurring word pairs restricted to fixed locations

We revisit the word analogy task in Section 2. The training data now comprises sentences of the form of a i pqrsb i , where ( a i , b i ) represents a frequently co-occurring word pair and p, q, r, s represent other words. For the ICL task, we consider predicting b i 3 from the prompt a i 1 b i 1 a i 2 b i 2 a i 3 , where

Table 7: ICL in failed scenarios, averaged over 10 repetitions, achieves zero accuracy for any architecture and embedding dimension ( d E ).

i 1 , i 2 , i 3 are distinct. As each training sentence always contains an ( a i , b i ) pair at a fixed location, we expect a well-trained model to output b i 3 to maintain the pattern in in-context examples: a i 1 b i 1 and a i 2 b i 2 . Yet Table 7 shows none of the models can identify the patterns and predict the correct token.

Formalization. We now formalize a generalization of this scenario. Let the vocabulary be { ( a i , b i ) } i ∈ [ I ] ∪ V , where V = { 1 , 2 , · · · , | V |} represent other words. As in Section 4.1, we autoregressively train a sufficiently large position-aware model that can achieve the minimum possible theoretical loss. The training sentences take the form a i v 1 v 2 · · · v 2 k b i , where i and v 1:2 k are independently chosen from [ I ] and V , respectively, uniformly at random. Theorem 6, whose proof is in Appendix I, states that ICL fails regardless of the number of in-context examples.

Theorem 6 (Failure of ICL: Different pattern structures) . Consider the generalized scenario in Section 4.2. For any 1 ≤ ℓ ≤ k + 1 , given an in-context prompt of the form a i 1 b i 1 a i 2 b i 2 · · · a i ℓ with distinct i j 's, the model never predicts b i ℓ : it predicts a uniform probability vector over V when 1 ≤ ℓ ≤ k , and b i 1 when ℓ = k +1 .

Results. Theorem 6 highlights the finding that the success of ICL relies heavily on how the patterns appear in the training data. In this scenario, the ( a i , b i ) pairs consistently appear at the beginning and end of each training sentence, and we anticipate the model to recognize this relationship for ICL to occur. However, as shown in Theorem 6 and Table 7, this is not the case.

4.3 Experiment on a synthetic corpus

We conduct an experiment on a synthetic corpus featuring (country)-(capital) relationships. Each sentence falls into one of four categories: (1) exactly one country-capital pair, (2) exactly two country-capital pairs, (3) a single country without a pair, and (4) no country. In sentences with one country-capital pair, the capital appears in the first position, the country in the last, and each sentence contains six words (as in Section 4.2). The corpus generation process is detailed in Appendix K.

We train a five-layer two-head autoregressive transformer on this corpus, with an embedding dimension of 100 . Similar to Section 2.4, we evaluate the ICL accuracies using prompts involving countries and their capitals. The results show zero ICL accuracy across varying in-context examples (one to five), supporting our theory.

5 Discussion

This paper examines how in-context learning (ICL) arises from pre-training on unstructured language data, with three key findings: (1) ICL for word analogy tasks can emerge from simple co-occurrence modeling, using models like continuous bag of words (CBOW) without positional encoding or attention; (2) positional information and structured nuisance tokens are essential for ICL in logic reasoning tasks that require recognizing rare patterns and generalizing to new tokens; and (3) the structure of training data significantly impacts ICL effectiveness.

Acknowledgements. This work was supported in part by the Office of Naval Research under grant number N00014-23-1-2590, the National Science Foundation under grant numbers 2231174 and 2310831, No. 2428059, and a Michigan Institute for Data Science Propelling Original Data Science (PODS) grant.

References

M. Abbas, Y. Zhou, P. Ram, N. Baracaldo, H. Samulowitz, T. Salonidis, and T. Chen. Enhancing in-context learning via linear probe calibration. In Artificial Intelligence and Statistics , 2024.

J. Abernethy, A. Agarwal, T. V. Marinov, and M. K. Warmuth. A mechanism for sample-efficient in-context learning for sparse retrieval tasks. In Algorithmic Learning Theory , 2024.

K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Neural Information Processing Systems , 2024.

K. Ahuja and D. Lopez-Paz. A closer look at in-context learning under distribution shifts. In Workshop on Efficient Systems for Foundation Models at ICML , 2023.

E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations , 2022.

E. Akyürek, B. Wang, Y. Kim, and J. Andreas. In-context language learning: Architectures and algorithms. arXiv preprint arXiv:2401.12973 , 2024.

Y. Bai, F. Chen, H. Wang, C. Xiong, and S. Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Neural Information Processing Systems , 2023.

S. Bhattamishra, A. Patel, P. Blunsom, and V. Kanade. Understanding in-context learning in transformers and LLMs by learning to learn discrete functions. In International Conference on Learning Representations , 2023.

A. Bietti, V. Cabannes, D. Bouchacourt, H. Jegou, and L. Bottou. Birth of a transformer: A memory viewpoint. In Neural Information Processing Systems , 2023.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Neural Information Processing Systems , 2020.

S. C. Chan, A. Santoro, A. K. Lampinen, J. X. Wang, A. K. Singh, P. H. Richemond, J. McClelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers. In Neural Information Processing Systems , 2022.

S. Chen, H. Sheen, T. Wang, and Z. Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint arXiv:2402.19442 , 2024.

Y. Chen, C. Zhao, Z. Yu, K. McKeown, and H. He. Parallel structures in pre-training data yield in-context learning. arXiv preprint arXiv:2402.12530 , 2024.

T.-R. Chiang and D. Yogatama. Understanding in-context learning with a pelican soup framework. arXiv preprint arXiv:2402.10424 , 2024.

L. Collins, A. Parulekar, A. Mokhtari, S. Sanghavi, and S. Shakkottai. In-context learning with transformers: Softmax attention adapts to function Lipschitzness. arXiv preprint arXiv:2402.11639 , 2024.

Y. Cui, J. Ren, P. He, J. Tang, and Y. Xing. Superiority of multi-head attention in in-context linear regression. arXiv preprint arXiv:2401.17426 , 2024.

D. Dai, Y. Sun, L. Dong, Y. Hao, Z. Sui, and F. Wei. Why can GPT learn in-context? Language models secretly perform gradient descent as meta optimizers. In Association for Computational Linguistics , 2023.

S. Dalal and V. Misra. The matrix: A Bayesian learning model for LLMs. arXiv preprint arXiv:2402.03175 , 2024.

N. Ding, T. Levinboim, J. Wu, S. Goodman, and R. Soricut. CausalLM is not optimal for in-context learning. In International Conference on Learning Representations , 2024.

D. Fu, T.-Q. Chen, R. Jia, and V. Sharan. Transformers learn higher-order optimization methods for in-context learning: A study with linear models. In Workshop on Mathematics of Modern Machine Learning at NeurIPS , 2023.

S. Garg, D. Tsipras, P. S. Liang, and G. Valiant. What can transformers learn in-context? A case study of simple function classes. In Neural Information Processing Systems , 2022.

T. Guo, W. Hu, S. Mei, H. Wang, C. Xiong, S. Savarese, and Y. Bai. How do transformers learn in-context beyond simple functions? A case study on learning with representations. In International Conference on Learning Representations , 2023.

M. Hahn and N. Goyal. A theory of emergent in-context learning as implicit structure induction. arXiv preprint arXiv:2303.07971 , 2023.

C. Han, Z. Wang, H. Zhao, and H. Ji. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305.12766 , 2023.

X. Han, D. Simig, T. Mihaylov, Y. Tsvetkov, A. Celikyilmaz, and T. Wang. Understanding incontext learning via supportive pretraining data. In Association for Computational Linguistics , 2023.

A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy. Transformer language models without positional encodings still learn positional information. In Empirical Methods in Natural Language Processing , 2022.

Y. Huang, Y. Cheng, and Y. Liang. In-context convergence of transformers. In Workshop on Mathematics of Modern Machine Learning at NeurIPS , 2023.

H. J. Jeon, J. D. Lee, Q. Lei, and B. Van Roy. An information-theoretic analysis of in-context learning. arXiv preprint arXiv:2401.15530 , 2024.

D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

J. Kossen, Y. Gal, and T. Rainforth. In-context learning learns label relationships but is not conventional learning. In International Conference on Learning Representations , 2024.

S. Li, Z. Song, Y. Xia, T. Yu, and T. Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXiv:2304.13276 , 2023.

X. Li and X. Qiu. Finding support examples for in-context learning. In Empirical Methods in Natural Language Processing , 2023.

Y. Li, M. E. Ildiz, D. Papailiopoulos, and S. Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning , 2023.

Z. Lin and K. Lee. Dual operating modes of in-context learning. arXiv preprint arXiv:2402.18819 , 2024.

A. V. Mahankali, T. Hashimoto, and T. Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In International Conference on Learning Representations , 2023.

T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Empirical Methods in Natural Language Processing , 2022.

C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022.

M. Panwar, K. Ahuja, and N. Goyal. In-context learning through the Bayesian prism. In International Conference on Learning Representations , 2023.

K. Peng, L. Ding, Y. Yuan, X. Liu, M. Zhang, Y. Ouyang, and D. Tao. Revisiting demonstration selection strategies in in-context learning. arXiv preprint arXiv:2401.12087 , 2024.

C. Qin, A. Zhang, A. Dagar, and W. Ye. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881 , 2023.

A. Raventós, M. Paul, F. Chen, and S. Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. In Neural Information Processing Systems , 2023.

J. Ren, Q. Guo, H. Yan, D. Liu, X. Qiu, and D. Lin. Identifying semantic induction heads to understand in-context learning. arXiv preprint arXiv:2402.13055 , 2024.

R. Ren and Y. Liu. In-context learning with transformer is really equivalent to a contrastive learning pattern. arXiv preprint arXiv:2310.13220 , 2023.

M. E. Sander, R. Giryes, T. Suzuki, M. Blondel, and G. Peyré. How do transformers perform in-context autoregressive learning? arXiv preprint arXiv:2402.05787 , 2024.

L. Shen, A. Mishra, and D. Khashabi. Do pretrained transformers really learn in-context by gradient descent? arXiv preprint arXiv:2310.08540 , 2023.

A. Singh, S. Chan, T. Moskovitz, E. Grant, A. Saxe, and F. Hill. The transient nature of emergent in-context learning in transformers. In Neural Information Processing Systems , 2023.

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 2024.

S. Swaminathan, A. Dedieu, R. Vasudeva Raju, M. Shanahan, M. Lazaro-Gredilla, and D. George. Schema-learning and rebinding as mechanisms of in-context learning and emergence. In Neural Information Processing Systems , 2023.

E. Todd, M. Li, A. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In International Conference on Learning Representations , 2024.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

M.-H. Van, X. Wu, et al. In-context learning demonstration selection via influence analysis. arXiv preprint arXiv:2402.11750 , 2024.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Neural Information Processing Systems , 2017.

M. Vladymyrov, J. von Oswald, M. Sandler, and R. Ge. Linear transformers are versatile in-context learners. arXiv preprint arXiv:2402.14180 , 2024.

J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , 2023.

X. Wang, W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Neural Information Processing Systems , 2023.

K. C. Wibisono and Y. Wang. On the role of unstructured training data in transformers' incontext learning capabilities. In Workshop on Mathematics of Modern Machine Learning at NeurIPS , 2023.

N. Wies, Y. Levine, and A. Shashua. The learnability of in-context learning. In Neural Information Processing Systems , 2023.

J. Wu, D. Zou, Z. Chen, V. Braverman, Q. Gu, and P. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In International Conference on Learning Representations , 2023.

S. M. Xie, A. Raghunathan, P. Liang, and T. Ma. An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations , 2021.

Y. Xing, X. Lin, N. Suh, Q. Song, and G. Cheng. Benefits of transformer: In-context learning in linear regression tasks with unstructured data. arXiv preprint arXiv:2402.00743 , 2024.

Z. Xu, Z. Shi, and Y. Liang. Do large language models have compositional ability? An investigation into limitations and scalability. In Workshop on Mathematical and Empirical Understanding of Foundation Models at ICLR , 2024.

S. Yadlowsky, L. Doshi, and N. Tripuraneni. Pretraining data mixtures enable narrow model selection capabilities in transformer models. arXiv preprint arXiv:2311.00871 , 2023.

J. Yan, J. Xu, C. Song, C. Wu, Y. Li, and Y. Zhang. Understanding in-context learning from repetitions. In International Conference on Learning Representations , 2023.

Z. Yu and S. Ananiadou. How do large language models learn in-context? Query and key matrices of in-context heads are two towers for metric learning. arXiv preprint arXiv:2402.02872 , 2024.

R. Zhang, S. Frei, and P. L. Bartlett. Trained transformers learn linear models in-context. Journal of Machine Learning Research , 2024.

Y. Zhang, F. Zhang, Z. Yang, and Z. Wang. What and how does in-context learning learn? Bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305.19420 , 2023.

Y. Zhao, Y. Sakai, and N. Inoue. NoisyICL: A little noise in model parameters calibrates in-context learning. arXiv preprint arXiv:2402.05515 , 2024.

A Related work

Large language models (LLMs), such as transformers, are widely recognized for their outstanding performance in in-context learning (ICL) [10]. ICL refers to the capability of LLMs to discern specific tasks and generate predictions based on prompt exemplars without needing any parameter updates. A multitude of studies have been dedicated to exploring this intriguing phenomenon from various theoretical and empirical perspectives. In this section, we provide a brief summary of some of these studies.

Some studies adopted a Bayesian approach to studying ICL. Xie et al. [60] posited that ICL can be viewed as implicit Bayesian inference. They demonstrated that LLMs can infer a latent documentlevel concept for next-token prediction during pre-training and a shared latent concept across inputoutput pairs in an ICL prompt, under the assumption that documents are generated from hidden Markov models (HMMs). Wang et al. [56] and Zhang et al. [67] expanded on this idea by exploring more realistic latent variable models beyond HMMs. Wang et al. [56] argued that large language models function as latent variable models, with latent variables containing task-related information being implicitly inferred. Zhang et al. [67] showed that without updating the neural network parameters, ICL can be interpreted as Bayesian model averaging parameterized by the attention mechanism. Panwar et al. [39] provided empirical evidence that transformers behave like Bayesian predictors when performing ICL with linear and non-linear function classes. Dalal and Misra [18] proposed a Bayesian learning framework to understand ICL through the lens of text generation models represented by multinomial transition probability matrices. Chiang and Yogatama [14] proposed the pelican soup framework to explain ICL without relying on latent variable models. This framework incorporates concepts such as a common sense knowledge base, natural language classification, and meaning association, enabling the establishment of a loss bound for ICL that depends on the number of in-context examples.

Garg et al. [21] formulated ICL as learning a specific function class F from prompts of the form ( x 1 , f ( x 1 ) , . . . , x n , f ( x n ) , x n +1 ) and their corresponding responses f ( x n +1 ) . Here, f ∈ F , where F is a function class. In this context, ICL refers to the capability of a transformer to output a number close to g ( y n +1 ) given a prompt of the form ( y 1 , g ( y 1 ) , . . . , y n , g ( x n ) , y n +1 ) , where g ∈ F . Many studies adopted this regression formulation of ICL, with some linking ICL to gradient descent. Akyürek et al. [5], Von Oswald et al. [55], and Dai et al. [17] proved that transformers are capable of implementing gradient descent, which results in their ICL ability. Bai et al. [7] established generalization bounds for ICL and proved that transformers can perform algorithm selection like statisticians. Zhang et al. [66] showed that the gradient flow dynamics of transformers converge to a global minimum that enables ICL. Huang et al. [27] investigated the learning dynamics of single-layer softmax transformers trained via gradient descent to perform ICL on linear functions. Ahn et al. [3] explored the optimization landscape of transformers and proved that the optimal parameters coincide with an iteration of preconditioned gradient descent.

In a related exploration, Li et al. [31] showed that softmax regression models learned through gradient descent are similar to transformers. Ren and Liu [44] related ICL with softmax transformers to contrastive learning, where the inference process of ICL can be viewed as a form of gradient descent. Mahankali et al. [35] proved that minimizing the pre-training loss is equivalent to a step of gradient descent in single-layer linear transformers. Vladymyrov et al. [54] established that linear transformers execute a variant of preconditioned gradient descent by maintaining implicit linear models. On the other hand, some studies argued that the ICL ability of transformers cannot be attributed to gradient descent. Fu et al. [20] showed that ICL for linear regression tasks arises from higher-order optimization techniques like iterative Newton's method rather than gradient descent. Wibisono and Wang [57] demonstrated that transformers can perform ICL on unstructured data whose prompt exemplars lack explicit pairings, with softmax attention playing an important role especially when using a single attention layer. Shen et al. [46] provided empirical evidence that the equivalence between gradient descent and ICL might not be applicable in real-world scenarios. In contrast to these studies, our work provides a connection between ICL and classical language models like continuous bag of words (CBOW). Specifically, we show that ICL for word analogy tasks with semantically related word pairs can arise by modeling co-occurrence patterns via CBOW.

Supplementary Material

Numerous studies focused on the pre-training aspects (e.g., data distribution and task diversity) of ICL. Min et al. [37] showed that the input-label mapping in the in-context examples does not significantly affect ICL performance. Chan et al. [11] demonstrated that the ICL capabilities of transformers depend on the training data distributions and model features. Kossen et al. [30] established that ICL considers in-context label information and is capable of learning entirely new tasks in-context. Li and Qiu [32] introduced an iterative algorithm designed to enhance ICL performance by selecting a small set of informative examples that effectively characterize the ICL task. Qin et al. [41] proposed a method based on zero-shot chain-of-thought reasoning for selecting ICL examples, emphasizing the importance of choosing diverse examples that are strongly correlated with the test sample. Han et al. [25] studied ICL by identifying a small subset of the pre-training data that support ICL via gradient-based methods. They discovered that this supportive pre-training data typically consist of more uncommon tokens and challenging examples, characterized by a small information gain from long-range context. Peng et al. [40] proposed a selection method for ICL demonstrations that are both data-dependent and model-dependent. Van et al. [52] introduced a demonstration selection method that enhances ICL performance by analyzing the influences of training samples using influence functions.

In a similar vein, Wu et al. [59] demonstrated that pre-training single-layer linear attention models for ICL on linear regression with a Gaussian prior can be effectively accomplished with a minimal number of independent tasks, regardless of task dimension. Raventós et al. [42] emphasized a task diversity threshold that differentiates the conditions under which transformers can successfully address unseen tasks. Yadlowsky et al. [63] attributed the impressive ICL capabilities of transformers to the diversity and range of data mixtures in their pre-training, rather than their inductive biases for generalizing to new tasks. Ding et al. [19] compared the ICL performance of transformers trained with prefixLM (where in-context samples can attend to all tokens) versus causalLM (where in-context samples cannot attend to subsequent tokens), finding that the latter resulted in poorer ICL performance. Chen et al. [13] discovered that the ICL capabilities of language models rely on the presence of pairs of phrases with similar structures within the same sentence. Zhao et al. [68] proposed a calibration scheme that modifies model parameters by adding random noises, resulting in fairer and more confident predictions. Abbas et al. [1] demonstrated that the ICL predictions from transformer-based models often exhibit low confidence, as indicated by high Shannon entropy. To address this issue, they introduced a straightforward method that linearly calibrates output probabilities, independent of the model's weights or architecture. Similar to these works, our work highlights the importance of co-occurrence, positional information, and training data structure for ICL to arise.

Other studies analyzed ICL from a learning theory perspective. Hahn and Goyal [23] proposed an information-theoretic bound that explains how ICL emerges from next-token prediction. Wies et al. [58] derived a PAC-type framework for ICL and finite-sample complexity results. Jeon et al. [28] introduced a novel information-theoretic view of meta-learning (including ICL), allowing for the decomposition of errors into three components. They proved that in ICL, the errors decrease as the number of examples or sequence length increase. Other studies focus on the mechanistic interpretability component of ICL. Olsson et al. [38] argued that transformers can develop induction heads that are able to complete token sequences such as [A][B] · · · [A] → [B], leading to impressive ICL performance. Bietti et al. [9] examined a setup where tokens are generated from either global or context-specific bigram distributions to distinguish between global and in-context learning. They found that global learning occurs rapidly, while in-context learning is achieved gradually through the development of an induction head. Ren et al. [43] identified semantic induction heads that increase the output logits of tail tokens when attending to head tokens, providing evidence that these heads could play a vital role in the emergence of ICL. Yu and Ananiadou [65] showed that the ICL ability of transformers arises from the utilization of in-context heads, where each query and key matrix collaborate to learn the similarity between the input text and each demonstration example.

Anumber of works delved into specific data generating processes to provide insight into the emergence of ICL. Bhattamishra et al. [8] examined the ICL ability of transformers by focusing on discrete functions. Specifically, they showed that transformers perform well on simpler tasks, struggle with more complex tasks, and can learn more efficiently when provided with examples that uniquely identify a task. Guo et al. [22] investigated ICL in scenarios where each label is influenced by the input through a potentially complex yet constant representation function, coupled with a unique linear function for each instance. Akyürek et al. [6] studied ICL of regular languages produced by random finite automata. They compared numerous neural sequence models and demonstrated that

transformers significantly outperform RNN-based models because of their ability to develop n-gram heads , which are a generalization of induction heads . Sander et al. [45] analyzed simple first-order autoregressive processes to gain insight into how transformers perform ICL to predict the next tokens. On the other hand, our work focuses on data generating processes containing semantically related word pairs and repeated token patterns to better understand several components that are crucial for ICL to occur from training on unstructured data.

Some studies explored how different components of transformers affect their ICL abilities. Ahuja and Lopez-Paz [4] compared the ICL performance of transformers and MLP-based architectures under distribution shifts. Their findings demonstrate that while both methods perform well in in-distribution ICL, transformers exhibit superior ICL performance when faced with mild distribution shifts. Collins et al. [15] showed that softmax attention outperforms linear attention in ICL due to its ability to calibrate its attention window to the Lipschitzness of the pre-training tasks. Xing et al. [61] focused on linear regression tasks to identify transformer components that enable ICL. They found that positional encoding is crucial, along with the use of multiple heads, multiple layers, and larger input dimensions. Cui et al. [16] proved that multi-head attention outperforms single-head attention in various practical scenarios, including those with noisy labels and correlated features. Chen et al. [12] investigated the ICL dynamics of a multi-head softmax attention model applied to multi-task linear regression. They proved the convergence of the gradient flow and observed the emergence of a task allocation phenomenon, where each attention head specializes in a specific task.

Finally, several studies proposed various hypotheses on the emergence of ICL and provided theoretical justifications. Swaminathan et al. [49] introduced clone-structured causal graphs (CSCGs) to explain how ICL can generalize to unseen sentences via a mechanism called rebinding. Li et al. [33] viewed ICL as an algorithm learning problem where a transformer implicitly constructs a hypothesis function at inference time. Han et al. [24] argued that the ability of transformers to execute ICL is attributable to their capacity to simulate kernel regression. Singh et al. [47] explored the interaction between ICL and in-weights learning (IWL) using synthetic data designed to support both processes. They observed that ICL initially emerges, followed by a transient phase where it disappears and gives rise to IWL. Yan et al. [64] studied ICL from the perspective that token co-occurrences play a crucial role in guiding the learning of surface patterns that facilitates ICL. Abernethy et al. [2] showed that transformers can execute ICL by dividing a prompt into examples and labels, then employing sparse linear regression to deduce input-output relationships and generate predictions. Lin and Lee [34] developed a probabilistic model that can simultaneously explain both task learning and task retrieval aspects of ICL. Here, task learning refers to the ability of language models to identify a task from in-context examples, while task retrieval pertains to their ability to locate the relevant task within the pre-training data.

B Proof of Theorem 1

Proof. Let | V | = 2 K + L denote the vocabulary size. Consider a sentence X represented by its one-hot encoding (i.e., X ∈ { 0 , 1 } | V |× S ). For every position i ∈ [ S ] , the loss for predicting the word in the i -th position given all the other words is given by || AX ( 1 S -e i ) -Xe i || 2 2 , where A = U ⊤ V S -1 ∈ R | V |×| V | and e i ∈ R S is a zero vector with 1 on its i -th entry. Here, U ( V ) is a matrix consisting of the center (context) embeddings of all tokens, and A is a matrix summarizing the similarity between each pair of words (one as a center word and the other as a context word). Our objective is to find A that minimizes the sum of losses for each position in each sentence. Lemma 1 gives a closed-form expression of the minimizer.

Lemma 1. The minimizer of the overall loss is given by A = B (( S -2) B + C ) -1 . Here, B is a matrix whose ( i, j ) -th entry is p ( i, j ) , the probability that for a given (center, context) pair, the center is i ∈ | V | and the context is j ∈ | V | . Moreover, C is a diagonal matrix whose i -th diagonal entry is p ( i ) = ∑ j ∈| V | p ( i, j ) .

Proof. Let L ( X ) = ∑ S i =1 || AX ( 1 S -e i ) -Xe i || 2 2 denote the sum of the losses corresponding to all tokens in sentence X . By direct calculation,

.

Note that ∑ S i =1 ( 1 S -e i )( 1 S -e i ) ⊤ = ( S -2) 1 S × S + I S × S and ∑ S i =1 e i ( 1 S -e i ) ⊤ = 1 S × S -I S × S . Now, let our sentences be X 1 , X 2 , · · · , X N . The minimizer of the overall loss thus satisfies

We denote the number of (center, context) pairs across all sentences in which the center is i and the context is j by #( i, j ) . Moreover, we define #( i ) = ∑ j ∈| V | #( i, j ) . It is easy to see that Equation (1) can be rewritten as

where ˜ B is a matrix such that its ( i, j ) -th entry is #( i,j ) N and ˜ C is a diagonal matrix such that its i -th diagonal element is #( i ) N . As N → ∞ , an application of the law of large numbers yields #( i,j ) N → S ( S -1) p ( i, j ) almost surely and #( i ) N → S ( S -1) p ( i ) almost surely, where p ( i, j ) is the probability that for a given (center, context) pair, the center is i and the context is j , and p ( i ) = ∑ j ∈| V | p ( i, j ) .

Thus, as N →∞ , we have

where B and C are defined in the statement of Lemma 1.

We now define

p 1 = p ( c i , c j ) = p ( d i , d j ) = p ( c i , d j ) = p ( d i , c j ) for any i = j ;

̸

p 2 = p ( r i , r j ) for any i = j ;

p 3 = p ( c i , d i ) = p ( d i , c i ) for any i ;

p 4 = p ( c i , r j ) = p ( d i , r j ) = p ( r j , c i ) = p ( r j , d i ) for any i, j ,

̸

where the equalities in the probabilities are a consequence of the data distribution.

For ease of presentation, we denote a square matrix with α on the diagonal and β off the diagonal as X α,β , and a matrix with all entries γ as Y γ . We then have

Now, define a = ( S -2) p 1 , b = ( S -2) p 2 , c = ( S -2) p 3 , d = ( S -2) p 4 , e = 2( K -1) p 1 + p 3 + Lp 4 , and f = ( L -1) p 2 +2 Kp 4 . It is easy to see that

Moreover, its inverse can be written as

where

By computing A = B (( S -2) B + C ) -1 , given the following center words, the similarities between them and all possible context words are as follows:

Center word = c i for any i

Center word = d i for any i

̸

̸

̸

̸

Center word = r i

-c j : 2( K -1) p 4 q 1 + p 4 q 5 + p 4 q 3 +( L -1) p 2 q 4 (for any j ); -d j : 2( K -1) p 4 q 1 + p 4 q 5 + p 4 q 3 +( L -1) p 2 q 4 (for any j ); -r i : 2 Kp 4 q 4 +( L -1) p 2 q 2 ; -r j : 2 Kp 4 q 4 +( L -2) p 2 q 2 + p 2 q 6 ( j = i ).

̸

Recall that the ICL problem of interest is the following: given context words c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 , we aim to predict d i ℓ +1 . Without loss of generality, we can rewrite the problem to predict d ℓ +1 given context words c 1 d 1 · · · c ℓ d ℓ c ℓ +1 . We now compute the total similarity for each possible center word, where ϵ ⊤ δ indicates the similarity between the word ϵ in the center and the word δ in the context.

c 1 (or any of c 2 , · · · , c ℓ ) : c ⊤ 1 c 1 + ℓc ⊤ 1 c 2 + c ⊤ 1 d 1 +( ℓ -1) c ⊤ 1 d 2 ;

d 1 (or any of d 2 , · · · , d ℓ ) : c ⊤ 1 d 1 + ℓc ⊤ 1 d 2 + c ⊤ 1 c 1 +( ℓ -1) c ⊤ 1 c 2 ;

r 1 (or any other r k 's) : ( ℓ +1) r ⊤ 1 c 1 + ℓr ⊤ 1 d 1 = (2 ℓ +1) r ⊤ 1 c 1 ;

c ℓ +1 : ℓc ⊤ 1 c 2 + ℓc ⊤ 1 d 2 + c ⊤ 1 c 1 ;

d ℓ +1 : ℓc ⊤ 1 d 2 + ℓc ⊤ 1 c 2 + c ⊤ 1 d 1 ;

c ℓ +2 (or any c k 's not in the context prompt) : ( ℓ +1) c ⊤ 1 c 2 + ℓc ⊤ 1 d 2 ;

d ℓ +2 (or any d k 's not in the context prompt) : ( ℓ +1) c ⊤ 1 d 2 + ℓc ⊤ 1 c 2 .

Note that correctly predicting d ℓ +1 is equivalent to the following conditions being simultaneously satisfied:

c ⊤ 1 d 1 > c ⊤ 1 c 1 , equivalent to p 3 q 5 > p 3 q 3 ;

c ⊤ 1 d 2 > c ⊤ 1 c 1 and c ⊤ 1 c 2 > c ⊤ 1 c 1 , equivalent to p 1 q 3 + p 3 q 1 + p 1 q 5 > 2 p 1 q 1 + p 3 q 3 ;

c ⊤ 1 d 1 > c ⊤ 1 c 2 and c ⊤ 1 d 1 > c ⊤ 1 d 2 , equivalent to 2 p 1 q 1 + p 3 q 5 ≥ p 1 q 5 + p 1 q 3 + p 3 q 1 ;

2 ℓc ⊤ 1 c 2 + c ⊤ 1 d 1 > (2 ℓ +1) r ⊤ 1 c 1 , equivalent to 2 ℓ (2( K -2) p 1 q 1 + p 1 q 5 + p 3 q 1 + p 1 q 3 + Lp 4 q 4 ) + 2( K -1) p 1 q 1 + p 3 q 5 + Lp 4 q 4 > (2 ℓ +1)(2( K -1) p 4 q 1 + p 4 q 5 + p 4 q 3 +( L -1) p 2 q 4 ) ;

In our data generating process, it is easy to see that p 1 = 0 , p 2 = ( S -2)( S -3) L ( L -1) , p 3 = 1 K , and p 4 = S -2 KL , where each p i is multiplied by a constant S ( S -1) > 0 (without loss of generalization) to make calculations easier. From here, we have a = 0 , b = ( S -2) 2 ( S -3) L ( L -1) , c = S -2 K , d = ( S -2) 2 KL , e = S -1 K , and f = ( S -1)( S -2) L . Substituting to the above, we have

We now check when these conditions are simultaneously satisfied. The first condition is equivalent to p 3 > 0 and K > 2( S -2) 3 ( S -1) 2 (2 S -3) , which always hold. The second condition reduces to p 3 > 0 and 2( S -2) 4 + K ( S -2) 2 ( S -1) 2 > 0 , which is also true. The third condition can be written as p 3 > 0 and K ( S -2)( S -1) 3 > 0 , which always hold. The last condition becomes

which is equivalent to

completing the proof.

Note that this condition ensures that the model predicts d ℓ +1 instead of one of the r i 's.

C Comparison of ICL performance using squared and cross-entropy loss across different numbers of examples

From Tables 8 and 9, we observe that ICL with CBOW on single-relationship tasks performs better with squared loss compared to cross-entropy loss and with fewer demonstration examples. Also, ICL tends to deteriorate after a certain number of in-context demonstrations. As detailed in Appendix B, a smaller number of examples (e.g., zero) allows the model to produce the correct output instead of one of the r i 's. This is in contrast with transformer-based LLMs, which achieve better ICL performance as the number of demonstrations increases. On the other hand, ICL on dual-relationship tasks as described in Section 2.2 requires at least one demonstration example to distinguish between the two tasks.

D Proof of Theorem 2

Proof. We show that given a prompt of the form c i 1 d i 1 · · · c i ℓ d i ℓ c i ℓ +1 with distinct i k 's, a trained CBOW model is more likely to predict d i ℓ +1 than e i ℓ +1 . If this is established, the other part of the theorem follows analogously. We now define

̸

̸

p 2 = p ( r i , r j ) for any i = j ;

where the equalities in the probabilities are a consequence of the data distribution. By direct calculation, we have p 1 = 1 K ( K -1) , p 2 = ( S -4)( S -5) L ( L -1) , p 3 = 1 K , and p 4 = S -4 KL , where each p i is multiplied by S ( S -1) > 0 (without loss of generalization) to make calculations easier. Moreover, it is easy to see that p ( c i , r j ) = p ( r i , c j ) = 2 p 4 for any i, j and p ( c i , c j ) = 2 p 1 for any i = j . Lastly, we define a = ( S -2) p 1 , b = ( S -2) p 2 , c = ( S -2) p 3 , d = ( S -2) p 4 , e = 2( K -1) p 1 + p 3 + Lp 4 , and f = 4 Kp 4 +( L -1) p 2 .

̸

The next step the proof is to use Lemma 1 in Appendix B to obtain the similarity matrix A . As previously, we denote a square matrix with α on the diagonal and β off the diagonal as X α,β , and a matrix with all entries γ as Y γ . We then have

and

Moreover, its inverse can be written as

for some q 1 , q 2 , · · · , q 10 . Recall that our task is show that given context words c i 1 , d i 1 , · · · , c i ℓ , d i ℓ c i ℓ +1 with distinct i k 's, the center word is more likely to be d i ℓ +1 than e i ℓ +1 . In other words, we need to establish that

̸

where ϵ ⊤ δ indicates the similarity between the word ϵ in the center and the word δ in the context. This similarity can be obtained from the matrix A = B (( S -2) B + C ) -1 . By symmetry, the inequality reduces to d ⊤ i d j > e ⊤ i d j for any i = j .

By computing the matrix A , we have

and

Thus, our problem again reduces to showing ( K -2) q 6 + q 5 > ( K -2) q 8 + q 7 as p 1 = 1 K ( K -1) > 0 . Upon multiplying (3) and (2) and equating the result with the identity matrix, we have the following equations:

Comparing (5) and (7) yields

As a = ( S -2) p 1 > 0 and e = 2 p 1 ( K -1)+ p 3 + p 4 L > 0 , we now only need to show that q 8 > q 6 . Comparing (4) and (6) as well as (5) and (7), we have

which reduces to ( q 8 -q 6 )( e 2 + ae ( K -2) -a 2 ( K -1)) = a. The conclusion follows since a > 0 and

E Proof of Theorem 3

Proof. Consider the instance of predicting a from abc , i.e., f ( { a, b } , c ) . By the assumption on the data distribution, it is equally likely that the task is predicting b from bac . In this case, the corresponding function is also f ( { a, b } , c ) . Thus, the sum of the cross-entropy losses corresponding to these two tasks is lower bounded by 2 log 2 > 0 . Also, it is easy to see that we cannot achieve perfect accuracy since the predictions for abc and bac must be the same.

̸

We now show that it is possible to attain zero loss and perfect accuracy when the model includes positional embeddings, so that ˜ f ( { ( a, 1) , ( b, 2) } , ( c, 3)) = ˜ f ( { ( b, 1) , ( a, 2) } , ( c, 3)) . As a special case, we consider a simplified version of the transformer architecture, where

and

for any token d . Here, x i and p j represent the embedding of token i and position j , respectively.

̸

Let p ⊤ 1 p 3 = p , p ⊤ 2 p 3 = q , p ⊤ 3 p 3 = r , x ⊤ i x i = s , x ⊤ i x j = t for any i = j , p ⊤ 1 x i = u for any i , p ⊤ 2 x i = v for any i , and p ⊤ 3 x i = w for any i . Note that this holds due to the assumed data generating process. We consider the following construction: p 1 = b 1 | V | , p 2 = p 3 = 1 | V | , and x i = ae i , where e i is a zero vector with 1 on the i -th entry. This implies p = b | V | , q = r = | V | , s = a 2 , t = 0 , u = ab , and v = w = a .

By direct calculation, the cross-entropy loss of predicting a from abc is given by

where

Letting b = a 2 and a →∞ , it is easy to see that we can bring the cross-entropy loss arbitrarily close to zero. Consequently, we also have a perfect prediction accuracy.

F Proof of Proposition 1

Proof. The intermediate representation of the first layer is given by f 1 ( { x i 1 } ) , f 2 ( { x i 1 } , x i 2 ) } , f 3 ( { x i 1 , x i 2 } , x i 3 ) , and f 4 ( { x i 1 , x i 2 , x i 3 } , x i 1 ) , for some functions f 1 , f 2 , f 3 , and f 4 . To predict the last x i 1 , we use the third coordinate of the second layer representation, which is given by t ( x i 1 x i 2 x i 3 ) := g 3 ( { f 1 ( { x i 1 } ) , f 2 ( { x i 1 } , x i 2 ) } , f 3 ( { x i 1 , x i 2 } , x i 3 )) , for some function g 3 . It is easy to see that in general, t ( x i 1 x i 2 x i 3 ) = t ( x i 2 x i 1 x i 3 ) .

̸

G Proof of Theorem 4

Proof. In the one-noisy scenario, each sentence takes one of the following forms: nabacdc , anbacdc , abnacdc , abancdc , abacndc , and abacdnc , where n ∈ N . In order to achieve the minimum possible theoretical loss, we minimize each loss term separately. Concretely, the minimum loss of predicting the sixth token given the first five tokens is attained by the following rule:

When the first five tokens do not contain any nuisance token, output a uniform probability vector over N .

Otherwise, output the conditional probability of c [2] given x , where ( x, c [2]) ∈ S 1 . Here, x represents the last non-nuisance token.

Under this rule, the predicted output for any in-context example abacd is never c , since c / ∈ N . In the block-noisy scenario, each sentence takes one of the following forms: n 1 n 2 n 3 abacdc , aban 1 n 2 n 3 cdc , and abacdcn 1 n 2 n 3 , where n 1 , n 2 , n 3 ∈ N . The minimum loss of predicting the ninth token given the first eight tokens is attained by the following rule:

When the seventh token is not a nuisance token, output the seventh token with probability one.

When the seventh token is a nuisance token, output a uniform probability vector over N .

Under this rule, the predicted output for any in-context example abacdcef is e , resulting in perfect ICL accuracy.

H Proof of Theorem 5

Proof. Recall that each training sentence is of the form x 11 x 12 x 11 x 21 x 22 x 21 · · · x N 1 x N 2 x N 1 . Note that we can decompose the total loss L into L 1 + L 2 + · · · + L 3 N , where L g denotes the loss of predicting the g -th token given all the other previous tokens. As the x i 1 x i 2 x i 1 blocks are generated independently, the optimal loss should satisfy L 1 = L 4 = · · · = L [3 N -2] = L [1] , L 2 = L 5 = · · · = L [3 N -1] = L [2] , and L 3 = L 6 = · · · = L [3 N ] = L [3] . Therefore, it is sufficient to minimize L [1] + L [2] + L [3] .

In order to achieve the minimum possible theoretical loss, we need to minimize L [1] , L [2] , and L [3] separately. It is easy to see that L [1] is minimized by outputting the marginal probability of c [1] , where c ∈ S 1 . Similarly, L [2] is minimized by outputting the conditional probability of c [2] given x i 1 , where ( x i 1 , c [2]) ∈ S 1 . On the other hand, it is possible to achieve an L [3] value of zero by outputting x i 1 with probability one.

Now, given an ICL prompt x 11 x 12 x 12 x 21 x 22 x 22 · · · x ℓ 1 x ℓ 2 where ℓ ≤ N , the trained model should predict x ℓ 1 with probability one since { c [1] | c ∈ S 2 } = V and our ICL prompt corresponds to L [3] . This completes the proof.

I Proof of Theorem 6

Proof. We proceed similarly as the proof of Theorem 5. Concretely, we separately minimize L g for g ∈ [2 k +2] , where L g denotes the loss of predicting the g -th token given all the other previous tokens. It is easy to see that L 1 is minimized by outputting a uniform probability vector over a 1:[ I ] , whereas L h (for any 2 ≤ h ≤ 2 k +1 ) is minimized by outputting a uniform probability vector over V . Moreover, it is possible to achieve an L 2 k +2 value of zero by outputting b i with probability one.

From here, given an ICL prompt of the form a i 1 b i 1 a i 2 b i 2 · · · a i ℓ , the trained model should predict a uniform probability vector over V if ℓ ≤ k , and b i 1 if ℓ = k +1 . In all cases, the model does not predict b i ℓ , completing the proof.

J Limitations and future work

This study has several limitations. Firstly, the experiments are conducted on a relatively small scale. However, they still provide sufficient evidence to support the theoretical findings. Secondly, the focus of this study is on specific types of in-context learning (ICL) tasks, as described in Section 1. Thirdly, the pre-training data considered in this work may not match the valid grammatical sentences that language models are usually trained on. Nonetheless, our co-occurrence results still apply to grammatical sentences, as the co-occurring pairs can appear naturally within them (e.g., "Beijing is the capital of China," or "the city of Beijing is located in China"). Lastly, real data sets are not utilized due to the lack of alignment with the study objectives.

Despite these limitations, we believe this work provides valuable understanding of the key factors enabling ICL to occur from training on unstructured natural language data, supported by both theoretical and empirical evidence from experiments involving prompting and synthetic data. Further analyses on other ICL tasks and their reliance on model architecture can be fruitful avenues for future work.

K Details of experiments and data sets

K.1 Architecture and implementation

All experiments utilize the Keras package in Python, employing the Adam optimizer [29] with a learning rate of 0.01. Early stopping is applied based on validation loss with a patience threshold of 5, utilizing a randomly selected subset representing 50% of the original data set. Each transformer layer uses two heads, as we empirically demonstrated that increasing the number of heads does not impact performance in our experiments. Each layer consists of the following components (in order): (1) Keras' multi-head causal self-attention block, with key_dim = value_dim = embed_dim / 2 ; (2) Skip connection and layer normalization; (3) One hidden layer feed-forward network using the ReLU activation with dimension = 2 × embed_dim; and (4) Skip connection and layer normalization.

K.2 Source and details of data sets

The world_population.csv data set, used for the experiments in Section 2, is obtained from Kaggle. According to the author, this data set is created from World Population Review.

The us-state-capitals.csv data set, used for the experiments in the beginning of Section 2, is obtained from this Github repository. Its source is unclear.

The uscities.csv data set, used for the experiments in the beginning of Section 2, is obtained from Simple Maps, with a CC 4.0 license.

K.3 Details of synthetic data used in experiments

Below we provide additional details regarding the synthetic data used in our experiments.

For experiments in Table 1, the training data consists of 50,000 sentences. In the clean version, sentences are generated uniformly as described in Section 2.1. In the corrupted version, sentences are generated in a similar manner, but each ( c i , d i ) pair is replaced by ( c i , r j ) or ( d i , r j ) with a probability of 1 / 4 each. Test sentences are generated according to the setup in Theorem 1. Some examples are as follows:

•

•

Clean -Training: c 1 d 1 r 1 r 2 r 3 r 4 r 5 r 6 or r 1 r 2 r 3 r 4 r 5 r 6 r 7 r 8 -Prompt: c 1 d 1 c 2 d 2 c 3 d 3 c 4 ? Corrupted -Training: c 1 r 1 r 2 r 3 r 4 r 5 r 6 r 7 or c 1 d 1 c 2 r 1 r 2 r 3 r 4 r 5 -Prompt: c 1 d 1 c 2 d 2 c 3 d 3 c 4 ?

For experiments in Table 2, the training data consists of 50,000 sentences. In the clean version, sentences are generated uniformly as described in Section 2.2. In the imbalanced and extreme versions, the 60 other words are divided into three categories: 20 for cd

sentences ( rcd · ), 20 for ce sentences ( rce · ), and 20 for both types ( r · ). In the imbalanced version, cd ( ce ) sentences are 4 times more likely to sample a cd ( ce ) word than a ce ( cd ) word. In the extreme version, cd ( ce ) sentences cannot contain any ce ( cd ) words. Test sentences are generated according to the setup in Theorem 2. Some examples are as follows:

Clean examples

-Training: c 1 d 1 r 1 r 2 r 3 r 4 r 5 r 6 or c 1 e 1 r 1 r 2 r 3 r 4 r 5 r 6

-Prompt: c 1 d 1 c 2 d 2 c 3 d 3 c 4 ? or c 1 e 1 c 2 e 2 c 3 e 3 c 4 ?

Imbalance examples

-Training: c 1 d 1 rcd 1 rcd 2 rcd 3 rce 4 r 5 r 6 or c 1 e 1 rcd 1 r 2 rce 3 rce 4 rce 5 r 6

-Prompt: c 1 d 1 c 2 d 2 c 3 d 3 c 4 ? or c 1 e 1 c 2 e 2 c 3 e 3 c 4 ?

Extreme examples

-Training: c 1 d 1 rcd 1 rcd 2 rcd 3 r 4 r 5 r 6 or c 1 e 1 r 1 r 2 r 3 rce 4 rce 5 r 6

-Prompt: c 1 d 1 c 2 d 2 c 3 d 3 c 4 ? or c 1 e 1 c 2 e 2 c 3 e 3 c 4 ?

Experiments in Table 3 follow the setup of experiments in Table 2, except that the pairs are now of the form ( c i , d i ) and ( e i , f i ) instead of ( c i , d i ) and ( c i , e i ) .

For experiments in Section 2.4, the corpus generation process is as follows:

Randomly select 10 countries and obtain their capital cities and IOC codes.

Generate 30 sentences containing exactly one country-capital pair (3 for each country). Example: Paramaribo is the vibrant heart of Suriname.

Generate 30 sentences containing exactly one country-IOC pair (3 for each country). Example: Gabon (GAB) protects its diverse rainforests and wildlife.

Generate 30 sentences containing exactly one country without any pair. Example: The banking sector is central to Liechtenstein's prosperity.

Generate 60 sentences without any country, capital city, or IOC code.

Example: Every country has its unique cultural identity and heritage.

Generate 810 sentences containing exactly two different country-capital pairs by concatenating sentences generated in Step 2.

Example: The city of Dushanbe reflects Tajikistan's vibrant spirit. Roseau is the cultural tapestry of Dominica.

Generate 810 sentences containing exactly two different country-IOC pairs by concatenating sentences generated in Step 3. Example: Mayotte (MAY) features lush landscapes and peaks. Turkmenistan (TKM) features the fiery Darvaza Crater.

The ICL prompts follow the form used in the country-capital city and US state-capital city experiments in the beginning of Section 2, with 1 to 5 in-context examples.

For experiments in Table 4, the training and test data consist of all sentences in the form abca , where a , b , and c are distinct. Each test sentence is different from any training sentence. In the first scenario (both), the first tokens of the training sentences cover the entire vocabulary. In the second scenario (either), each token can be the first token in either the training or test data, but not both.

For experiments in Table 5, the training data consists of 50,000 sentences generated uniformly as detailed in Section 3.1. The ICL prompt formats are also described in Section 3.1.

For experiments in Table 6, the training data consists of 50,000 sentences. In the clean scenario, the training data are of the form abcadefd and abcbdefe , with ICL prompts as abcadef ? and abcbdef ? . In the block-noisy scenario, the training data include sequences like n 1 n 2 n 3 n 4 abcadefd and abcbn 1 n 2 n 3 n 4 defe , with ICL prompts as abcadefdghi ? and abcbdefeghi ? .

For experiments in Table 7, the training data consists of 50,000 sentences generated uniformly according to the processes in Sections 4.1 and 4.2. The ICL prompt formats are also described in the same subsections.

For experiments in Section 4.3, the corpus generation process is as follows:

Randomly select 10 countries and obtain their capital cities and IOC codes.

Generate 130 sentences containing exactly one country-capital pair (13 for each country).

Example: Paramaribo stands as capital of Suriname.

Generate 30 sentences containing exactly one country without any pair. Example: The banking sector is central to Liechtenstein's prosperity.

Generate 60 sentences without any country, capital city, or IOC code. Example: Every country has its unique cultural identity and heritage.

Generate 1,000 sentences containing exactly two different country-capital pairs by concatenating sentences generated in Step 2.

Example: Brazil functions as heart of Brasilia. Turkmenistan operates as center for Ashgabat.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Abstract and introduction (Section 1) clearly summarize the paper's motivation (relevance), scope, and contributions. A summary of contributions at the bottom of Section 1 is provided to re-iterate the contributions of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: Limitations of this work are included in Appendix J.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: Assumptions of each theorem and lemma are clearly mentioned in the theorem/lemma statement. Correct and complete proofs of all theoretical results are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: Codes for each experiment are provided as a .zip file in the submission. Experimental details are also provided throughout the paper and in Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: Open access to all codes is provided (one folder for each result). File names are self-explanatory.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: Enough details are provided throughout the paper and in Appendix K. See code for more details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No] .

Justification: We only repeat each experiment 10 times due to computational constraints. However, most experimental results are backed up by theoretical analyses.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No] .

Justification: We do not include details on the type amd amount of compute required to run each experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: We have reviewed the NeurIPS Code of Ethics and concluded that this research conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: There is no societal impact as this work is foundational.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: We believe this paper poses no such risks as we do not train large-scale language models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: The existing models used in this paper (e.g., LLaMA) are properly credited. The sources of the data sets used in our experiments are mentioned in Appendix K.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not include any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper basically presents three theoretical analyses related to ICL. The section 2 shows that we can use CBOW to do the (country)-(capital) kind of ICL. The section 3 shows that positional embeddings, multiple layers in autoregressive LM, and blocked noise structures are important for ICL. The section 4 shows that ICL could fail when there are systematic and consistent mismatches between the training sequence and testing sequence.\n\nSTRENGTHS:\nI think this paper is easy to follow and most explanations are clear (One minor suggestion: it would be more clear to also illustrate the correct answer of each prompt and provide some brief explanations such as the prompt in section 3 tries to repeat the first letter of a word). I choose fair in the presentation rating because I feel that the paper oversells its contributions in the title and abstract.\n\nAll the claims are supported by both strong theoretical conclusions and empirical simulations. The theoretical contributions are novel to me but I am not a theoretical researcher. Since the situations/preconditions of the claims are extremely simple, I think its significancy is not high for practitioners, but the contributions might be significant for theoretical researchers and might inspire the follow-up work.\n\nWEAKNESSES:\nI think the main weakness of this paper is the mismatch between scope it seems to cover and its actual scope. The title and abstract suggests that this paper tries to study why the ICL works well given the unstructured training data in practice, but what the paper actually did is thoroughly studying 3 toy situations. \n\nI understand that we often have to simplify the situations in order to get strong theoretical conclusions. I also think that, at least to me, it is difficult to derive those theoretical results in such simplified situations and all the toy situations are relevant to the ICL. Nevertheless, I think these situations are not very representative to most of the practical ICL settings. After all, most ICL is beyond just relying the co-occurrence statistics of the sentences like CBOW, finding the first letter of the word, and repeating some words in the context. \n\nI understand that nowadays, one paper often needs to oversell its scope in the title and abstract to get attentions. Hence, although I suggest that the authors can revise the main storyline to reduce the overselling, I am fine with the current title and abstract if this paper is accepted at the end. I am also not a researcher who studies theory, so I do not know how significant or novel these theoretical results are. Therefore, I would like to let other researchers who have better theoretical background to rate this paper.\n\nQUESTIONS:\nI have no specific question to the authors, so I think the rebuttal won't change my opinion. I won't lower my score if the authors choose to skip the rebuttal to my comments.\n\nLIMITATIONS:\nThe limitation discussion in the appendix K is fair.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the emergence of in-context learning (ICL) in both CBOW (Mikolov et al,. 2013) and Transformer models. The focus is on simple synthetic settings that can be studied both theoretically and through synthetic experiments with small models. The paper identifies co-occurence as a key ingredient for ICL to emerge in CBOW. Then the paper considers how positional information is critical for a Transformer (or any) model to identify certain order-dependent patterns. Finally, the paper presents two sythetic scenarios involving repetition in which ICL fails with a simple model.\n\nSTRENGTHS:\n- The paper begins by identifying synthetic scenarios in which co-occurrence within a sentence is sufficient for a continous bag-of-worlds (CBOW) model to be able to perform ICL. The paper proves two theorems identifying when co-occurrence statistics are sufficient to ensure that CBOW could perform ICL. \n- The paper then proves that positional information is required to perform certain sythetic tasks related to the ordering of tokens.\n- Finally, the paper identifies two sythetic settings in which one might expect ICL to work.\n- Sythetic experiments support each of the above claims.\n\nWEAKNESSES:\n- The paper states that \"ICL is achievable by only modeling co-occurrence information using [CBOW]\". However, this seems to miss the generality with which the term ICL is used. That is, ICL is commonly used for generation tasks such as full-sentence machine translation (not just the simple token-level translation examples in this paper). So to say that \"ICL is achievable\" seems like a misuse of the terminology. Without a more careful definition of ICL, this statement is invalid.\n- After showing that Llama 2 is unable to use ICL to translate the word English words \"soon\" and \"main\" to Indonesian, the paper claims that \"these models should be equally likely to produce the correct answer for any given [word], irrespective of its relevance to the in-context examples. However, our experiment demonstrates that this is not the case\". This is a huge leap for a poorly designed experiment. Llama 2 was trained on 98.08% English data. The amount of Indonesian language data may have been miniscule. As such, co-occurence may offer an explanation for the result, but adjacency might be equally informative. To speak of co-occurrence without any discussion of adjacency seems a bit odd here. This same issue appears later in the paper's claim \"This suggests ICL may arise from co-occurrence information\", whereas a claim that it is informed by co-occurrence might be more apt.\n- It is not clear to this reader why one would expect the setting in Section 4.1 to succeed via ICL in the first place. For example, we also wouldn't expect these settings to suceeed if they were presented to a supervised learner either because of the mismatch between the training examples and the prompt example.\n- The paper relegates the entire 2.5 page related work section to the appendix. It would be better to include more in the main paper; at present only lines 25-32 in the Intro address prior work making it difficult to position this paper early on.\n\nQUESTIONS:\n- In line 258, the paper claims that \"each token in V should be present as the first token in both the training and test sets.\" But shouldn't we be interested in whether this is really required in the largest of LLMs? Is there any way to connect this result back to larger models?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper studies the emergence of ICL using a synthetic setting. Particularly, it focuses on the importance of concurrence statistics to ICL, and shows that under some simplified conditions, a CBOW-styled model is proven to complete the correct completion for an ICL example. The paper additionally proves the importance of position encodings in the studied setting, showing that when the ICL task is inherently task dependent, position encodings is necessary for good performance.

STRENGTHS:
The paper studies an important problem. The approach---reconstructing LM behavior in much "shallower" models---is intriguing and can be applied to additional problems concerning LMs. The technical claims are well presented and the paper is overall very readable.

WEAKNESSES:
The main weakness is that the paper studies a very synthetic setting. I understand some simplification are needed for the derivation of theoretical results, and this is OK. But, for example, it would be interesting to try deriving results on cases where the input consists of valid grammatical sentences, rather than a concatenation of tuples. If that is not possible, the paper should clearly state the disparity between "real" ICL setting and this setting. While LMs can be presented with tuples in inference time, they are usually not trained on such tuples, but rather on free form language.

QUESTIONS:
The experimental results use cross entropy loss rather than the squared loss used in the theory. Why?

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-14

comments: SUMMARY:
The paper investigates the emergence of ICL from training on unstructured data. It explores two types of ICL tasks: the first involves input-output pairings that frequently co-occur within sentences, and the second comprises recognizable patterns that do not commonly co-occur. The authors demonstrate that the first task can be addressed by modeling co-occurrence information, and highlight the importance of positional information and blocked noise structures through the second task. Additionally, the paper discusses scenarios where ICL fails. Both theoretical and experimental evidence are provided in the paper.

STRENGTHS:
- It enhances understanding of how the structure of training data influences the emergence of ICL capabilities
- the paper provides a mix of theoretical proofs and empirical validations to support its claims

WEAKNESSES:
- There is a lack of experiment details in the paper, such as the number of training sentences used, the frequency of each input-output pair's repetitions within training sentences, and the methodology for generating training and evaluation data.
- The scope of the experiments is limited, using small datasets and simplistic model architectures. Moreover, there is an absence of real-world data.
- There is uncertainty about whether the findings would scale well to complex real-world data, larger models and higher embedding dimensions.

QUESTIONS:
Can you add examples of training sentences and prompts for experiments?

LIMITATIONS:
Limitations are discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16