Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning

## Abstract

Abstract The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demonstrations.
However, relationships between the two abilities and how such relationships affect the emergence of ICL is unclear.
In this paper, we take the first step by examining the pre-training dynamics of the emergence of ICL.
With carefully designed metrics, we find that these two abilities are, in fact, competitive during pre-training.
Moreover, we observe a strong negative correlation between the competition and ICL performance.
Further analysis of common pre-training factors ( i.e., model size, dataset size, and data curriculum) demonstrates possible ways to regulate the competition.
Based on these insights, we propose a simple yet effective method to better integrate these two abilities for ICL at inference time.
Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters.
The code is available at https://github.com/RUCAIBox/Competitive-ICL .

## 1 Introduction

In-context learning (ICL) Brown et al. () represents a significant advancement in the capabilities of large language models (LLMs).
It allows models to rapidly adapt to new tasks without updating the parameters by adding only a few examples as demonstrations to the input.
This capability has profound applications on a wide range of tasks Dong et al. (); Lin et al. ().

To explore the underlying mechanism, existing work Pan et al. (); Wei et al. () mainly focuses on how LLMs perform ICL during inference.
Two main abilities are considered to play important roles in ICL: task recognition (TR), which recognizes the task from demonstrations and utilizes pre-trained priors, and task learning (TL), which directly learns to solve the task from demonstrations.
Furthermore, recent research Pan et al. () has found that TR is relatively easier to obtain and can be observed in small models with only 350M parameters, while TL would often emerge in large models with billions of parameters.
Based on this, Wei et al. () further explore the relationships between these two abilities and show that TR takes the dominant in smaller LLMs while TL is more emphasized in larger LLMs.
However, how these two abilities quantitatively affect the emergence of ICL is under-explored.

Figure: (a) MiniCPM-2B
Refer to caption: x1.png

In this work, we take the first step towards unraveling the mystery, *i.e.,* *relationships between the two abilities and how such relationships affect the emergence of ICL*, by examining the pre-training dynamics of LLMs.
To achieve this goal, we first propose to disentangle the two abilities by manipulating the input-label settings Pan et al. (), so as to measure the performance of TR and TL for each model checkpoint in the pre-training.
As illustrated in Figure 1, we can observe that the emergence of ICL encounters many fluctuations, along with competition between its two abilities (*i.e.,* their performance actually changes in the opposite direction).
To quantify such a competitive relationship, we propose new measurements to reflect how one ability suppresses the other.

With the proposed metrics, we find that the competitive relationship widely exists for existing LLMs with various training settings.
More importantly, it demonstrates a strong correlation with ICL.
First, during pre-training, the competition exhibits a “stable–rise” pattern, typically reflecting fluctuations and improvements in the performance of ICL.
Second, with respect to the entire pre-training process, the average intensity of competition (defined in Section 2) is negatively correlated with the final ICL performance: the less competition, the better the ICL performance.
These findings suggest that regulating the competition between the two abilities of ICL could be crucial for its emergence.
We further investigate the influence of common pre-training factors (*i.e.,* model size, dataset size, and data curriculum) on the competition.
We find that: (1) scaling model size can lead to the early appearance of competition but effectively reduce the average intensity of competition;
(2) scaling dataset size can postpone the competition;
and (3) specific data curricula can adjust the intensity of competition for the enhancement or specialization of LLMs.

Furthermore, we propose a simple yet effective method to fuse the two abilities of ICL for better performance at inference time.
Specifically, we first select two checkpoints from the pre-training process with the best abilities of TR and TL, respectively.
Then, they are fused with adaptive ensemble learning, where the contribution of each one is adaptively determined by its performance.
To validate the effectiveness of our approach, we conduct experiments on extensive datasets and LLMs with various training settings.
Experimental results show that this simple method can effectively boost the performance of ICL and outperform several competitive baselines, even if the total parameters are less than half of a single larger LLM.

Our contributions can be summarized as follows:

$\bullet$ To the best of our knowledge, this is the first time that the competitive relationship between the two abilities of ICL (*i.e.,* TR and TL) and its emergence has been investigated.
By examining the pre-training dynamics of ICL, we demonstrate a strong negative correlation between the emergence of ICL and the competition between TR and TL.

$\bullet$ We conduct a fine-grained analysis of common pre-training factors (*i.e.,* model size, dataset size, and data curriculum) to understand their influence on the competition between TR and TL.

$\bullet$ We propose a simple but effective method to better integrate TR and TL for ICL at inference time.
Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters.

## 2 Background and Measurement

In this section, we introduce the background of TR and TL and further propose new measurements to quantify the competition between them.

### 2.1 Task Recognition and Task Learning

Typically, an LLM performs ICL by using input-label pairs from the target task as demonstrations, *i.e.,* $D_{k}=\{(x_{1},y_{1}),\dots,(x_{k},y_{k})\}$, to predict the label for the test input.
In existing literature Pan et al. (); Lin and Lee (), it has been widely recognized that ICL can be attributed to two major underlying abilities, namely *task recognition (TR)* and *task learning (TL)*.
Specifically, TR refers to the ability of an LLM to recognize the target task from demonstrations and only utilize its own knowledge obtained from pre-training to solve the task, while TL refers to the ability of an LLM to solve the target task solely based on demonstrations.

To disentangle ICL into the two main abilities, existing studies Pan et al. (); Lin and Lee () are mainly developed based on an important assumption: the mapping information between input and label in demonstrations is more important for TL.
Following Pan et al. (), we consider three settings to study the effect of TR and TL:

$\bullet$ *Gold*:
It refers to the standard ICL setting, where we use the correct input-label pairs.
This reflects both TR and TL abilities.

$\bullet$ *Random*:
To evaluate TR ability, we randomly sample labels from the label space of the target task for each input in demonstrations.

$\bullet$ *Abstract*:
To evaluate TL ability, we map the original labels in demonstrations to semantically unrelated ones (*e.g.,* numbers, letters, or symbols).

With the above settings, we can conduct the corresponding empirical experiments by manipulating the input-label relations (*i.e.,* *random* and *abstract* settings) to quantify the effect of TL and TR.

### 2.2 Competition Measurement

In this paper, an important hypothesis is that
competitive relationships exist between TR and TL during pre-training.
To investigate this, we assume that the intermediate checkpoints of LLMs are available, denoted as $\mathcal{M}_{\theta}=\{{M}_{\theta_{1}},{M}_{\theta_{2}},\cdots,{M}_{\theta_{t
}}\}$.
Specifically, we first calculate the performance change of TR and TL during pre-training as follows:

$$ $\displaystyle\Delta\text{TR}_{i}$ $\displaystyle=\text{Acc}_{i+1}^{\text{rand}}-\text{Acc}_{i}^{\text{rand}},$ (1) $\displaystyle\Delta\text{TL}_{i}$ $\displaystyle=\text{Acc}_{i+1}^{\text{abs}}-\text{Acc}_{i}^{\text{abs}},$ (2) $$

where $\text{Acc}_{i}^{\text{rand}}$ and $\text{Acc}_{i}^{\text{abs}}$ denote the accuracy of the intermediate checkpoint ${M}_{\theta_{i}}$ under the *random* and *abstract* settings introduced in Section 2.

If the performance of TR and TL changes in opposite directions, it would indicate that competition actually occurs, which can be represented as:

$$ $\displaystyle C_{i}^{h}={}$ $\displaystyle\mathbb{I}(\Delta\text{TR}_{i}\cdot\Delta\text{TL}_{i}<0)$ $\displaystyle\cdot\mathbb{I}(\left|\Delta\text{TR}_{i}\right|>\epsilon)\cdot \mathbb{I}(\left|\Delta\text{TL}_{i}\right|>\epsilon),$ (3) $$

where $\mathbb{I}(\cdot)$ is the indicator function.
Here, we consider two additional indicator functions to reduce the influence of inaccurate performance estimation.
$\epsilon$ is set to 0.01 in our experiment.
Furthermore, to measure the intensity of competition $C_{i}^{s}$, we consider using the ratio between the performance changes of TR and TL:

$$ $\displaystyle C_{i}^{s}=C_{i}^{h}\cdot$ $\displaystyle\left[\mathbb{I}(\Delta\text{TR}_{i}<0)\cdot\left|\frac{\Delta \text{TR}_{i}}{\Delta\text{TL}_{i}}\right|\right.$ $\displaystyle\left.+\ \mathbb{I}(\Delta\text{TL}_{i}<0)\cdot\left|\frac{\Delta \text{TL}_{i}}{\Delta\text{TR}_{i}}\right|\ \right].$ (4) $$

Here, we assume that an increase in the performance of one ability at the cost of a decrease in the performance of the other ability indicates the intensity of competition.
A larger value of $C_{i}^{s}$ suggests more intense competition, as it implies a greater decrease in the performance of one ability for a given increase in the performance of the other.
Moreover, to investigate the dynamics of competition during pre-training, we calculate the cumulative intensity score $R_{i}$ as follows:

$$ $R_{i}=\frac{\sum_{j=1}^{i}{C_{j}^{s}}}{\sum_{j=1}^{N}{C_{j}^{s}}}.$ (5) $$

This measure tracks the cumulative proportion of the total competition intensity up to the $i$-th training step, providing insight into how competition evolves over time.

## 3 Empirical Analysis

In this section, we present the empirical analysis of the competition relationships between TR and TL for ICL.

### 3.1 Experimental Setup

Tasks and Datasets.
Following Pan et al. (), we select 16 datasets across four types of tasks for the experiment: sentiment analysis, topic/state classification, toxicity detection, and natural language inference/paraphrase detection.
Details about the datasets are depicted in Appendix A.
Due to computational constraints, we sample 1000 examples from each dataset for evaluation.

Models.
Since our work focuses on the pre-training dynamics of ICL, we select LLMs that have more than 350M parameters and provide their intermediate checkpoints: the Pythia suite (6 model sizes ranging from 410M to 12B) Biderman et al. (), MiniCPM-2B Hu et al. (), Baichuan2-7B Yang et al. (), Amber-7B Liu et al. (), CrystalCoder-7B Liu et al. (), and OLMo-7B Groeneveld et al. ().
Due to computational constraints, we sample 16 checkpoints in addition to the final one.
They are evenly distributed in the pre-training process.
Experiments with other numbers of checkpoints yield similar results, which are shown in Appendix B.1.
To make the output as deterministic as possible, we set temperature=0.

Other Details.
We use 16 randomly sampled examples as demonstrations by default across the paper following Min et al. ().
The discussion about the number of examples can be found in Appendix B.2.
We use minimal templates to construct demonstrations following Pan et al. ().
Specifically, we use a single newline character (*i.e.,* \n) to connect each input-label pair and three ones to separate examples.
We utilize symbols as labels in the abstract setting. Other kinds of abstract labels yield similar results as discussed in Appendix B.3.
The results are averaged across five random seeds.

### 3.2 Task Recognition and Task Learning Are Competitive During Pre-Training

Figure 1 shows that the emergence of ICL meets many fluctuations, along with competition between its two abilities (*i.e.,* TR and TL).
In this section, we delve into this competition and unveil its relationship with ICL.

Figure: Figure 2: Average ratio of competition for LLMs.
Refer to caption: x3.png

The Existence of Competition.
To confirm the existence of competition between TR and TL, we investigate the pre-training process of 8 LLMs with various training settings.
Specifically, we calculate the average ratio of competitions according to the indicator metric $C_{i}^{h}$ defined in Eq. (3).
As illustrated in Figure 2, all the LLMs exhibit certain levels of competition during pre-training.
For some LLMs, even more than half the time there exists competition.
It suggests that the competition between TR and TL is a widespread phenomenon during pre-training.

Figure: (a) MiniCPM-2B
Refer to caption: x4.png

The Dynamic of Competition.
We further explore the intensity of competition and its evolution during pre-training.
Specifically, we choose MiniCPM-2B and Amber-7B, which are trained with over a trillion tokens with different amounts of parameters.
We track the evolution of the intensity of competition using the metric $R_{i}$ defined in Eq. (5).
Results are shown in Figure 3.
We can observe that the intensity of competition typically repeats the “stable–rise” pattern, which usually corresponds to the fluctuation and increase in the performance of ICL.
Such an interesting phenomenon inspires us to further examine the relationship between the competition and the performance of ICL.

The Relationship Between Competition and ICL.

Figure: Figure 4: ICL performance of the final checkpoint and the average intensity of competition ($\bar{C}^{s}$) for LLMs.
Refer to caption: x6.png

We first examine the relationship between the competition and the performance of ICL based on pre-training dynamics.
As shown in Figure 3, when there exists competition, the performance of ICL tends to increase (78% of the time for MiniCPM-2B and 57% for Amber-7B).
However, when there is no competition, the performance of ICL shows fluctuations, making the situation complicated.
To make it clear, we shift our perspective to the entire pre-training process.
Specifically, we examine the relationship between the average intensity of
competition $\bar{C}^{s}$ defined in Eq. (5) and the ICL performance of the final checkpoint.
As illustrated in Figure 4, with the increase of $\bar{C}^{s}$, the ICL performance tends to drop, with the exception of MiniCPM-2B and CrystalCoder-7B (they will be discussed in Section 3.3).
To further verify their correlation, we calculate the Pearson correlation coefficient.
The result is -0.591, validating their negative correlation.
This finding has important implications for optimizing pre-training processes and suggests that managing the competition between TR and TL could be crucial for enhancing ICL ability.

### 3.3 How Do Factors of Pre-Training Influence the Competition?

As discussed in Section 3.2, the competition between TR and TL during pre-training demonstrates a strong correlation with the final ICL performance.
This motivates us to investigate the influence of pre-training factors on the competition level.
Specifically, we investigate several common factors, *i.e.,* model size, dataset size, and data curriculum.

#### 3.3.1 Effect of Model Size

Figure: (a) Evolving process
Refer to caption: x7.png

We investigate the effect of model size on the competition between TR and TL.
Specifically, we use the Pythia suite
for experimentation since these models share the same training setting in addition to the number of parameters.

We first pay attention to the differences in the evolution of competition.
We can observe from Figure 5(a) that as the model size increases, the evolving curve of competition keeps moving to the left.
This suggests that scaling up model size could make the appearance of competition appear earlier.
One possible reason is that the learning ability of larger LLMs is stronger, and they can possess TR and TL more quickly, thus causing competition between them to occur earlier.

Then, we focus on the changes in the average competition intensity.
As illustrated in Figure 5(b), the average competitive intensity sharply decreases with the increase of model size, with the exception of Pythia-1B.
This indicates that scaling up the model size is helpful in reducing the overall competition.
This may be attributed to the fact that LLMs with more parameters have a larger capacity, where TR and TL can be allocated with more exclusive resources (*e.g.,* neurons).
As a result, although the competition becomes earlier in larger LLMs, the average intensity of competition becomes lower.
Interestingly, we can observe that overall, the average intensity of competition scales as a power-law with model size, which follows a similar pattern with the training loss in the scaling law of LLMs Kaplan et al. ().
We leave the exploration of the relationship between the scaling law of LLMs and the competition between TR and TL as future work.

#### 3.3.2 Scaling Dataset Size

Figure: Figure 6: Competition evolving process ($R_{i}$) of LLMs trained with different dataset sizes.
Refer to caption: x9.png

In this part, we explore the impact of dataset size on the competition between TR and TL.
We conduct experiments using models with roughly the same number of parameters but trained with different dataset sizes.
Specifically, we make the comparison among two sets of LLMs: (Pythia-2.8B and MiniCPM-2B) and (Pythia-6.9B, Amber-7B, and OLMo-7B).

Figure 6 illustrates the evolution of competition during pre-training for these two sets of LLMs.
It can be observed that, for both sets of LLMs, the evolving curve keeps moving to the right with the increasing of dataset size.
This suggests that scaling up dataset size could postpone the competition.
The possible reason behind this is that, when pre-trained on a small dataset, LLMs can quickly memorize the knowledge contained in the dataset.
Thus, they can develop the TR ability for performing ICL in an early stage.
Meanwhile, the TL ability can also be easily acquired, as it primarily involves directly utilizing the information in context, as discussed by Singh et al. ().
As a result, the competition between TR and TL occurs in the early stage of pre-training.
With the increase in dataset size, the development of the TR ability becomes slower since there is more knowledge required to memorize.
Therefore, more competition happens at a later time, which makes the evolving curve shift to the right.

Figure: Figure 7: ICL performance and competition intensity ($C^{s}$) of MiniCPM-2B and Pythia-2.8B. The dashed line is used to distinguish between different training stages.
Refer to caption: x11.png

Figure: Figure 8: ICL performance and competition intensity ($C^{s}$) of CrystalCoder-7B and Amber-7B. Dash lines are used to distinguish between different training stages.
Refer to caption: x12.png

#### 3.3.3 Scheduling Data Curriculum

In this part, we explore the influence of data curriculum on the competition between TR and TL.
Here, we consider two representative strategies for scheduling data curriculum: (1) quality curriculum, which makes arrangements for data of different qualities, and (2) domain curriculum, which makes arrangements for data from different domains.

We first pay attention to the influence of quality curriculum on the competition.
Specifically, we use MiniCPM-2B for experimentation, which utilizes coarse-quality unlabeled data in the first stage and mixes high-quality labeled data in the second stage.
We compare its pre-training process with that of Pythia-2.8B, which has a similar model size and dataset size.
As illustrated in Figure 7, the ICL performance of MiniCPM-2B traps in fluctuations in the latter half of the first stage while starting to increase again in the second stage.
Meanwhile, the competition in MiniCPM-2B also becomes active in the second stage, and its intensity rapidly increases.
In contrast, the ICL performance of Pythia-2.8B keeps in fluctuation in the later stages of pre-training, and the competition is relatively less active.
One possible reason for the success of quality curriculum is that part of the knowledge in high-quality labeled data is actually covered by large-scale coarse-quality data, which may cause competition between TR and TL to enhance their learning for the high-quality data.

We then focus on the influence of domain curriculum on the competition.
Specifically, we use CrystalCoder-7B for experimentation, which utilizes general domain data (*i.e.,* SlimPajama Soboleva et al. ()) in the first stage, mixes general and code domain data (*i.e.,* SlimPajama and StarCoder Li et al. ()) in the second stage, and mainly uses specific programming language data (*i.e.,* Python and web-related data sampled from StarCoder) in the final stage.
Similar to the quality curriculum, we compare its pre-training process with that of Amber-7B, which has a similar model size and dataset size.
As shown in Figure 8, compared with Amber-7B, CrystalCoder-7B demonstrates much less competition in the second stage, along with a higher performance improvement.
The underlying reason may be the domain difference between general text and code, which brings additional knowledge for LLMs to develop TR and postpone the competition between TR and TL on shared knowledge.
In the final stage of CrystalCoder-7B, the competition becomes active again.
This could be attributed to data duplication since training data is sampled from StarCoder, which has been utilized in the second stage.
Duplicated data can stimulate the competition between TR and TL to strengthen their learning of specific skills or knowledge (*e.g.,* specific programming languages for CrystalCoder-7B), thus achieving better model specialization.

## 4 From Competition to Collaboration at Inference Time

As discussed in Section 3.2, the competitive relations between TR and TL abilities would lead to a decrease in ICL performance, and our idea is to mitigate the competition effect and facilitate their collaboration at inference time.
In this section, we first introduce the proposed adaptive ensemble learning method and then show the experimental results.

### 4.1 Method

Our previous analysis in Section 3.2 shows that although ICL achieves the best performance at the end of training under the gold setting (correct input-label mapping), the corresponding dual abilities usually do not achieve the best performance simultaneously.
This observation suggests that it would achieve better ICL performance if we could integrate the best TR and TL capabilities in one model.
Based on this idea, we propose to fuse the corresponding model checkpoints with ensemble learning.
Specifically, we first select two checkpoints with the best ability of TR and TL respectively, and then integrate their probability distributions to make the prediction:

$$ $\mathop{\arg\max}\limits_{y\in\mathcal{Y}}\left[w_{r}\text{Pr}_{{r}}^{\text{ rand}}(y|x)+w_{l}\text{Pr}_{{l}}^{\text{abs}}(y|x)\right],$ (6) $$

where $\text{Pr}_{{r}}^{\text{rand}}(y|x)$ and $\text{Pr}_{{l}}^{\text{abs}}(y|x)$ denote the probability for the TR and TL models to predict the label $y$ under the random and abstract settings respectively, and $w_{r}$ and $w_{l}$ denote the weights for the prediction of the TR and TL models respectively.

In addition, considering the contribution of dual abilities to ICL is usually not equal , we further propose an adaptive ensemble learning method for fusion.
Specifically, we control the contribution of each checkpoint by setting the weight according to their performance, which is calculated as follows:

$$ $\displaystyle w_{r}$ $\displaystyle=\frac{\text{Acc}_{r}^{\text{rand}}-b}{(\text{Acc}_{r}^{\text{ rand}}-b)+(\text{Acc}_{l}^{\text{abs}}-b)},$ (7) $\displaystyle w_{l}$ $\displaystyle=\frac{\text{Acc}_{l}^{\text{abs}}-b}{(\text{Acc}_{r}^{\text{rand }}-b)+(\text{Acc}_{l}^{\text{abs}}-b)},$ (8) $$

where $\text{Acc}_{r}^{\text{rand}}$ is the performance of the model for TR under the random setting, $\text{Acc}_{l}^{\text{abs}}$ is the performance of the model for TL under the abstract setting, and $b$ is the performance of random guess.

### 4.2 Experimental Setting

To comprehensively validate the effectiveness of our method, we consider three different combinations of TR and TL models for fusion:
(1) the same model (*i.e.,* Pythia-1B),
(2) two models with a similar training setting (*i.e.,* Pythia-1B and Pythia-2.8B), and
(3) two models with different training settings (*i.e.,* Pythia-1B and MiniCPM-2B).
Each model is selected to play the roles of both TR and TL, respectively.
We select checkpoints with the best performance for the required ability.
Other settings are the same as in Section 3.1.

We compare our method with three types of baselines:
(1) the TR or TL model itself used for fusion,
(2) LLMs whose parameters are more than the sum of TR and TL models, and
(3) fusion using the same weight (*i.e.,* $w_{r}=w_{l}$) (“fixed” in Table 1).
All the baselines are tested in the gold setting.

**Table 1: Averaged accuracy across 16 datasets for different models and their fusion. We highlight the highest numbers among fusion with the same model combination.**
| Model | # Params | Acc. |
| --- | --- | --- |
| Large models |  |  |
| Amber-7B${}_{\texttt{ICL}}$ | 7B | 50.08 |
| Pythia-6.9B${}_{\texttt{ICL}}$ | 6.9B | 50.22 |
| Pythia-12B${}_{\texttt{ICL}}$ | 12B | 51.74 |
| OLMo-7B${}_{\texttt{ICL}}$ | 7B | 52.10 |
| Baichuan2-7B${}_{\texttt{ICL}}$ | 7B | 52.77 |
| CrystalCoder-7B${}_{\texttt{ICL}}$ | 7B | 55.66 |
| Small models |  |  |
| Pythia-1B${}_{\texttt{ICL}}$ | 1B | 44.55 |
| Pythia-2.8B${}_{\texttt{ICL}}$ | 2.8B | 45.38 |
| MiniCPM-2B${}_{\texttt{ICL}}$ | 2.7B | 50.08 |
| Fusion of small models |  |  |
| Pythia-1B${}_{\texttt{TR}}$+ Pythia-1B${}_{\texttt{TL}}$ (fixed) | 2B | 56.16 |
| Pythia-1B${}_{\texttt{TR}}$+ Pythia-1B${}_{\texttt{TL}}$ (adaptive) | 56.25 |  |
| Pythia-1B${}_{\texttt{TR}}$+ Pythia-2.8B${}_{\texttt{TL}}$ (fixed) | 3.9B | 56.62 |
| Pythia-1B${}_{\texttt{TR}}$+ Pythia-2.8B${}_{\texttt{TL}}$ (adaptive) | 56.83 |  |
| Pythia-1B${}_{\texttt{TL}}$+ Pythia-2.8B${}_{\texttt{TR}}$ (fixed) | 55.23 |  |
| Pythia-1B${}_{\texttt{TL}}$+ Pythia-2.8B${}_{\texttt{TR}}$ (adaptive) | 55.39 |  |
| Pythia-1B${}_{\texttt{TR}}$+ MiniCPM-2B${}_{\texttt{TL}}$ (fixed) | 3.8B | 55.21 |
| Pythia-1B${}_{\texttt{TR}}$+ MiniCPM-2B${}_{\texttt{TL}}$ (adaptive) | 55.31 |  |
| Pythia-1B${}_{\texttt{TL}}$+ MiniCPM-2B${}_{\texttt{TR}}$ (fixed) | 54.31 |  |
| Pythia-1B${}_{\texttt{TL}}$+ MiniCPM-2B${}_{\texttt{TR}}$ (adaptive) | 55.85 |  |

**Table 2: Ablation study for model fusion. Results are averaged across 16 datasets. We highlight the highest numbers among fusion with the same models for TR and TL. “Random” means that the checkpoint is randomly selected, while “Best” means that the checkpoint has the best performance for TR/TL.**
| Models | TR Model | TL Model | Accuracy |
| --- | --- | --- | --- |
| TR: Pythia-1B<br>TL: Pythia-1B | Random | Random | 52.66 |
| Best | Random | 53.52 |  |
| Random | Best | 54.19 |  |
| Best | Best | 56.25 |  |
| TR: Pythia-1B<br>TL: Pythia-2.8B | Random | Random | 48.10 |
| Best | Random | 50.76 |  |
| Random | Best | 55.42 |  |
| Best | Best | 56.83 |  |
| TR: Pythia-2.8B<br>TL: Pythia-1B | Random | Random | 52.48 |
| Best | Random | 53.89 |  |
| Random | Best | 54.39 |  |
| Best | Best | 55.39 |  |
| TR: Pythia-1B<br>TL: MiniCPM-2B | Random | Random | 53.99 |
| Best | Random | 54.79 |  |
| Random | Best | 54.51 |  |
| Best | Best | 55.31 |  |
| TR: MiniCPM-2B<br>TL: Pythia-1B | Random | Random | 52.07 |
| Best | Random | 53.61 |  |
| Random | Best | 53.73 |  |
| Best | Best | 55.85 |  |

### 4.3 Results

As presented in Table 1, our proposed method can significantly boost performance compared to the single TR or TL model.
In addition, such an improvement is consistent across various model combinations, demonstrating that our method is widely applicable.
To our surprise, two small models together can even outperform larger models by using this method, despite their total parameters being less than half of the larger ones.
It suggests that our method can effectively fuse the abilities of TR and TL to achieve better ICL performance.

Furthermore, to verify the effectiveness of each component in our method, we conduct the ablation study.
We consider substituting the best checkpoints with random ones or setting the weights of TR and TL to the same, respectively.
As shown in Table 1 and 2, removing any design would lead to a decrease in performance.
It demonstrates the effectiveness of all the components of our approach.
In addition, the selection of checkpoints with the best sub-ability seems to be more important, which yields a larger performance drop after being removed.
Checkpoints with the best sub-ability are more diverse in their predictions, which is important for successful fusion.

## 5 Related Work

Our work is closely related to the studies on the mechanisms of ICL and model fusion.

The Mechanism of ICL.
Existing work primarily explores the mechanisms of ICL from the pre-training and inference stages of LLMs.
Some work discusses how ICL emerges from pre-training by conducting analysis on pre-training factors like data Chan et al. (); Reddy () and optimization Singh et al. (); Anand et al. ().
Other work Pan et al. (); Min et al. (); Dai et al. () studies the operating mechanism of ICL at inference time.
Researchers empirically find two main abilities in ICL: task recognition (TR) for recognizing the task and utilizing pre-trained priors of LLMs Min et al. () and task learning (TL) for learning from demonstrations Dai et al. ().
In this paper, we explore how TR and TL affect the emergence of ICL.
By examining the pre-training dynamics of LLMs, we demonstrate a strong correlation between the emergence of ICL and the competition between TR and TL.

Model Fusion.
Model fusion aims to enhance performance by combining the strengths of multiple models Li et al. ().
One line of work aims to reduce the difference among different models from perspectives like mode connectivity Nagarajan and Kolter () and alignment Tatro et al. ().
Another line of work studies how to leverage the diversity among models through techniques like weight average Wang et al. () and ensemble learning Sagi and Rokach ().
In this paper, we propose adaptive ensemble learning to fuse checkpoints proficient in TR and TL and achieve better ICL performance.

## 6 Conclusion

In this paper, we presented the first study of the competitive relationship of TR and TL abilities, and quantified their effect on the emergence of ICL.
With specially designed metrics, we found that the competition between dual abilities widely exists in existing LLMs, and the competition intensity is negatively correlated with the ICL performance.
Then, we conducted a detailed analysis of several pre-training factors (*i.e.,* model size, dataset size, and data curriculum) to demonstrate possible ways to regulate the competition.
Furthermore, we proposed a simple yet effective method to better integrate dual abilities at inference time.
Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters.
Overall, our work provides new insights and approaches to study and understand the underlying mechanism of ICL, which is worth deep exploration for improving the capacity of LLMs.

## 7 Limitations

Although our study provides valuable insights into the dual abilities of ICL and its emergence, several limitations should be noted.
First, our research focuses on classification tasks since they can be easily adapted for the three evaluation settings of ICL.
Other types of tasks are left for future work.
Second, our investigation is confined to conventional ICL paradigms and does not explore alternative paradigms such as chain-of-thought prompting (CoT).
Third, due to computational constraints, our study mainly considers LLMs with up to 12 billion parameters.
Replicating our study with larger-scale LLMs could provide further insights and validate the robustness of our findings across different model sizes.

## Appendix A Tasks and Datasets

We conduct experiments on four types of tasks: Sentiment Analysis, Topic/Stance Classification, Toxicity Detection, and Natural Language Inference/Paraphrase Detection.
For Sentiment Analysis, we use datasets including SST-2 Socher et al. (), financial_phrasebank Malo et al. (), emotion Saravia et al. (), and poem_sentiment Sheng and Uthus ().
For Topic/Stance Classification, we utilize TREC Voorhees and Tice (), tweet_eval_atheist, and tweet_eval_feminist Mohammad et al. (); Basile et al. ().
For Toxicity Detection, we include tweet_eval_hate, ethos_race, ethos_gender, ethos_national_origin, and ethos_religion Mollas et al. ().
For Natural Language Inference/Paraphrase Detection, we employ SICK Marelli et al. (), SNLI Bowman et al. (), WNLI Levesque et al. (), and MRPC Dolan and Brockett ().

We follow  Min et al. () to select samples from the training set as demonstrations.
Additionally, we randomly sample 300 examples as the development set for validation in Section 4 and another 1000 examples as the test set for evaluation in all experiments from the development set.

## Appendix B More Experiments

### B.1 The Number of Intermediate Checkpoints

**Table 3: Different numbers of intermediate checkpoints**
| # Intermediate checkpoint | 8 | 16 | 32 |
| --- | --- | --- | --- |
| Pythia-6.9B | 12.50 | 37.50 | 53.12 |
| OLMo-7B | 50.00 | 37.50 | 43.75 |
| MiniCPM-2B | 62.50 | 56.25 | 37.50 |

In the paper, we use 16 checkpoints in addition to the final one.
In this part, we conduct experiments using different numbers of checkpoints (*i.e.,* 8 and 32).
We report the average competition ratio across 16 datasets and 5 random seeds.
Table 3 shows that the number of checkpoints does not affect the experimental results.
They consistently demonstrate that there is a competitive relationship between TR and TL during the pre-training process.

### B.2 The Numbers of Examples in Demonstration

**Table 4: Different numbers of examples in demonstration**
| # Examples | 4 | 8 | 16 |
| --- | --- | --- | --- |
| Pythia-6.9B | 18.75 | 25.00 | 37.50 |
| OLMo-7B | 5.00 | 37.50 | 37.50 |
| MiniCPM-2B | 43.75 | 31.25 | 56.25 |

In the paper, we use 16 randomly sampled examples as demonstrations.
To explore the impact of the number of examples, we report the average competition ratio with other numbers (*i.e.,* 4 and 8) of demonstrations.
As presented in Table 4, it can be observed that the number of examples does not affect the competitive relationship during the pre-training process.

### B.3 The Type of Abstract Labels

**Table 5: Different types of abstract labels.**
| Abstract<br>labels | Symbols | Numbers | Letters |
| --- | --- | --- | --- |
| Pythia-6.9B | 37.50 | 18.75 | 37.50 |
| OLMo-7B | 37.50 | 50.00 | 43.75 |
| MiniCPM-2B | 56.25 | 43.75 | 50.00 |

In the paper, we utilize symbols in the abstract setting.
In this part, we follow Pan et al. () to use other types of semantically unrelated labels (*i.e.,* numbers and letters).
Table 5 shows the average competition ratio by using different labels.
It indicates that, regardless of the choice of semantically unrelated labels, the conclusions are consistent with the abstract symbols.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - I am not sure why the TL and TR curves appear to have identical data points in both plots of Figure 1.
- Could you discuss potential mechanisms or hypotheses on why the competition between TL and TR occurs, based on your observations or existing literature
- Apart from TL and TR, discuss any limitations in your experimental design that might not fully control for other factors that might affect ICL. Please suggest potential ways to address these in future work.
- In Figure 6a, it seems $C_i^S$ tend to increase in the second phase. Could you provide more detailed analysis of the relationship between $C_i^S$ and ICL performance in this specific case, and discuss any potential confounding factors.
- In Figure 6b, there is no clear evidence that CrystalCoder-7B experiences less competition than Amber-7B. Could you provide quantitative evidence supporting your conclusion about this figure or clarify if there are specific aspects of the figure that led to this interpretation.
- In Table 1, it would be helpful to report the standard deviation of the accuracies and TFLOPs. I would also recommend to include a statistical test to evaluate the significance of the obtained results.
soundness: 2
strengths: - The paper is well-written and easy to follow
- The analysis of TR and TL behaviours and their effect on ICL performance is interesting
- The fusion of small models performs comparably to larger models, making a step toward the feasibility of training LLMs with fewer resources
summary: This work explores the dynamics between two abilities of in-context learning (ICL) in large language models (LLM), i.e., task recognition (TR) and task learning (TL). This paper argues that TL and TR are indeed competitive during pre-training. To show this quantitatively, the authors proposed a metric called Competition Intensity that measures the extent to which one ability exceeds the other. The paper suggests that the Competition Intensity is negatively correlated with the ICL performance, and that regulating the Competition Intensity via specific choices of pre-training factors (model and dataset size, and data curriculum) is crucial for improving ICL performance. By using this hypothesis, the paper proposes a technique to fuse small models.

The contributions are:
1) Explore the relationship between TR and TL and their impact on ICL performance,
2) Analyze the influence of the pre-training factors on the competition between TR and TL, and
3) Propose an approach to fuse small LLMs more effectively.
weaknesses: - The novelty of the methodology section is limited.
- There is no significant improvement observed in Table 1 compared to other fusion models, small models, and large models in terms of Accuracy
- Limited discussion, limitations and future work. It would be interesting to explore how the fusion technique performs on each of the 16 individual datasets, and possibly identify common characteristics of datasets where the technique performs well/poorly.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: - Are there a few benchmarks that skew the average to Pythia-1B p-fixed instead of Pythia-1B p-adaptive? We do not observe p-fixed performing better than p-adaptive in other scenarios
- Could you compare your proposed ensembling method to other ensembling methods? (You can choose Pythia-1B + Pythia-2B final checkpoints for baselines to compare to your selection of checkpoints for ensembling)
soundness: 3
strengths: - The paper is well-written and easy to follow
- The paper shows a clear sign of competition between TR and TL, and its effect on ICL improvement throughout pretraining
- The authors clearly defined each metric, and defended their relevance
- They have numerous ablation studies to show the effects of factors (such as dataset size and curriculum training) to ICL (or TR/TL competition)
- They have ablation studies on various ensembling techniques to show their performance improvement is not just due to model fusion, but something more systematic. However, this may require more studies (see weaknesses)
summary: The authors examine the competition between Task Learning (TL) and Task Recognition (TR) throughout pretraining, and its effect on the in-context learning (ICL) capabilities of LLMs. They show that when there exists a competition, this usually leads to stagnation in ICL capabilities or a decrease. Motivated by this, they propose an ensembling technique to between different checkpoints of the same (or different) models that performed best in TL or TR. They show that, this combination can lead to performance improvements in ICL that can rival models with even larger model parameters than the sum of the fused smaller models.
weaknesses: - I believe the results on Table 2 can be better represented. Especially choosing a "random" checkpoint can be very misleading, and frankly I don't think it's necessary to look at such data point. I'm interested in the {TR or TL}_best + ICL_best checkpoints for the same models. And, ICL_best_model_1 + ICL_best_model_2 when ensembling different models.
- I also believe p-adaptive should be better motivated. I don't see the performance improvements big enough, and also Pythia-1B models, p-fixed performs significantly better.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - The paper offers very little discussion regarding the choice of the number of checkpoints. Does the method’s performance remain robust when applied with varying numbers of checkpoints? An ablation study would be beneficial to address this.

- It would be helpful if the authors could elaborate more on the intuition behind the dynamics of pre-training and clarify how studying this aspect could directly benefit in-context learning (ICL).

- A key aspect of evaluating the method is the reliability of its modeling of competition intensity dynamics $R_i$. How can we systematically assess the quality of the proposed $R_i$ score, and in what ways does this score contribute to improving ICL capability in practice?

- In the ablation study, it would be beneficial to include results for a non-adaptive ensemble baseline, as well as standard ensemble models incorporating more than two LLMs.

- The proposed metrics rely on checkpoint fluctuations to measure task learning (TL) and task robustness (TR). Would the method remain consistent and valid if checkpoints were sparse? It would be helpful to include discussion or experiments around this. 

-  It would be helpful if standard deviations could be added to the tables and figures from the paper. Could the authors provide statistical test results to determine whether the observed fluctuations are significantly different from random noise? 

- For Fig 3, the relationship between ICL performance and $R_i$ sometimes misalign in certain intervals. Is it possible to quantify the correlations between them and confirm that the positive relationship statistically hold? Additionally, a detailed discussion on potential reasons for those observed misalignments could provide valuable insights.
soundness: 1
strengths: - The paper brings a fresh perspective to ICL by examining the roles of task recognition (TR) and task learning (TL), shedding light on how these two abilities interact and shape ICL performance.

- The authors introduce some new metrics, including the competition indicator and cumulative intensity score, which add depth to our understanding of TR and TL dynamics over time. 

- Through a comprehensive experimental setup, the paper examines model behavior across multiple checkpoints, allowing for a nuanced view of how TR and TL evolve during training. 

- The adaptive ensemble method presented is somewhat novel, selecting models based on their TR and TL strengths. By leveraging each model’s unique abilities, this approach suggests a practical way to optimize ICL performance through complementary ensemble strategies.
summary: This paper investigates how task recognition (TR) and task learning (TL) abilities influence in-context learning (ICL) performance in large language models (LLMs). Building on prior work by Pan et al. (2023) and Lin & Lee (2024), the authors assess TR and TL by constructing Gold, Random, and Abstract input-label pairs to isolate these abilities. They introduce several new metrics based on ICL performance across a sequence of LLM checkpoints:  (1) **competition indicator** $C_i^H$; (2) **competition intensity** $C_i^S$; (3) **cumulative intensity**, which is used to infer the dynamic interplay between TR and TL. Through a series of empirical evaluations, the authors demonstrate key insights into how these abilities interact. Additionally, they propose an adaptive ensemble method that fuses two smaller models to improve ICL performance, suggesting a promising direction for enhancing context-based learning in LLMs.
weaknesses: - The clarity of the paper could be improved to make key concepts more accessible. For instance, terms like "dynamics of pre-training" are introduced without clear definitions and make it challenging for readers to grasp the intended meaning. Also, it's encouraged to clearly highlight what are the key assumptions and intuitions more clearly in the paper.

- The paper also  lacks substantial technical novelty and depth of insight into the ICL problem. The new metrics proposed in for measuring competition indicator, intensity and dynamics heavily rely on an assumption that these metrics could be derived from the change of TL and TR scores from consequent checkpoints.  I'm not convinced that the  change of performance on checkpoints could reflect the capability of TL or TRs. It is hard to figure out what kind of training the model has gone through for each new release and version of checkpoints could be arbitrary too. Especially for large-scale models where checkpoints are not saved at every step.  How could the $\triangle$TL and $\triangle$TR evaluated on consequent checkpoints robustly tell the true capability of TL and TR, which is of interest to the audience? This reliance could limit the broader applicability and scalability of the proposed method.

- The cumulative intensity score $R_i$  used to track the dynamics of competition intensity over time, raises some concerns. Since it simply aggregates $C_i^S$  values, which are strictly positive, this score inevitably increases, potentially making it less informative as a dynamic metric.

- Key figures, such as Figure 1 and Figure 6, lack error bars and the tables do not come with standard deviations, making it difficult to evaluate the robustness of the observed trends. In the paper, the authors discuss a lot about the  "fluctuations" of the checkpoints, which in fact often can be somewhat random, and without error bars, it’s not able to tell whether these patterns are consistent or potentially artifacts.

- One fundamental assumption in this work is that the relationship between TR and TL can be inferred directly from changes in ICL ability across checkpoints. However, training conditions and checkpoint improvements are not always straightforward, and observed changes in ICL ability may not consistently reflect meaningful performance shifts. A more detailed discussion validating this assumption would strengthen the technical foundation of the method.


 - Sections 3 and 4 lack a cohesive connection, and the title of Section 4 may be somewhat misleading, as it suggests that TL and TR no longer compete. Although Section 4 introduces an ensemble model by selecting the best-performing models for TR and TL, it remains unclear whether this ensemble approach results in the highest combined TR and TL scores compared to individual checkpoints. The two chosen models are combined using simple weights inferred from their respective accuracy, but further clarity is needed on the effectiveness of this weighting approach. The experiments would be more robust if the authors included comparisons with other ensemble methods and conducted an ablation study on using fixed, non-adaptive weights (e.g., 0.5, 0.5).

- In Figure 3, the authors plot ICL performance and $R_i$  curves together, showing increasing trends. However, the increase and decrease patterns do not always align, raising questions about the strength of the correlation between ICL performance and $R_i$. More statistical analysis or validation would help clarify the relationship and strengthen the visual interpretations.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - In lines 371-373 where the authors discusses the results shown in figure 6(a), the author attributes higher ICL performance of the mini-CPM model to reduced competition. However, from the figure we can clearly see that the competition intensity of mini-CPM 2B is higher than that of Pythia-2.8B on the right side of the dashed line, which seems to contradict with the author's reasoning. I would like to hear about more explanation on this.
- I would like to hear about more reasoning on how the value of the adaptive weights $w_r$ and $w_l$ is determined, and more experiments with other possible value of $w_r$ and $w_l$.
- Regarding the results in figure 4b, does the authors have any possible explanation on why the Pythia-1B model does not follow the general trend? 
- Related to the point above, I want to note that different Pythia models have different width-depth ratios. If possible, it might be helpful to further investigate the effect of the width and depth of the model separately. The fact that the width-depth ratio of the model might have a big effect on the TR/TL competition also raises a small concern about whether the comparison in table 1 is actually the most fair, if the 1B+1B model has a different width-depth ratio from the other bigger models. I would hope to get more info about this.
- Other than the method proposed in section 4 of the paper, are there other potential implications of the TR/TL competition phenomenon?
- Does the TR/TL competition phenomenon only occur in pre-training, or it also occurs in fine-tuning?
soundness: 3
strengths: - The main insight of this paper that there exists competition between TR and TL and such competition is correlated with worse ICL performance is very novel and interesting to me. Discovering this insight and investigating how different pre-training factors affect TR/TL competition already makes it a good paper, and the fact that the authors propose a well motivated method to improve ICL performance makes the paper a great one.
- The presentation of the paper is good. The authors provide a clear outline in the beginning and smoothly dives into each part of the story.
- The authors provides anonymous code which allows easy reproduction of the experiment results.
summary: The paper studies how two mechanisms related to in-context learning (ICL), task recognition (TR) and task learning (TL), develops over the course of LLM pre-training. Through the experiments, the authors find that there is competition between the TR ability and the TL ability through the pre-training process. The authors further propose a measure to quantify such competition, and fine a negative correlation between the magnitude of such competition to the final ICL performance. The paper also studies how different factors, including parameter count, dataset size, and data curriculum affects the magnitude of the competition and ICL performance. Finally the paper proposes a method to ensemble checkpoints with best TR performance and best TL performance to get better ICL performance, outperforming the ICL performance of a larger model with more than twice the parameters.
weaknesses: - The related work section is a little bit limited. If constrained by space in the main text, it can be elaborated in the appendix. 
- Many experiments in section 3 involves comparisons between only 2-3 models (many of them are trained with datasets and architectures). While I understand that it is very challenging to run carefully controlled experiments due to the lack of pre-trained models in many settings, this still limits the significance of the conclusions.
- See more potential concerns in the "questions" section.

Meta-review: This paper studies a certain competition between task learning (identifying patterns in the new data) and task recognition (recognizing similar patterns from the pre-training data) how this affects the performance of in-context learning. The paper discusses an approach to ensemble models at different stages of training by selecting for these two different properties. This approach can lead to improvements in in-context learning performance. I am happy to recommend that this paper be accepted.

I would encourage the authors to incorporate some of the concerns of the reviewers in the narrative. For example, even in the standard supervised learning and fine-tuning regimen for few-shot learning, one might notice that correct predictions on the test datum (aka in-context learning) occur because, either (i) the few-shot training data is similar to some parts of the pre-training data, and therefore the effective sample size relevant to the test datum is amplified during the fine-tuning phase, or (ii) the few-shot training data is dissimilar to the pre-training data and one only gets test predictions of a quality that is commensurate with the sample size of the few-shot data. It is difficult to write down a precise distinction between “task learning” and “task recognition” because they are two ends of a spectrum (as explored elaborately in Lin and Lee 2024 which the authors cite).

Decision: Accept (Poster)