Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
Authors: Xiaolei Wang
Xinyu Tang
Junyi Li
Xin Zhao
Ji-Rong Wen
Venue: ICLR 2025 Poster (2025)
Keywords: In-context learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demonstrations. However, relationships between the two abilities and how such relationships affect the emergence of ICL is unclear. In this paper, we take the first step by examining the pre-training dynamics of the emergence of ICL. With carefully designed metrics, we find that these two abilities are, in fact, competitive during pre-training. Moreover, we observe a negative correlation between the competition and the performance of ICL. Further analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) demonstrates possible ways to regulate the competition. Based on these insights, we propose a simple yet effective method to better integrate these two abilities for ICL at inference time. Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters.

Published as a conference paper at ICLR 2025
INVESTIGATING THE PRE-TRAINING DYNAMICS OF IN-CONTEXT LEARNING: TASK RECOGNITION VS. TASK LEARNING
Xiaolei Wang Xinyu Tang"";Jung Li', Wayne Xin Zhao and Ji-Rong Wen
GGolling School of Artificial Intelligence, Renmin University of China School of Information, Renmin University of China
*Beijing Key Laboratory of Big Data Management and Analysis Methods
* *Department of Computer Science, National University of Singapore wx wx11999e fosmail com, txy200103100166. com, batmanfly@gmaie com
ABSTRACT
The emergence in-context learning (ICL) is potentially attributed to two ma- jor abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-trained priors, and task learning (TL) for learning from demon- strations. However, relationships between the two abilities and how such re- lationships affect the emergence of ICL is unclear. In this paper, we take the first step by examining the pre-training dynamics of the emergence ICL. With carefully designed metrics, we find that these two abilities are. in fact, com- petitive during pre-training. Moreover, we observe a negative correlation be- tween the competition and the performance of ICI. Further analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) demon- strates possible ways to regulate the competition. Based these insights, we propose a simple yet effective method to better integrate these two abilities for ICL at inference time. Through adaptive ensemble learning, the performance ICL can be significantly boosted, enabling two small models to outperform larger one with more than twice the parameters. The code is available https:/ /ttp//ggttt/t///////////////tttt Cow//RUCAABox/CCooeeee
INTRODUCTION
In-context learning (ICL) Brown et al. 2020) represents a significant advancement in the capa- bilities of large language models (LLMs) (Zhao et al. 2023 Hu et al. 2024b). It allows models to rapidly adapt to new tasks without updating the parameters by adding only a few examples demon- strations to the input. This capability has profound applications on a wide range of tasks (Dong et al., 2022
Lin et al. 2023).
To explore the underlying mechanism, existing work (Pan et al. 2023 Wei et al. 2023) mainly focuses on how LLMS perform ICL during inference. For the underlying mechanisms of ICL, two major abilities are considered to play important roles: task recognition (TR), which recognizes the target task from demonstrations and utilizes the prior knowledge learned from pre- pre-training to solve, and task learning (TL), which directly learns from demonstrations to deal with the task. Furthermore, recent research (Pan et al.) 2023) has found that TR is relatively easier to obtain and can be observed in smaller models with only 350M parameters, while TL would often emerge in larger models with billions of parameters. Based on this observation, Wei et al. (2023) further explore the relationships between these two abilities and reach the same conclusion that TR takes the dominant in smaller LLMs while TL is more emphasized in larger LLMs. However, how these two abilities quantitatively affect the emergence of ICL. under-explored.
In this work, we take the first step towards unraveling the mystery, i.e., relationships between TR and TL and how such relationships affect the emergence of ICL, by examining the performance
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2025
50
TL
TR
45
Accuracy
Accuracy
40
35
ICL
30
30
TR
10 Checkpoint Index
Checkpoint Index
(a) MiniCPM-2B
(b) Amber- 7B
Figure 1: The performance of ICL and its two abilities (i.e., task recognition (TR) and task learning (TL)). The emergence of ICL encounters many fluctuations, where the performance of task recog- nition and task learning changes in the opposite direction.
changes of ICL during pre-training. To achieve this goal, we first disentangle the two abilities by manipulating the input-label pairs following previous settings (Pan et al.) 2023), upon which
can easily measure the performance of TR and TL individually. As illustrated in Figure [ we can observe that the emergence of ICL encounters many fluctuations, along with competition between its two abilities (i.e. their performance actually changes in the opposite direction). To quantify such competitive relationship between TR and TL, we propose competition intensity, a new measurement based on the performance change of TR and TL between two adjacent checkpoints to reflect the extent to which one ability surpasses another.
With the proposed metric, we find that the competitive relationship is prevalent across many LLMs with various training settings. More importantly, demonstrates a negative correlation with ICL. First, during pre-training, the competition exhibits "stable-risc" pattern, and simultancously, the performance of ICL improves or fluctuates in correspondence. Second, with respect to the entire pre-training process, the average competition intensity is negatively correlated with the final ICL performance: the less the competition, the better the ICL performance. These findings suggest that regulating the competition between TR and TL would be crucial for the emergence of ICL. We further investigate the influence of common pre-training factors (i.e. model size, dataset size, and data curriculum) the competition. We conclude that: (1) scaling model size lead to the early appearance of competition but effectively reduce the average intensity of competition; (2) scaling dataset size can postpone the competition; and (3) specific data curricula can adjust the intensity of competition for the enhancement of L.LMs.
Our analysis reveals that with effective regulation of the competition between TR and TL, LLMs could achieve better ICL performance. To this end, we propose simple yet effective method to fuse the two abilities for better ICL performance at inference time. Specifically, we first select two check- points from the pre-training process with the best abilities of TR and TL, respectively. Then, they are fused with adaptive ensemble learning, where the contribution of each one is adaptively deter- mined by its performance. To validate the effectiveness of our approach, conduct experiments extensive datasets and LLMs with various training settings. Experimental results demonstrate that this simple method can effectively enhance ICL performance, outperforming several competitive baselines, even with less than half the parameters of a larger LLM.
Our contributions can be summarized as follows:
To the best of our knowledge, we are the first to study the relationship between the two abilities of ICL (i.e., TR and TI.) and its effect the emergence of ICI.. With newly proposed measurements,
2

---

Published as a conference paper at ICLR 2025
conduct empirical analysis of the pre-training dynamics of ICL, and discover competitive relationship between TR and TL and its negative correlation with the emergence of ICL.
We conduct fine-gratd fine-grained analysis of common pre-training factors (i.e., model size, dataset size, and data curriculum) to understand their influence on the competition between TR and TL, e.g., scaling model size can effectively decrease the average competition intensity.
We propose a simple but effective approach to better integrate TR and TL at inference time. By fusing two smaller models through adaptive ensemble learning, the performance of ICL can outperform larger model with much more parameters than the smaller models.
2 BACKGROUND AND MEASUREMENT
In this section, we introduce the background of task recognition (TR) and task learning (TL. and further propose our measurements quantify the competitive relationship between them.
2.1 TASK RECOGNITION AND TASK LEARNING
Typically, an LLM performs ICL by using input-label pairs from the target task as demonstrations, ie., D2 (r1 y1 ...... (xgg yk }. to predict the label for the test input. In existing literature (Pan et al., 2023 Lin & Lee 2024 Jang et al. 2024), it has been widely recognized that ICI. can be attributed to two major underlying abilities, namely task recognition (TR) and task learning (TL). Specifically, TR refers to the ability of an LLM to recognize the target task from demonstrations and only utilize its own knowledge obtained from pre-training to solve the task, while TL refers to the ability of LLM to solve the target task solely based demonstrations.
To disentangle the two main abilities from ICL, existing studies (Pan et al. [203] Lin & Lee 2024) mainly developed based input and the label in demonstrations is more important for TI. than TR. Under this assumption, three settings can be used evaluate these two abilities and ICL:
important assumption: the mapping information between the
Gold: It refers to the standard ICL setting, where we use the correct input-label pairs. This setting used evaluate the ICL performance of LLMs.
Random: To evaluate TR ability, we randomly sample labels from the label space of the target task for each input in demonstrations.
Abstract: To evaluate TL ability, we map the original correct labels in demonstrations to semanti- cally unrelated tokens (e.g., numbers, letters, or symbols).
The above settings are widely used in existing work (Pan et al. 2023 Lin & Lee 2024 Jang et al.
2024
To align with them, follow the same settings quantity the performance of TL and TR.
2.2
COMPETITION MEASUREMENT
In our prior analysis shown in Figure we discover that there exists certain degree of competition between TR and TL during pre-training. To quantify this, we propose competition intensity, new assume that
measurement based the performance changes of TR and TL. As prerequisite, the intermediate checkpoints of an LLM are available, denoted as Mg {Me, .M...... M,,....,,,.... }, where the index increases with the number of training steps and N is the total number of checkpoints. Based on this, to track the pre-training dynamics of TR and TL., we can calculate their performance changes as follows:
ATR, Acc
Acc mind
ATL: Acc Acc.
(1)
where Acclnn and Acet denote the accuracy of the intermediate checkpoint Mo, under the random
and abstract settings introduced in Section 2.1 To determine whether there exists competition between TR and TL, one feasible way is to check whether their performance changes in the opposite
direction since competition typically results trade-off between these two abilities. To indicate such an existence of competition, we compute competition indicator C.Hl as follows:
Cf TATR, ATL, 0) T(IATR >) ATATTL|| c),
3

---

Published as a conference paper at ICLR 2025
where II()) is the indicator function and we add two additional indicator functions reduce the in- fluence of inaccurate performance estimation. é is set to 0.01 in the experiment. Furthermore, based on the competition indicator, we propose competition intensity to quantify the degree of competition at different pre-training stages. We denote competition intensity as C,S, which is represented as the ratio between the performance changes of TR and TL.:
c = - [TEATR, <0)
ATR; ATL:
ATL: ATR;
T(ATL, <
Here, if the competition exists (i.e., C = 1), an increase in the performance of one ability will lead decline in the performance of another (i.e., ATR; ATI, < 0 in Eq. Eq. Thus, larger value of C suggests higher competition intensity, as it implies a greater decrease in the performance of one ability (i.e., numerator in Eq. 3 for given increase in the performance of another (i.e., denominator in Eq. 3) Moreover, to track the dynamics of competition intensity, we calculate the accumulative intensity R; as follows:
Σj= C R
Σ Cj
This measure tracks the cumulative proportion the intensity up to the i-th intermediate checkpoint, providing insight into how competition evolves over time.
EMPIRICAL ANALYSIS
In this section, we present the empirical analysis of the competition between TR and TL.
3.1 EXPERIMENTAL SETUP
Tasks and Datasets. Following Pan al. 2023)
select 16 datasets across four types of tasks for the experiment: sentiment analysis, topic/state classification, toxicity detection, and natural lan- guage inference/paraphrase detection. Details about the datasets are depicted in Appendix [ Due to computational constraints, we sample 1000 examples from each dataset for evaluation.
Models. Since our work focuses on the pre-training dynamics of ICI., we select L.MSs with more than 350M parameters and access their intermediate checkpoints: the Pythia suite (6 model sizes ranging from 410M to 12B) (Biderman et al.) 2023), MiniCPM-2B (Hu et al. [024a),, Amber- 7B (Liu et al. 2023). CrystalCoder-7B (Liu et al. 2023). OLMo-7B (Groeneveld Baichuan2-7B Yang et al. 2023), and K2-65B (Liu et al. 2024). Due to computational constraints, sample 16 checkpoints for each model, which are evenly distributed in the pre-training pro- cess. Experiments with other numbers of checkpoints yield similar results, which are shown in Appendix B.I To make the output as deterministic as possible, we temperature=0 when sampling.
al. 2024),
Other Details. We randomly sample 16 examples demonstrations by default across the paper fol- lowing[Min et (202)). The discussion about the number of examples can be seen in Appendix(B.. [B.2 We use minimal templates construct demonstrations following Pan al. (2023). Specifically, we use single newline character (i.e., \nn to connect each input-label pair and three ones to separate examples. We utilize symbols as labels in the abstract setting. Other kinds of abstract labels yield similar results discussed in Appendix B.3 The results are averaged across five random seeds.
3.2 TASK RECOGNITION AND TASK LEARNING ARF COMPETITIVE DURING PRE- TRAIINGNN
Figure T shows that the emergence of ICL is along with competition between its two abilities (i.e., TR and TL.). In this section, we delve into this competition and unveil relationship with ICI.
The Existence of Competition. To confirm the existence of competition between TR and TL, investigate the pre-training process of LLMs with various training settings. Specifically, we calculate the average occurrence of competitions according the indicator C²²² (Eq. 2). As illustrated in
4

---

Published as a conference paper at ICLR 2025
Pythia410M
0.5
1.50
Pythia 1.48
1.25
1,00
0.3
Pythis-2.88
MiniCPM-28
0.50
Pythia-18
Pythia
0.25
Amber
DLMo-78
0.1
K2-65B
0025
Pythia Pythia Pythia
Mi.4 Amber sytnia (28) (79) (128) (65B)
49
(410M) (1B)
ICL
(a) Existence of competition
(b) Relationship between competition and ICL
Figure 2: (Left) Average occurrence of competition among seven LLMs. (Right) Average competi- tion intensity C w.r. ICL performance of the final checkpoint.
100
50
80
49
80
60
60
0
47
44
46
20
45
ICL
R,
R,
0
10 Checkpoint index
5
10 Checkpoint index
15
(a) MiniCPM-2B
(b) Amber-7B
Figure ICL performance w.r.f. accumulative competition intensity R2 during pre-training.
Figure 2a all the LLMs exhibit certain levels of competition during pre-training. For some LLMs, there exists competition for more than half of the time (i.e., over 0.5). It means that the competition between TR and TL is widespread phenomenon during pre-training.
The Dynamic of Competition. We further explore the dynamics of competition during pre-training. Specifically, we choose MiniCPM-2B and Amber-7B, which are trained with trillion tokens with different amounts of parameters. We show the accumulative competition intensity (Eq. 4 with respect ICL performance in Figure We observe that the accumulative competition intensity displays a "stable-rise" pattern. When the accumulative curve flattens (i.e., the intensity C, is low), ICL performance rises in fluctuation. Conversely, when the accumulative curve rises (i.e., the intensity C is high). ICL performance shows no obvious improvement and may even decline. Such an interesting phenomenon inspires further examine the relationship between the competition and ICI..
The Relationship Between Competition and ICL, this part, we delve into the relationship be- tween the competition and ICL performance based their pre-training dynamics. As shown in Figure 3] when there exists competition (i.e., the red curve rises), the performance of ICL tends to decrease does not improve. However, in the absence of competition (i.e., the red curve flattens),

---

Published as a conference paper at ICLR 2025
100
Pythia
Pythia suite
Pythia 2:88
1.50
80
Pythia 6.98
Pythia
60
1.00
<
40
0.75
20
0.25
5
10
15
10
Checkpoint index
Model Size (B)
(a) Evolving process
Average competition intensity
Figure 4: The evolution (Rt) and average intensity (CS) of competition with different model sizes.
the performance of ICL rises in fluctuation. To demonstrate the global impact of competition, depict the average competition intensity C with respect to the ICL performance the final check- point. As shown in Figure 2b) with the increase of C the ICL performance tends to drop. To further verify their correlation, we calculate the Pearson correlation coefficient. The result -0.714, validating their negative correlation. This finding suggests that regulating the competition between TR and TL could be crucial for enhancing the ICI. ability.
3.3 How Do FACTORS OF PRE-TRAINING INFLUENCE THE COMPETITION?
As discussed in Section 3.2 the competition between TR and TL during pre- pre-training demonstrates negative correlation with the final ICL performance. This motivates us to investigate the influence of pre-training factors the degree of competition. Specifically, we investigate three common factors, Le., model size, dataset size, and data curriculum.
3.3. MODEL SIZE
We investigate the effect of model size on the level of competition between TR and TL. Specifically, we use the Pythia suite (Biderman et al. 2023) for experimentation since this family of models share the same training setting but only differ in the number of parameters.
We first pay attention to their differences the evolution competition. We can observe from Figure 4a| that as the model size increases. the evolving curve of competition keeps moving the left. This means that scaling up model size could make the appearance of competition earlier. The main reason that the learning ability of larger LLMs stronger (Kaplan al. 2020), and they can possess TR and TL more quickly, thus causing the competition between them to occur earlier.
Then, focus on the changes in the average competition intensity. As shown in Figure 4b] the av- erage competitive intensity sharply decreases with the increase of model size, with the exception of Pythia- 1B. This means that scaling up the model size helpful in reducing the overall competition. This be attributed to the fact that an LLM with more parameters has larger capacity, where TR and TL can be allocated with more exclusive resources (e.g., neurons). As a result, although the competition becomes earlier in larger LLMs, the average intensity of competition becomes lower.
3.3.2 DATASET SIZE
In this part, we explore the impact of dataset size on the competition between TR and TL. We conduct experiments using models with roughly the same amount of parameters but trained with different dataset sizes. Specifically, we make the comparison with two sets LLMs: (Pyth and MiniCPM-2B) and (Pythia-6.9B, Amber-7B, and OLMo-BB).

---

Published as a conference paper at ICLR 2025
100
Pythia- 2.88 (300B)
100
MiniCPM-28 (1T)
80
60
40
20
Pythia-6.98
Amber-78 (1.2T)
GLMo (2.5T)
0-
5
10
15
10 Checkpoint index
15
Checkpoint index
Figure 5: Competition evolving process (Rş) of LLMS trained with different dataset sizes.
ICL (MiniCPM-2B)
(CrystalCoder-7B)
53
(Pythia-2.68)
ICL (Amber-7B) 7B)
(MiniCPM-2B)
C (CrystalCoder-78)
50
c* (Pythia-2.881
C² (Amber-78)
12
40
12
45-
10
Checkpoint Index
Checkpoint index
(a) MiniCPM-2B Pythia-2.8B
(b) CrystalCoder-77B vs. Amber-7B
Figure 6: Comparisons in ICL performance and competition intensity (CS). Dashed lines are used to distinguish between different training phases in data curriculum.
Figure illustrates the evolution of competition during pre-training for these two sets of LLMs. can be observed that, for both sets, the evolving curve keeps moving to the right with the increasing dataset size. This means that scaling up the dataset size could postpone the competition. The small dataset, LLMs quickly develop the TR ability since the pre- pre-training knowledge for them to memorize is limited. Mcanwhile, the it primarily involves direct utilization of the information in
possible reason behind this is that, when pre-trained
TI. ability is also easy acquire, context (Singh et al. (2024). As result, the competition between TR and TL. occurs in the early stage of pre-training. With the increase in dataset size, the development of the TR ability becomes slower since there is more pre-training knowledge to memorize. Therefore, more competition happens at the later stage pre- pre-training, making the evolving curve shift the right.
3.3.3 DATA CURRICULUM
In this part, we explore the influence data currieulum on the competition between TR and TL. Here, consider two representative strategies for scheduling data curriculum: (1) quality curricu- lum, which makes arrangements for data with different levels of quality, and (2) domain curriculum, which makes arrangements for data from different domains.
We first pay attention to the influence of quality curriculum on the competition. Specifically, we MiniCPM-28 for the experiment, which utilizes coarse-quality unlabeled data in the first phase and mixes high-quality labeled data in the second phase. We compare its pre- pre-training dynamics with Pythia-2.88 8B, which has similar model size and dataset size but lacks any data curriculum.

---

Published as a conference paper at ICLR 2025
As illustrated in Figure ba the ICL performance of MiniCPM-2B boosts in the second phase with a decline of competition intensity at the end. In contrast, the ICL performance of Pythia-2.88 traps in fluctuations in the final stage of pre-training with increase in competition intensity. From the perspective of competition, the success of quality curriculum can be attributed to the reduced intensity of competition, resulting from less noise in high-quality labeled data.
We then focus on the influence of domain curriculum. Specifically, we use CrystalCoder-7B for the experiment, which utilizes general domain data (i.e., SlimPajama (Soboleva al. (023)) in the first phase, then mixes with code domain data (i.e., StarCoder (Li et al. (023a in the second phase, and mainly uses specific programming language data (i.e., Python and web-related data sampled from StarCoder) in the final phase. Similar to quality curriculum, we compare its pre-training dynamics with Amber-7B, which has similar model size and dataset size but lacks any data curriculum. As shown in Figure (6b] compared with Amber-7B, CrystalCoder-77 exhibits less competition in the second and third phases, along with a larger overall performance improvement. This is due to domain duplication since part of the training data in each phase shares the same domain with the previous phase. Domain duplication can make the training process smoother, which helps reduce competition intensity and leads to better ICL performance.
4 FROM COMPETITION TO COLLABORATION AT INFERENCE TIME
As discussed in Section 3.2 the competition between TR and TL would lead to a decrease in the performance of ICL. To alleviate this, idea is to facilitate their collaboration at inference time. In this section, we first introduce the proposed method and then demonstrate the experimental results.
4.1 ADAPTIVE ENSEMBLE LEARNING
Our analysis in Section 3.2 and 3.3 has shown that regulating the competition between TR and TL can help the model achieve better ICL performance. One potential solution to balancing the TR and TL abilities during pre-training to design special loss functions and train models from scratch. However, this method is costly and cannot be applied to trained LLMs. To save the cost and be applicable trained LLMs, we propose an adaptive ensemble learning approach to fuse TR and TL abilities at inference time.
Specifically, we first select two checkpoints with the best ability of TR and TL respectively. and then integrate their probability distributions by ensemble learning:
arg max [w.,Pr,,d (ulæ) + wyPrth yEy
(5)
where Pr, (z/(z) and Pr. (y/z) denote the probability for the TR and TL models to predict the label y under the random and abstract settings respectively, and w, and wy denote their weights respectively. Here, considering that the contribution of TR and TL abilities to ICL usually are not equal (Lin & Lee 2024), propose adaptive weights based on their performance follows:
Accfand
Acc b
Wr
(Acefand - b) + (Acc -6)
(Acc/and - 6) (Aco abs -))
where Aad. is the performance of the TR model under the random setting, Accths is the perfor- mance of the TL model under the abstract setting, and is the performance of random guessing.
4.2 EXPERIMENTAL SETTING
To comprehensively validate the effectiveness of our method, we consider three different combi- nations of TR and TI. models for fusion: (1) only the backbone model (e.g., Pythia-1B), (2) two models with similar training setting (e.g., Pythia- IB and Pythia-2.8B), and (3) two models with different training settings (e.g., Pythia- and MiniCPM-2B). We select checkpoints with the best performance for the required ability. Other settings are the same as Section 3.1
In Table D we compare our method with four types of baselines: (1) the backbone TR or TL model for fusion ("Small models"), (2) LLMs with more parameters than the sum of TR and TL models Laage models"), ) and (3) model parameter fusion with fixed and adaptive weights calculated using the same method ours ("Prameter fusion").. All the baselines are tested in the gold setting.
8

---

Published as a conference paper at ICLR 2025
Table 1: Averaged accuracy and TFLOPs across 16 datasets. "Fixed" and "adaptive" denote fixed and adaptive fusion weights. We highlight the best performance among various model selections for TR and TL. Numbers marked with indicate that the improvement is statistically significant compared with baselines (t-test with p-value < 0.05). More results are shown in Table
Model
Parameters
TFLOPs
Accuracy
Large models
Amber-7Brc.
7B
9.38
50.0840.18
OLMo-7B
7B
8.79
52.1040.19
Baichuan2-7B
7B
8.18
52.77+0.16
CrystalCoder-7BB.
7B
10.17
55. 55.66
Pythia-12B
12B
15.51 10.57
52.77
Small models
Pythia- IB eCL
1B
1 1.24 Lo.15
44.5510.33 10.33
Pythia-2.8B MiniCPM-2B1eu.
2.8B
3.601..20
10.38 10.32
2.7B
3..4
50.08
Parameter fusion of small models
Pythia- Wythia-IB IB Pythia- 1B, (fixed)
46.631..26
1B
1.2 1.24
Pythia 1B + Pythia- IBT. (adaptive)
45.61
2ythia-2.8B.. + Pythia-2.8B. (fixed)
47.57+0023
2.8B
3.60 3.60
Pythia- + Pythia-2.8B Tu (adaptive)
47. 47.1840.18
MiniCPM-2Bre++ MiniCPM-2B: (fixed)
52.18
2.7B
3.24
MiniCPM-2B MiniCPM-2B-. (adaptive)
52.06
fusion of small models (Ours)
Pythia- B + Pythia- (fixed)
56. 56.16.0.40
2B
2.48 10.21
Pythia- B m Pythia- IB; (adaptive)
56.25 10.38
1ythia-IBye'
Pythia-2.888 -. (fixed)
56.62..0.41
Pythia- Aahia--BBra++ Pythia-2.8B-r (adaptive)
56.834.0.4
3.8B
4.84
aythia-IBT.++ Pythia-2.8B.. (fixed)
55.2340.47
Pythia-IB,
P2thia-2.8B (adaptive)
55.39
Pythia-IB MiniCPM-2BBr.. (fixed)
55.21
Pythia-IB
MiniCPM-2B-r. (adaptive)
55.31
3.7B
5.08 Lo.2
Aithia-BBr.+ MiniCPM-2B (fixed)
54.31+0.63
Pythia- + MiniCPM-2Brgg (adaptive)
55.85 10.75
4.3 RESULTS
As shown in Table our proposed method can significantly boost performance compared to the single model. In addition, such an improvement is consistent across various model combinations, demonstrating that our method is widely applicable. To our surprise, by using our method, two small models together can even outperform larger models with lower inference overhead, despite their total parameters being less than half of the larger ones. It suggests that our method can effectively and efficiently fuse the abilities of TR and TL. to achieve better ICI. performance.
Furthermore, to verify the effectiveness of each component in our method, we conduct the ablation study. We substitute the best checkpoints with random/empty ones (Table 3 and set the weights of TR and TL to the same ("fixed" in Table []. respectively. We can observe that removing any design would lead to a decrease in performance. of our approach. In addition, the selection of checkpoints with the best TR/TL ability seems to be more important, which yields a larger performance drop after being removed. Checkpoints with the best TR/TL. ability are more diverse in their predictions, which is important for successful fusion.
demonstrates the effectiveness of all the components

---

Published as a conference paper at ICLR 2025
5 DISCUSSION
Why does the competition between TL and TR occur? According to our observations and ex- isting literature, we think that the competition between TL and TR may originate from the limited resources in models (e.g. neurons layers). As shown in existing work (Kaplan et al., 2020; Tiru- mala et al. 2022), larger language models can memorize more training data and obtain smaller test loss with the same training steps. This indicates that the amount of resources (e.g., neurons or lay- ers) is important for the capacity of LLMs. We also investigate the influence of model size on the competition in Section 3.3.I] the results demonstrate that the average competitive intensity sharply decreases with the increase of model size. Thus, we infer that the competition could result from the limited resources in models, such as neurons layers, being dynamically allocated between TL. and TR during pre-training.
Limitations. When investigating the influence of specific pre-training factor in Section 3 we do our best to keep other factors as consistent as possible. However, it is very challenging to run care- fully controlled experiments due to the lack of pre-trained models in many settings. For example, when investigating the impact of dataset size, we only find that models that have roughly the same amount of parameters, but cannot control other factors like data mixture and optimization. As future work, will try to collect enough computational resources and conduct fully controlled pre-tra trainiii from scratch to ensure that only one factor differs to investigate its influence.
RELATED WORK
Our work is closely related to the studies on the mechanisms of ICL and model fusion.
The Mechanism of ICL. Existing work primarily explores the mechanisms of ICL from the pre- training and inference stages of LLMs. Some work discusses how ICL emerges from pre-training by conducting analysis pre-training factors like data (Chan et al.) [022 Reddy (0233) and opti- mization (Singh et al. 2024] Annnd et al. 2024). Other work (Pan et al. 2023 Min et al. 2022 Dai et al., 2023) studies the operating mechanism of ICL inference time. Researchers empirically find two main abilities in ICL: task recognition (TR) for recognizing the task and utilizing pre-trained priors of LLMs (Min et al., 2022) and task learning (TL) for learning from demonstrations (Dai et al. 2023). In this paper, we explore how TR and TL affect the emergence of ICL by examining the pre- pre-training dynamics of LLMs.
Model Fusion. Model fusion aims to enhance performance by combining the strengths of multiple models (ii et al. 2023b). One line of work aims to reduce the difference among different models from perspectives like mode connectivity (Nagarajan & Kolter, 2019) and alignment (Tatro et al.) 2020 Another line of work studies how to leverage the diversity among models through techniques like weight average (Wang al. 2019) and ensemble learning (Sagi & Rokach 2018). In this paper, we propose adaptive ensemble learning to fuse TR and TI. and achieve better ICL. perforrmne...
CONCLUSION
In this paper, we presented the first study of the competitive relationship between TR and TL, and quantified its effect on the emergence of ICI. With specially designed metrics, we found that this competition widely exists in existing I.MMs, and the competition intensity negatively correlated with the ICL performance. Then, conducted detailed analysis of several pre-training factors (i.e., model size, dataset size, and data curriculum) to demonstrate possible ways to regulate the competition. Furthermore, we proposed simple yet effective method to better integrate TR and TL at inference time. Through adaptive ensemble learning, the performance of ICL can be significantly boosted, enabling two small models to outperform a larger one with more than twice the parameters.
Overall, our work provides novel approaches and insights to study and understand the underlying mechanism of ICL., which is worth deep exploration for improving the capacity of L.MSs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by National Natural Science Foundation of China under Grant No. 92470205 and 62222215, Beijing Municipal Science and Technology Project under Grant No. Z231 103000000333009 Beijing Natural Science Foundation under Grant No. L233008, and the Out- standing Innovative Talents Cultivation Funded Programs 2022 of Renmin University of China. Xin Zhao is the corresponding author.
REFERENCES
Suraj Anand, Michael A Lepori, Jack Merullo, and Ellie Pavlick. Dual process learning: Con- trolling use of in-context vs. in-weights strategies with weight forgetting. arXiv preprint SKKi:24006000553. 2024.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In SemFval NAACL-HLT, pp. 54-6.. Association for Computational Linguistics, 2019.
Stella Biderman. Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling. In ICML., volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. In EMNLP, pp. 632-642. The Association for Computational Linguistics, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang. Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- ing in transformers. Advances in Neural Information Processing Systems, 35:18878-188 18891,, 2022.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 4005-4019, 2023.
William B. Dolan and Chris Brockett. Automatically constructing corpus sentential paraphrases. In /WP@ IJCNLP. Asian Federation of Natural Language Processing, 2005.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkin- son, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani. Mitchell Worts- man, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo. Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. CoRR, Abs/2402.00884 2024.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang. Yuan Yao. Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai. Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.06395, 2024a.

---

Published as a conference paper at ICLR 2025
Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, et al. Yulan-mini: An open data-efficient language model. arXiv preprint arXhh:2412.17745 2024b.
Joonwon Jang, Sanghwan Jang, Wonbin Kweon, Minjin Jeon, and Hwanjo Yu. Rectifying demon- stration shortcut in in-context learning. Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 4294-4321, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08330 2020.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR. AAAI Press, 2012.
Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, II Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023a.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprini arXiv:2309. 2023b.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell base Ilms: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552 2023.
Ziqian Lin and Kangwook Lee. Dual operating modes of in-context learning. arXiv preprint arXiv:2402.18819 2024.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He. Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. LLM360: towards fully transparent open-source llms. CORR, abs/2312.06555 2023.
Zhengzhong Liu, Bowen Tan, Hongyi Wang. Willie Neiswanger, Tianhua Tao. Haonan Li, Fajri Koto, Yugi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Liqun Ma, Liping Tang, Nikhil Ranjan Ranjan, Zhuang Yonghao, He Guowei, Renxi Wang, Mingkai Deng, Robin Algayres, Yuanzhi Li, Zhiqiang Shen, Preslav Nakov, and Eric Xing. Llm360 k2-65b: Scal- ing up fully transparent open-source Ilms, 2024. URL htttp::: 11m360 ai/paper2. pdf
Pekka Malo, Ankur Sinha, Pekka J. Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Assoc. Inf. Sci. Technol., 65(4); 782-796, 2014.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In IREC, pp. 216-223. European Language Resources Association (ELRA), 2014.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, pp. 11048- 1064. Association for Computational Linguistics, 2022.
Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval-2018 task 1: Affect in tweets. In SemEval@NAACL-HH pp. 1-17. Association for Computational Linguistics, 2018.
Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. ETHOS: an online hate speech detection dataset. CoRR, fas/20066083288, 2020.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generaliza- tion in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" cn-context: Disentangling task recognition and task learning. In ACL (Findings), pp. 8298-83199 Association for Computational Linguisties, 2023.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in in-context classification task. In The Twelfth International Conference on Learning Representations, 2023.
Omer Sagi and Lior Rokach. Ensemble learning: survey. Wiley intendisciplinary reviews: data mining and knowledge discovery, 8(4):e1249) 2018.
Elvis Saravia, Hsien- Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: contex- tualized affect representations for emotion recognition. In EMNLP, pp. 3687-3697. Association for Computational I Linguistics, 2018.
Emily Sheng and David C. Uthus. Investigating societal biases in a poetry composition system. 2020.
CoRR,
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hes- tness, and Nolan Dey. SlimPajama: plicated version of RedPajama. of jama s @limpajanas a waw.aa--t-te------eeeeeee
A 627B token cleaned and dedu-
and
June 2023.
URL
https: http:///gggggg...coooootttteeeee
3limPa. jama- 627p
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pp. 1631-1642. ACL, 2013.
Norman Tatro, Pin- Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Op- timizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300-1533 2020.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
Ellen M. Voorhees and Dawn M. Tice. Building a question answering test collection. SIGIR, pp- 200-207. ACM, 2000.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Represen- tations. 2019.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Huang, Denny Zhou, et al. Larger language models do in-context learning differently, arXiv preprint arXiv:2303. X33::2303.038466 2023.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang. Dong Yan, Fan Yang. Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin. Nuolan Nie, Peidong Guo, Ruiyang Sun. Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng. Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xue- hai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models. CoRR, abs/2309.10300 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 20t.:23333..2223,, 18223, 2023.
13

---

Published as a conference paper at ICLR 2025
(a) Different numbers of intermedi- ate checkpoints.
(b) Different numbers of examples in the demonstration.
(c) Different types of abstract la- bels.
Checkpoint
32
Examples
4
Abstract label
Symbels
Pythia-6.9B
53.12
Pythia-6.9B
18.75
25.00
37.50
Pytha-66998
37.50
18:75
37.50
OLMo-7B 50,00
37.50
43.75
OLMo-7B
5.00
37.50
37.50
OLMo-7B
|
37.50
50.00
43.75
MiniCPM-2B
62,50
56.25
37.50
MiniCPM-2B
43.75
31,25
56.25
MiniCPM-26
56.25
43.75
50.00
A TASKS AND DATASETS
We conduct experiments four types of tasks: Sentiment Analysis, Topic/Stance Classifica-
tion. Toxicity Detection, and Natural Language Inference/Paraphrase Detection. For Sentiment Analysis, we use datasets including SST-2 (Socher al.) (201)). financial phrasebank (Malo et al. 2014), emotion (Saravia et al.) 2018), and poem sentiment (Sheng & Uthus 2020). For
Topic/Stance Classification, we utilize TREC (Voorhees & Tice 2000), sweet_eval_atheine and
evect_eval.feminis (Mohammad et al.) 2018 Basile et al.] 2019). For Toxicity Detection, we in-
clude tweet_eval_hatc ethos_race race, ethos-gender ethos_ national_origin. and ethos relosine (Mollas ct al. 2020). For Natural Language Inference/Paraphrase Detection, employ SICK (Marelii
et al.
2014), SNLI (Bowman et al. 2015), WNLI (Levesque et al. 2012), and MRPC (Dolan &
Brockett 2005).
We follow Min et al. (2022) to select samples from the training set as demonstrations, Additionally, randomly sample 300 examples the development set for validation in Section 4 and another 1000 examples as the test set for evaluation in all experiments from the development set.
B MORE EXPERIMENTS
B.1 THE NUMBER OF INTERMEDIATE CHECKPOINTS
In the paper, we use 16 checkpoints in addition the final one. In this part, we conduct experiments using different numbers of checkpoints (i.e., 8 and 32). We report the average competition ratio across 16 datasets and random seeds. Table 7a shows that the number of checkpoints does not affect the experimental results. They consistently demonstrate that there is a competitive relationship between TR and TL during the pre-training process.
B.2 THE NUMBERS OF EXAMPLES IN DEMONSTRATION
In the paper,
16 randomly sampled examples demonstrations, To explore the impact of the number of examples, we report the average competition ratio with other numbers (i.e., and 8) of demonstrations. As presented in Table 7b it can be observed that the number of examples does not affect the competitive relationship during the pre-training process.
B.3 THE TYPE OF ABSTRACT LABELS
In the paper, we utilize symbols in the abstract setting. In this part, we follow Pan et al. (2023) to use other types of semantically unrelated labels (i.e., numbers and letters). Table 7c shows the average competition ratio by using different labels. It indicates that, regardless of the choice of semantically unrelated labels, the conclusions are consistent with the abstract symbols.
C RELATED WORK
Our work is closely related to the studies on the mechanisms of ICI. and model fusion.
The Mechanism of ICL. In-context learning (ICL) has garnered significant interest as a core ca- pability of large language models (LLMs), enabling them to perform tasks without fine-tuning by simply leveraging input demonstrations. Existing studies investigate ICL mechanisms through both pre-training and inference dynamics. Some work discusses how ICL emerges from pre-training by
14

---

Published as a conference paper at ICLR 2025
2023) and optimiza- tion (Singh et al. 2024 Anand et al. 2024). Other work (Pan et al. 2023 Min et al. 2022: Dai et al, 2023) studies the operating mechanism of ICL at inference time. Researchers empirically find two main abilities in ICL: task recognition (TR) for recognizing the task and utilizing pre-trained priors of LLMs (Min al. 2022) and task learning (TL.) for learning from demonstrations (Dai et al.) 2023). The interaction of TRR and TL is crucial for effective ICL, and understanding this interaction has been focus of empirical and theoretical research Pan et al. (2023): Wei et al. (2023). While these studies provide valuable insights, they often examine TR and TL in isolation. 'hhe interplay between these abilities, particularly how they emerge during pre-training, remains underexplored. In this work, we bridge this gap by investigating how TR and TL affect the emergence of ICL through examining the pre-training dynamies of LLMs.
conducting analysis pre-training factors like data (Chan al. 2022 Reddy
Model Fusion. Model fusion aims enhance performance by combining the strengths of mul- tiple models (Li et al,, 2023b). One line of research focuses on minimizing discrepancies among models through strategies like mode connectivity (Nagarajan & Kolter 2019) and alignment tech- niques (Tatro et al. 2020). Mode connectivity methods (Nagarajan & Kolter 2019) align model rep- resentations by finding low-loss pathways in parameter space, facilitating smoother transitions and improved generalization. Alignment techniques (Tatro et al. (020) focus on harmonizing model pre- dictions or representations, often through methods like knowledge distillation contrastive learn- ing. These approaches aim to create more consistent models that are easier to integrate. Another line of research capitalizes on diversity to enhance ensemble performance. Techniques such as weight averaging (Wang et al. 2019) and ensemble learning (Sagi & Rokach] 2018) combine models with varying inductive biases or training histories. By aggregating predictions or interpolating model parameters, these methods harness complementary strengths to achieve robust and adaptive perfor- mance. The potential of model fusion to improve ICL performance a promising but underexplored area. While TR and TL abilities have been analyzed independently in individual models, their inte- gration through fusion strategies remains unaddressed. In this paper, we introduce a novel adaptive ensemble learning approach that combines TR and TL., demonstrating its efficacy in enhancing ICL. across diverse tasks.
15

---

Published as a conference paper at ICLR 2025
Table 2: Averaged accuracy and TFLOPs across 16 datasets. "Fixed" and "adaptive" denote fixed and adaptive fusion weights. We highlight the best performance among various model selections for TR and TL. Numbers marked with indicate that the improvement is statistically significant compared with baselines (t-test with p-value 0.05).
Model
# Parameters
TFLOPs
Accuracy
Large models
Amher-7Bgeu.
7B
9.98
50.08 10.18
OlMo-7Baea.
8.79
52. 52.1010.19
Baichuan2-7B
7B
8,18
52.77+0.16
CrystalCoder-7B.
7B
10.17
55.66 55.6640.31
Pythia-12B re..
12B
15.51
52. 52.7740.21
Small models
Pythia-1B
1.24
44..55+0.33
Pythia 2.8ia-2.8Brec
2.8B
3. 20.20
45.38 40.a2
MiniCPM-2B:c..
2.7B
2..4
50.0840.16
Parameter fusion of small models
Pythia-IB 1Bah Pythia 1Br.. (fixed)
46. 46.6340.26
IB
1.24
Pythia BB Pythia- IB (adaptive)
45 45.6140.29
Pythia-2.85 2.8aa-22.BBB 8B r Pythia-2.8B. (fixed)
47.5740.23
2.8B
0.
Pythia-2. r Pythia-2.8B-. (adaptive)
47.1840.13
MiniCPM-2B MiniCPM-2B-.. (fixed)
52.1840.26
2.7B
0.84
MiniCPM-2B + MiniCPM-2B-., (adaptive)
52.0640.27
Logit fusion of small models (Ours)
Pythia-IB::a+ Pythia-1B:, (fixed)
56.1610.40
2.48
Pythia-1B..g Pythia-IB:,. (adaptive)
56.25
Pythia-IB:x*+ 2.8 (fixed)
56.62.1.41
PBthia-1B:a.+
Pyth (adaptive)
56.83
3.8B
1..4
Pytia---B (fixed)
55.23
Pythia-1B...
Pythia-..8 (adaptive)
55.39+0.40
Pythia- MiniCPM-2B (fixed)
55.21 4:0.43
Pythia-1B.s.+ MiniCPM-2B.1, (adaptive)
55.31:40.34 4:0.34
3.7B
5.08.4.26
MiniCPM-2B- (fixed) Pythia-
54.31
Pythia-
MiniCPM-2B (adaptive)
55.85
PB 8B MiniCPM-2B-:. (fixed)
53.55 53.588to.ss
PB Pythia-2.BB.a MiniCPM-2B (adaptive)
55.00 55.00+1.75
Pythia-IB:x PyrbiaB-.. Pyth MiniCPM-2B (fixed)
53.67
Pythia-IB:a++ Pythia-2.8B:..++ MiniCPM-2B
(adaptive)
56.32++042
Pythia-IB.u.+ 2yth MiniCPM-2B.
(fixed)
53. 03.0044..8
Pythia-IB..-1 2,t MiniCPM-2B
(adaptive)
55. 0..226
6.5B
8.68
Pythia-IB-..+ 2yth MiniCPM-2B
(fixed)
57 57.2444.42
Pythia-IB..+ 2yth MiniCPM-2B
(adaptive)
57. 37.35-+0.28
Pythia-1B...+ Pythia-2.8B..+ MiniCPM-2R
(fixed)
55.94 1
Pythia-1B::2 2. MiniCPM-2B
(adaptive)
56.45
Pythia-IB Aibia-IB-aa IB P.th MiniCPM-2B
(fixed)
58 18..054..22
Pythia-IB-..f PB MiniCPM-2B...
(adaptive)
58.27 10.40
16

---

Published as a conference paper at ICLR 2025
Table Ablation study for model fusion. We highlight the best performance among various model selections for TR and TL. "Random" means that the checkpoint is randomly selected, while "Best" means that the checkpoint has the best performance for TR/TL. Numbers marked with indicate that the improvement is statistically significant compared with others (t-test with p-value < 0.05).
Models for fusion
Model selection for TR
Model selection for TL
Accuracy
Be-test
17.47:10.19
TL-Best
44.63-2032
Random
Random
52. 52.6620.44
TR-Best
Random
53.52.20.22
TR: Pythia- IB
Random
Lestest
54. 54.1940.04
TL: Pythia-IB
TR-Best
ICL Best
55.53
ICL-Best
TL- le-Best
54.7640.29
ICL-Best
ICL Best
54.5620.40
TR-Best
TL-Best
56.25
TR-Best
47.47
TL-Best
12.05
Random
Random
10:0.1
TR-Best
Random
50.76.10.2
TR: Pythia-
Random
Best
55.42
TL.: Pythia-2.8B
TR-Best
ICL-Best
54. 54.902..44
ICL-Best
Best
55.93-0.32
ICL.BBstt
ICI Best
54. 1:.129
TR-Best
TL-Best
56.831..44
TR-Best
44.28.
TL-Best
44.63
Random
Random
52.48.40.00
TR-Best
Random
53.89+0.3s
TR: Pythia-2.8B
Random
TL-Best
54..99
TL: Pythia- BB
TR-Best
CL-BBstt
54..6220.43
ICL-Best
TL-
54.9011..0
ICL-Best
ICL Best
53.5640.34
TR-Best
Best
55.39%...40
TR-Best
47.4720.19
TL-Best
47.4920.24
Random
Random
53.90 53.99
TR-Best
Random
79..3
TR: Pythia- 1B
Random
TL-Best
54.51
TL: MiniCPM-2B
TR-Best
ICL-Bes
54.81
ICL-Best
TL- Best
55.02220.17
ICL-Best
ICL-Best
54.55_20.22
TR-Best
TL-Bes
55.31
TR-Best
45.10.0.15
TL-Best
11.63.1..35
Random
Random
52.07
TR-Best
Random
53.61=0.11
TR: MiniCPM-2B
Random
Te-Best
53. 73
TL:: Pythia-1B
TR-Best
ICL-
55.76 76..33
ICL-Best
TL-Best
54.88
ICL-Best
ICL-
54.3940.44
Best
Tl- Best
55.85 +0.73
17


Reviews:
[{"review_id": "DKl9QxXxCn", "reviewer": "Reviewer_zV49", "summary": "This work explores the dynamics between two abilities of in-context learning (ICL) in large language models (LLM), i.e., task recognition (TR) and task learning (TL). This paper argues that TL and TR are indeed competitive during pre-training. To show this quantitatively, the authors proposed a metric called Competition Intensity that measures the extent to which one ability exceeds the other. The paper suggests that the Competition Intensity is negatively correlated with the ICL performance, and that regulating the Competition Intensity via specific choices of pre-training factors (model and dataset size, and data curriculum) is crucial for improving ICL performance. By using this hypothesis, the paper proposes a technique to fuse small models.\n\nThe contributions are:\n1) Explore the relationship between TR and TL and their impact on ICL performance,\n2) Analyze the influence of the pre-training factors on the competition between TR and TL, and\n3) Propose an approach to fuse small LLMs more effectively.", "questions": "- I am not sure why the TL and TR curves appear to have identical data points in both plots of Figure 1.\n- Could you discuss potential mechanisms or hypotheses on why the competition between TL and TR occurs, based on your observations or existing literature\n- Apart from TL and TR, discuss any limitations in your experimental design that might not fully control for other factors that might affect ICL. Please suggest potential ways to address these in future work.\n- In Figure 6a, it seems $C_i^S$ tend to increase in the second phase. Could you provide more detailed analysis of the relationship between $C_i^S$ and ICL performance in this specific case, and discuss any potential confounding factors.\n- In Figure 6b, there is no clear evidence that CrystalCoder-7B experiences less competition than Amber-7B. Could you provide quantitative evidence supporting your conclusion about this figure or clarify if there are specific aspects of the figure that led to this interpretation.\n- In Table 1, it would be helpful to report the standard deviation of the accuracies and TFLOPs. I would also recommend to include a statistical test to evaluate the significance of the obtained results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-written and easy to follow\n- The analysis of TR and TL behaviours and their effect on ICL performance is interesting\n- The fusion of small models performs comparably to larger models, making a step toward the feasibility of training LLMs with fewer resources", "weaknesses": "- The novelty of the methodology section is limited.\n- There is no significant improvement observed in Table 1 compared to other fusion models, small models, and large models in terms of Accuracy\n- Limited discussion, limitations and future work. It would be interesting to explore how the fusion technique performs on each of the 16 individual datasets, and possibly identify common characteristics of datasets where the technique performs well/poorly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9d1jZEMjkG", "reviewer": "Reviewer_p21T", "summary": "The authors examine the competition between Task Learning (TL) and Task Recognition (TR) throughout pretraining, and its effect on the in-context learning (ICL) capabilities of LLMs. They show that when there exists a competition, this usually leads to stagnation in ICL capabilities or a decrease. Motivated by this, they propose an ensembling technique to between different checkpoints of the same (or different) models that performed best in TL or TR. They show that, this combination can lead to performance improvements in ICL that can rival models with even larger model parameters than the sum of the fused smaller models.", "questions": "- Are there a few benchmarks that skew the average to Pythia-1B p-fixed instead of Pythia-1B p-adaptive? We do not observe p-fixed performing better than p-adaptive in other scenarios\n- Could you compare your proposed ensembling method to other ensembling methods? (You can choose Pythia-1B + Pythia-2B final checkpoints for baselines to compare to your selection of checkpoints for ensembling)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The paper shows a clear sign of competition between TR and TL, and its effect on ICL improvement throughout pretraining\n- The authors clearly defined each metric, and defended their relevance\n- They have numerous ablation studies to show the effects of factors (such as dataset size and curriculum training) to ICL (or TR/TL competition)\n- They have ablation studies on various ensembling techniques to show their performance improvement is not just due to model fusion, but something more systematic. However, this may require more studies (see weaknesses)", "weaknesses": "- I believe the results on Table 2 can be better represented. Especially choosing a \"random\" checkpoint can be very misleading, and frankly I don't think it's necessary to look at such data point. I'm interested in the {TR or TL}_best + ICL_best checkpoints for the same models. And, ICL_best_model_1 + ICL_best_model_2 when ensembling different models.\n- I also believe p-adaptive should be better motivated. I don't see the performance improvements big enough, and also Pythia-1B models, p-fixed performs significantly better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqNKXRfxK1", "reviewer": "Reviewer_za13", "summary": "This paper investigates how task recognition (TR) and task learning (TL) abilities influence in-context learning (ICL) performance in large language models (LLMs). Building on prior work by Pan et al. (2023) and Lin & Lee (2024), the authors assess TR and TL by constructing Gold, Random, and Abstract input-label pairs to isolate these abilities. They introduce several new metrics based on ICL performance across a sequence of LLM checkpoints:  (1) **competition indicator** $C_i^H$; (2) **competition intensity** $C_i^S$; (3) **cumulative intensity**, which is used to infer the dynamic interplay between TR and TL. Through a series of empirical evaluations, the authors demonstrate key insights into how these abilities interact. Additionally, they propose an adaptive ensemble method that fuses two smaller models to improve ICL performance, suggesting a promising direction for enhancing context-based learning in LLMs.", "questions": "- The paper offers very little discussion regarding the choice of the number of checkpoints. Does the method’s performance remain robust when applied with varying numbers of checkpoints? An ablation study would be beneficial to address this.\n\n- It would be helpful if the authors could elaborate more on the intuition behind the dynamics of pre-training and clarify how studying this aspect could directly benefit in-context learning (ICL).\n\n- A key aspect of evaluating the method is the reliability of its modeling of competition intensity dynamics $R_i$. How can we systematically assess the quality of the proposed $R_i$ score, and in what ways does this score contribute to improving ICL capability in practice?\n\n- In the ablation study, it would be beneficial to include results for a non-adaptive ensemble baseline, as well as standard ensemble models incorporating more than two LLMs.\n\n- The proposed metrics rely on checkpoint fluctuations to measure task learning (TL) and task robustness (TR). Would the method remain consistent and valid if checkpoints were sparse? It would be helpful to include discussion or experiments around this. \n\n-  It would be helpful if standard deviations could be added to the tables and figures from the paper. Could the authors provide statistical test results to determine whether the observed fluctuations are significantly different from random noise? \n\n- For Fig 3, the relationship between ICL performance and $R_i$ sometimes misalign in certain intervals. Is it possible to quantify the correlations between them and confirm that the positive relationship statistically hold? Additionally, a detailed discussion on potential reasons for those observed misalignments could provide valuable insights.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- The paper brings a fresh perspective to ICL by examining the roles of task recognition (TR) and task learning (TL), shedding light on how these two abilities interact and shape ICL performance.\n\n- The authors introduce some new metrics, including the competition indicator and cumulative intensity score, which add depth to our understanding of TR and TL dynamics over time. \n\n- Through a comprehensive experimental setup, the paper examines model behavior across multiple checkpoints, allowing for a nuanced view of how TR and TL evolve during training. \n\n- The adaptive ensemble method presented is somewhat novel, selecting models based on their TR and TL strengths. By leveraging each model’s unique abilities, this approach suggests a practical way to optimize ICL performance through complementary ensemble strategies.", "weaknesses": "- The clarity of the paper could be improved to make key concepts more accessible. For instance, terms like \"dynamics of pre-training\" are introduced without clear definitions and make it challenging for readers to grasp the intended meaning. Also, it's encouraged to clearly highlight what are the key assumptions and intuitions more clearly in the paper.\n\n- The paper also  lacks substantial technical novelty and depth of insight into the ICL problem. The new metrics proposed in for measuring competition indicator, intensity and dynamics heavily rely on an assumption that these metrics could be derived from the change of TL and TR scores from consequent checkpoints.  I'm not convinced that the  change of performance on checkpoints could reflect the capability of TL or TRs. It is hard to figure out what kind of training the model has gone through for each new release and version of checkpoints could be arbitrary too. Especially for large-scale models where checkpoints are not saved at every step.  How could the $\\triangle$TL and $\\triangle$TR evaluated on consequent checkpoints robustly tell the true capability of TL and TR, which is of interest to the audience? This reliance could limit the broader applicability and scalability of the proposed method.\n\n- The cumulative intensity score $R_i$  used to track the dynamics of competition intensity over time, raises some concerns. Since it simply aggregates $C_i^S$  values, which are strictly positive, this score inevitably increases, potentially making it less informative as a dynamic metric.\n\n- Key figures, such as Figure 1 and Figure 6, lack error bars and the tables do not come with standard deviations, making it difficult to evaluate the robustness of the observed trends. In the paper, the authors discuss a lot about the  \"fluctuations\" of the checkpoints, which in fact often can be somewhat random, and without error bars, it’s not able to tell whether these patterns are consistent or potentially artifacts.\n\n- One fundamental assumption in this work is that the relationship between TR and TL can be inferred directly from changes in ICL ability across checkpoints. However, training conditions and checkpoint improvements are not always straightforward, and observed changes in ICL ability may not consistently reflect meaningful performance shifts. A more detailed discussion validating this assumption would strengthen the technical foundation of the method.\n\n\n - Sections 3 and 4 lack a cohesive connection, and the title of Section 4 may be somewhat misleading, as it suggests that TL and TR no longer compete. Although Section 4 introduces an ensemble model by selecting the best-performing models for TR and TL, it remains unclear whether this ensemble approach results in the highest combined TR and TL scores compared to individual checkpoints. The two chosen models are combined using simple weights inferred from their respective accuracy, but further clarity is needed on the effectiveness of this weighting approach. The experiments would be more robust if the authors included comparisons with other ensemble methods and conducted an ablation study on using fixed, non-adaptive weights (e.g., 0.5, 0.5).\n\n- In Figure 3, the authors plot ICL performance and $R_i$  curves together, showing increasing trends. However, the increase and decrease patterns do not always align, raising questions about the strength of the correlation between ICL performance and $R_i$. More statistical analysis or validation would help clarify the relationship and strengthen the visual interpretations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxxeJ7Rsal", "reviewer": "Reviewer_Q4EL", "summary": "The paper studies how two mechanisms related to in-context learning (ICL), task recognition (TR) and task learning (TL), develops over the course of LLM pre-training. Through the experiments, the authors find that there is competition between the TR ability and the TL ability through the pre-training process. The authors further propose a measure to quantify such competition, and fine a negative correlation between the magnitude of such competition to the final ICL performance. The paper also studies how different factors, including parameter count, dataset size, and data curriculum affects the magnitude of the competition and ICL performance. Finally the paper proposes a method to ensemble checkpoints with best TR performance and best TL performance to get better ICL performance, outperforming the ICL performance of a larger model with more than twice the parameters.", "questions": "- In lines 371-373 where the authors discusses the results shown in figure 6(a), the author attributes higher ICL performance of the mini-CPM model to reduced competition. However, from the figure we can clearly see that the competition intensity of mini-CPM 2B is higher than that of Pythia-2.8B on the right side of the dashed line, which seems to contradict with the author's reasoning. I would like to hear about more explanation on this.\n- I would like to hear about more reasoning on how the value of the adaptive weights $w_r$ and $w_l$ is determined, and more experiments with other possible value of $w_r$ and $w_l$.\n- Regarding the results in figure 4b, does the authors have any possible explanation on why the Pythia-1B model does not follow the general trend? \n- Related to the point above, I want to note that different Pythia models have different width-depth ratios. If possible, it might be helpful to further investigate the effect of the width and depth of the model separately. The fact that the width-depth ratio of the model might have a big effect on the TR/TL competition also raises a small concern about whether the comparison in table 1 is actually the most fair, if the 1B+1B model has a different width-depth ratio from the other bigger models. I would hope to get more info about this.\n- Other than the method proposed in section 4 of the paper, are there other potential implications of the TR/TL competition phenomenon?\n- Does the TR/TL competition phenomenon only occur in pre-training, or it also occurs in fine-tuning?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main insight of this paper that there exists competition between TR and TL and such competition is correlated with worse ICL performance is very novel and interesting to me. Discovering this insight and investigating how different pre-training factors affect TR/TL competition already makes it a good paper, and the fact that the authors propose a well motivated method to improve ICL performance makes the paper a great one.\n- The presentation of the paper is good. The authors provide a clear outline in the beginning and smoothly dives into each part of the story.\n- The authors provides anonymous code which allows easy reproduction of the experiment results.", "weaknesses": "- The related work section is a little bit limited. If constrained by space in the main text, it can be elaborated in the appendix. \n- Many experiments in section 3 involves comparisons between only 2-3 models (many of them are trained with datasets and architectures). While I understand that it is very challenging to run carefully controlled experiments due to the lack of pre-trained models in many settings, this still limits the significance of the conclusions.\n- See more potential concerns in the \"questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)