Why In-Context Learning Models are Good Few-Shot Learners?
Authors: Shiguang Wu
Yaqing Wang
Quanming Yao
Venue: ICLR 2025 Poster (2025)
Keywords: In-Context Learning
Meta-Learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: We explore in-context learning (ICL) models from a learning-to-learn perspective. Unlike studies that identify specific learning algorithms in ICL models, we compare ICL models with typical meta-learners to understand their superior performance. We theoretically prove the expressiveness of ICL models as learning algorithms and examine their learnability and generalizability. 
Our findings show that ICL with transformers 
can effectively construct data-dependent learning algorithms instead of directly follow existing ones 
(including gradient-based, metric-based, and amortization-based meta-learners). 
The construction of such learning algorithm is determined by the pre-training process, as a function fitting the training distribution, which raises generalizability as an important issue.
With above understanding, we propose strategies to transfer techniques for classical deep networks to meta-level to further improve ICL. As examples, we implement meta-level meta-learning for domain adaptability with limited data and meta-level curriculum learning for accelerated convergence during pre-training, demonstrating their empirical effectiveness.

Published as a conference paper at ICLR 2025
WHY IN-CONTEXT LEARNING MODELS ARE GOOD FEW-SHOT LEARNERS?
Shiguang Wu
Yaqing Wang
Department of Electronic Engineering, Tsinghua University
Beijing Institute of Mathematical Sciences and Applications wangyaqing ng@b @b msa. cn
wsg23emails 1s. ts inghaa. hggua.edu.... on
Quanming Yao"
Department of Electronic Engineering, Tsinghua University State Key laboratory of Space Network and Communications, Tsinghua University qyacaaltss iyaoaa@tsinghua, ngguaa edu. cn
ABSTRACT
We explore in-context learning (ICL) models from a learning-to-leam perspective. Unlike studies that identify specifie learning algorithms in ICL models, compare ICL models with typical meta-learners to understand their superior performance. We theoretically prove the expressiveness of ICL models as learning algorithms and examine their learnability and generalizability. Our findings show that ICL with transformers can effectively construct data-dependent learning algorithms instead of directly follow existing ones (including gradient- -base,, metric-based, and amortization-ban based meta-learners). The construction of such learning algorithm is determined by the pre-training process, as a function fitting the training distribution, which raises generalizability as an important issue. With above understanding, we propose strategies to transfer techniques for classical deep networks to meta-level to further improve ICL. As examples, we implement meta-level meta-learinn for domain adaptability with limited data and meta-level curriculum learning for accelerated convergence during pre-training, demonstrating their empirical effectiveness.
INTRODUCTION
Large Language Models (LLMs) (Achiam et al. 2023) have witnessed remarkable progress in recent years. Beyond traditional natural language processing tasks such as machine translation and sentiment analysis, LLMs have gained prominence in solving more complex tasks by understanding instructions and examples from human input and generating coherent, human-like text. LLMs use in-context learning (ICL) (Brown 2020) to understand and generate responses based on the input text. Given prompt containing examples (input-output pairs) from a task and query input, ICL allows the LLM to generate the corresponding output without altering their weights. For example, given "happy positive; sad -> negutive; blue ->", the model can output "negative", while given "green -> cool; yellow -> warm; blue ->" the model can output "cool". Formally, ICL can be formulated as follows: given input (x(1), ( (1)), (.),,... x(m), (()), x(*+1)), where there an underlying task such that (0) = fx()), the model outputs a prediction of fx((++1)) By pre- pre-training to simulate the above behavior over a distribution of f, the ICL model can generalize to unseen tasks.
The remarkable performance of LLMs across a wide range of applications has garnered significant attention, to understand how the ICL ability is acquired and executed. However, ICL. has thus far been well understood primarily in highly simplified settings: linear- traaafrransorrmrrs trained on linear regression tasks. In these cases, the model is shown precisely learn to perform pre-conditioned gradient descent based on input examples, with explicit weights corresponding to the global minimum
during pre-training (Von Oswald et al. 2023
Mahankali et al. 2024 Ahn et al. 2023 Gatmiry
"Correspondence author

---

Published as a conference paper at ICLR 2025
This is fantastic \\ Positive What a bad day! \\ Negative
learning algorithm space
support [ V/1
Good job! \\? ?
set
Lx2 V/2
query
-X3 ?
metric-based meta-learner
gradient-hased meta-learner
LLM (ICL Model)
amortzation-based meta-learner
optimal
Ys
Positive
algorithm
ICL Model
Figure I:
In-context learning models can find the data-dependent optimal learning algorithm, in space which is more inclusive than typical meta-learners.
et al. 2024).
Nevertheless, this setting is so simplified that it is far removed from real-world
scenarios, and no more complex settings currently offer such a transparent understanding. To achieve more generalizable understanding of ICL, researchers have approached the problem from various perspectives, including theoretical results on expressiveness (Wang et al. [2024b] Bai et al. )023).
learning dynamics and convergence (Tian et al. 2023: Li et al. 2023b Huang et al.) 2024 Zhang
et al.] (2244 [Sander et al.) (2024), generalization error (Li et al.
2023a 2024| Wies et al. 2023)) ) and
observations of ICL model behaviors (Akyürek et al. 2023 Bhattamishra et al. 2024 Zhang et al.,
2023
Although precisely understanding what and how do ICI. models learn from pre-training is chal- lenging and depends on various problem settings and data distributions, basic consensus has been reached, Specifically, it is understood that an ICL model learns a learning algorithm that maps (()) x(m), y(()) to f through pre- vre-training. The inference process is then interpreted as first learning f from (x()), .y()))))... .... x("), y (()) and subsequently applying
new input x (*+1)
This consensus highlights the nature of ICL. models as meta-learners (Kirsch et al. 2022 Dai et al. 2023), which involves learning a learning algorithm to enable systems to quickly adapt to new tasks essentsstlillly learning-to -o-larmm (Schmidhuber meta-training (pre-training), the goal is to learn a learner function (i.e., a learning algorithm) that can make inferences for given input based on provided set of labeled examples, enabling generalization to meta-testing (unseen) tasks. While typical meta- meean-esrseess have been extensively studied, none have demonstrated the level of general intelligence achieved by LLMs, or ICL models. This naturally raises the question: what distinguishes ICL models from typical meta-learners? While existing works on understanding ICL focus identifying the exact learning algorithms that ICL. models learn, we aim to address a different question: why are ICL models more prominent compared to typical meta-learners?
1987; Thrun & Pratt 1998). Given tasks for
The difference between ICL models and typical meta-learners lies in their hypothesis spaces. ICL is seen as meta-learning with minimal inductive bias (Kirsch et al., 2022). In machine learning, less inductive bias, often leads to better performance with sufficient data (LeCun et al., 2015). Human- designed knowledge may not always be relevant, while data-driven knowledge, optimized through training data, performs well when the hypothesis space is expressive and training is generalizable. Meta-learners' hypothesis spaces define the knowledge needed for a learning algorithm. They use function with two inputs: support set with labeled examples and query input, producing the prediction. Typical meta-learners rely on strong prior knowledge to structure their algorithms. In contrast, ICL models use black-box model, like transformers (Vaswani, 2017), which enable data-driven interactions among samples, can be stacked into deep architectures, and incorporate necessary inductive biases. These properties enhance their generalizability. We study if the advantage of data-driven approaches over human-designed knowledge contributes to the success of ICL models (Figure []. We believe that ICL models are prominent due to their ability to learn data-dependent optimal learning algorithms. However, their optimality is data-dependen posing potential risks in generalizability.
'The formal definitions of an optimal learning algorithm and a data-dependent optimal learning algorithm are provided in Appendix B
2

---

Published as a conference paper at ICLR 2025
In this paper, first prove the expressiveness of ICL with transformers as learning algorithms, and then we verify the aforementioned conjecture. Our findings indicate that ICL models are capable of learning data-dependent optimal algorithms, but their generalizability is limited. This limitation evident from their instribution-sessiiii performance when the algorithm is implicit, prompting reconsider the interpretation of ICI. as "algorithm selection' in existing literature. Building on this insight, we identify shared challenges between training deep models in supervised learning and pre-training ICL models. We then propose leveraging deep-learning techniques to enhance ICL by mapping supervised learning approaches to meta-learning. Our contributions are summarized as follows:
We investigate ICL model from a leamiingtt--eea perspective by comparing it with typical meta-learners, understanding it as a meta-learner with minimal inductive bias.
We theoretically prove that ICL with transformer is expressive enough to mimic existing meta- learners. We demonstrate its ability to construct data-dependent optimal algorithms across a wide range of settings. We also show that its generalizability is limited, it implicitly fits the training distribution rather than learning an explicit algorithm.
We proposed to transfer techniques in classical deep networks to improve ICL. As examples, we enhance the domain adaptability of ICL models through pre-training with meta-level meta-learning and accelerate convergence through pre-training with meta-level curriculum learning.
EXPRESSIVENESS OF ICL AS META-LEARNERS
2.1 A UNIFIED FORMULATION OF LEARNING-TO-LEAR
We first construct the formulation to align ICL models and typical meta-learners. To investigate learning algorithms as learnable functions, a learning algorithm (Kirsch et al. 2022) is considered as mapping from labeled dataset D = {(x(), ( and query input x to prediction (())
The function of a learning algorithm can all be represented as a learner function g:
(9) g(x (4) D).
Learning to-leam /Vilalta Drissi 2002
Hospedales et al., 2021). also known as meta-learning. aims to optimize learnable function g 8) through meta- meta-traing training. The process of training ICL
models or other meta-learners exemplifies learning-to-leam
In-Context Learning with Transformer Generally, there a input matrix Zo composed of and x(9), which is fed into a M-layer transformer TF Denote the collection of all model weights in TF as O.. ICL can thus be represented as learner function gas:
gM (4) D: MM) TF (Zo: 0m),
(2)
with details on the construction of Zo. the model architecture of TFM, and the optimization of 0 provided in Appendix(A..
Meta-Learning Methods Typical meta-leamers are more restricted certain learning algorithm structures designed by human experts. Strong inductive biases are introduced into g(: 0). defining how to adapt to 'D and make inference for x Typical meta-learners are generally classified into three categories (Bronskill et al. 202T]: gradient-based, metric-based and amortization-bast Below is summary of the function of each category.
Boudientt-aaeed Given prediction model h x x y and a loss function ((.,),, gradient-based meta-learners (Finn al. 2017) perform gradient-descent using the labeled data
Ogd (x(9) D: = h(x
V Vel(h( 0), ('))
(3)
Metric-Based Metric-based learners (Koch et al. 2015 Garcia Bruna 2018 Sung et al.) 2018) learn compare the query with examples by optimizing distance metric in the feature space. Let
3

---

Published as a conference paper at ICLR 2025
do(-;-) denote distance function. Pair-wist wise metrie-based algorithm makes prediction based on the pair-wise distance between the query and examples:
9sim (x (9) D; = Σ dx(x))
x( )y(
(4)
For classification tasks where y( E {ce)e=1- one can also adopt class- peoss-protttpee metric-based algorithm (Snell et al., 2017), which compares the query with the class prototypes:
9pn (x (0) D;A) E
c= dol C Ener
x(()
Cc:
(5)
where C. is the class prototype of class c.
Amortization-Basel Amortization-ban based meta-learners replace the optimization process with a single forward pass of the encoder to efficiently learn informative modulations (Garnelo et al. 2018 Ravi & Beatson, 2019). Typically, they follow framework that uses set encoder (Zaheer et al. 2017) to map D to vector representing the task context, then feeds the context and the query to a prediction model fo (x, e) + y. Considering the universal approximation property of neural networks, an amortization-ben based meta-learner can be formulated as:
9am (x (0) D) fe(x 1
[x ( (1))
(6)
The meta-training process is provided in Appendix A.2 A.2
2.2 UNDERSTANDING ICL THROUGH UNIFIED FORMULATION
With the unified formulation, we study the expressiveness of ICL models compared with typical meta-learners. Expressiveness in deep-learning refers to a model's 's ability to capture complex patterns and relationships within data (LeCun al.. 2015), fundamental property that enables deep learning models to achieve high performance on intricate tasks. Here, we focus on expressiveness at the sett-levee-ssccificalll the ability to capture interaction patterns and relationships among samples in DD and x9) which corresponds the expressiveness of learning algorithms. In this section, prove that ICL with transformer expressive enough to perform any learning algorithm that typical meta-learners can.
Specifically, we show that, with certain parameter instantiations, ICL with transformer 9M can perform gradient-based ged (3). pair-wise metric-based !sim 44), class-prototype metric-based Jort (5) and amortization- classification tasks, we consider standard C-class classification tasks. Formally, we present the
Iam (6). Since class-prototype metric-based methods are applicable only
following theorems for classification problems where C < oc:
Theorem 2.1. V E {9gd) 9sim: 9pm. Jam V 0 R!0,, 3 M E N < oc. 3 0 IM (x(1), 0M) = g(x("), O)
Rx
The detailed settings, the mild assumptions and proofs for this part, are provided in Appendix
ICL MODEL DOES LEARN
OPTIMAL ALGORITHM
We have shown that the ICL model is expressive by proving that its hypothesis space is inclusive, at least covering the capabilities of typical meta-learners. However, the specific solution that the ICL model achieves within this space through pre- mreirrani on a given task set-ie. i.e., the exact learning algorithm it learns directly determines its performance and generalizability. Understanding this is crucial investigating its prominence. In this section, we investigate whether the ICL model, with sufficient pre-training, learns an optimal learning algorithm from the training tasks, and examine its generalizability when the specific learning algorithm is not explicitly known.
Algorithm Criterion. To determine whether two learning algorithms are identical, particularly for classification tasks, their classification boundaries can be visualized through Monte Carlo sampling of query inputs. Given multiple trials with different sets of labeled examples, if two learner functions consistently produce the same classification boundary and exhibit identical end-to-end performance, can infer with high probability that they represent the same learning algorithm.
4

---

Published as a conference paper at ICLR 2025
Generalizability of Learning Algorithm. We use the terms explicit and implicit optimal learn- ing algorithm to distinguish the generalizability of a learning algorithm. Formally, we define explicit optimal algorithm g(: F,; of a function family F as: when n - cc, Vf E F, Vp(z). Bpta)la(x(1) FFFF,)] x)] f(x(()), where D {(x(*) f(x()
x(9) p(x), x() p(x). In other words, An explicit optimal learning algorithm for F is generalizable across any data distribution p(z), allowing it to learn any function F. In contrast, implicit optimal learning algorithms are sensitive to the specific data distribution. We denote Gn as the set of all ground-truth explicit optimal algorithms for a task set Q. For example, Ordinary Least Squares is explicit optimal learning algorithm g(: F. +) for linear regression tasks (F = {f f(x) = w x}), while memorizing and looking-up up an implicit optimal learning algorithm for any problem.
3.1 GENERATING TASKS WITH EXPLICIT OPTIMAL ALGORITTMM
To determine whether ICL with transformers learns the optimal learning algorithm, we generate tasks whose optimal predictions can be precisely achieved by certain explicit learning algorithms. For specific algorithms, we select representatives from each category of typical meta-learners: 9sim- Jpn and 9am- Specifically, denoting set of tasks N (Dr) 1+ generate three types of tasks: pair-wise metric-based tasks Qam where MatchNet (Vinyals et al. 2016) (E gsim) is the optimal learner, class-prototype metric-based tasks S2m where ProtoNet (Snell al. (017) (E 9pn) is the optimal learner, amortization- based tasks Sam where CNPs GGanelo et al. 2018) (E 9am) is the optimal learner. We do not consider Igd for classification tasks and the corresponding h to guarantee the optimum; and (ii) proving that ICL can express 9gl is not considered results from Bai et al. (2023); Wang et al. (2024b). Details regarding task generation and experimental settings are provided in Appendix D
reasons: (i) challenging to define family of
contribution of this paper, as straightforward by leveraging
3.2 ICL MODEL LEARNS EXPLICIT OPTIMAL ALGORITHM ON SIMPLE TASKS
To verify that ICL with transformers learns the optimal algorithm, we perform meta meta-training using a single type of task corresponding to one explicit optimal algorithm. We then draw conclusions by comparing the classification boundaries of the trained ICL model with those of the known optimal algorithm. Additionally, we augment the above-mentioned optimal meta-learners with parameterized feed-forward layers, allowing them to be meta-trained alongside the ICL model. This ensures that their optimality becomes data-dependent, similar to the ICL model. We conduct this investigation using the three types of tasks described above.
Accuracc
ProtoNet
ProtoNet
ProtoNet
MatchNet
MatchNet
NetchNet
635
CNPs
CNPs
CNPs
Example Number
Example Number
Example Number
(a) Pair-wise pectric-based
(b) Class-prototype metric-based.
(c) Amortizaaionn
Figure 2: Meta-testing performance learners meta-trained and tested on the same single type of
tasks.
Figure 2(a) shows the end-to-end performance of learners trained on Qsi and tested on D' (unseen tasks). The results indicate that the end-to-end performance of the ICI. model only marginally different from MatchNet, the parameterized optimal meta-learner obtained through meta-training. To further verify this, we visualize how the ICL model classifies each sample given a few fixed labeled examples. An example is shown in the first column in Figure 5 with additional cases provided in Appendix E. These visualizations confirm that the ICL model learns the same algorithm as MatchNet, which is data-dependent and optimal for R Similarly, learns ProtoNet on Ppms where ProtoNet is the optimal learner for this task type, as shown in Figure 2(b) and the second column in Figure Likewise, the ICI. model learns CNPs Pum. the optimal
observe that the ICL model
5

---

Published as a conference paper at ICLR 2025
learner for these tasks, demonstrated in Figure 2 and the third column in Figure 5 Results of more trials are provided in Appendix E. Thus, we conclude that when pre-trained on S2 S2et or Sam the ICL model learns the explicit optimal algorithm.
3.3 ICL MODEL LEARNS IMPLICIT OPTIMAL ALGORITHM ON MIXED TASKS
In real-world scenarios, tasks often complex and diverse. Therefore, further investigate what learning algorithm the ICL model learns when the pre-training tasks come from various types that do not share single optimal explicit learning algorithm.
Specifically, we mix the above 2im- Sgn and am to form meta-training task set m such that I7.... = Net, ProtoNet, CNPs). We meta-train ICL model, MatchNet, ProtoNet and CNPs with mixx. and evaluate their performance on unseen 2 D and Raam respectively. Figure 3 shows the results. We also compare with the performance of data-dependent optimal algorithm (D.-Dpt. Optimal) for each type of testing tasks. In Figure B(a) D.-Dpt. Optimal refers to MatchNet trained on Sam and tested on Sim Figure ((b) it refers to ProtoNet trained on Qen and tested on S and in Figure 3(c) it refers CNPs trained on Summ and tested on Lmm We also visualize the classification boundaries of the ICL model on testing tasks, which align with those of the optimal algorithms.
Examples of these patterns are shown in Figures 10-12 and are further detailed in Appendix E.2
Accuracy
Accuracy
MetchNet
MarchWet
MarchWet
CNPs
CNPs
ChPs
O.-Ott Optmel
D. D.-Opt. Octimel
D. D.-Opt Octimel
Example
Example
Example Number
(a) Pair-wise metric-based tasks. (b) Class-prototype metric-based.
(c) Amortization-baan
Figure 3: Meta-testing performance of learners meta-trained on hybrid tasks, tested on each seen task type separately.
From the above results, draw several conclusions. First, typical meta-learners trained L (MatchNet, ProtoNet, CNPs) fail to achieve the data-dependent optimal (D.-Dpt. Optimal) perfor- mance 2 2g Dmmm indicating that of thses explicit learning algorithm can simultane- ously be optimal för Qeim. 12n and Qam- Second, the ICL model trained on 2 (ICL) demonstrates performance very similar to deta-dependent optimal (D.--pt. Optimal) models. Furthermore, the identical classification boundaries presented in Appendix [..2 suggest that the ICL model successfully learns a Rata-dependent optimal learning algorithm on S2mix- Finally, these findings collectively demonstrate that the ICI. model is more expressive than typical meta-learners and is capable of learning data-dependent optimal algorithm that & 9 However, the precise nature of this learned algorithm remains unknown.
concern about the question of whether the data-dependent optimal learning algorithm on Qmix is implicit explicit, as this is directly related to the generalizability of the ICL model. Existing works have studied ICL models trained on mixed types of tasks to perform "algorithm selection" (Li et al., 2023al Bai et al. 2023) Bhattamishra al. 2024 Waag al. (2024b). "Algorithm selection" refers to the process where, among all algorithms that are explicitly optimal for specific pre-training tasks, the ICL model selects the most suitable algorithm based on the specific task context. This can be formally described when trained with 8
IM (x(9)) D:0M) gx((x)), (D),
s.t., g" arg min EocoEn (x10) WODED 1(g(x(0, ((1).
gEg Pegis
which is end-to-end explicit optimal algorithm.
However, we question whether the ICL model truly learns the "algorithm selection" algorithm when trained on mixed types of tasks. While this interpretation may appear reasonable based existing

---

Published as a conference paper at ICLR 2025
True
ICL
E Q
sim
T2
0 pet
E E
T4 E
0 rad
Ω rad
T6 E f rad
Figure 5: Comparing ICL's predictions and true labels with different types of tasks. Upper: true label. Lower: ICL prediction.
empirical results, it raises concerns about the model's generalizability unseen task types and out-of-distributtin data-an an aspect that has not been thoroughly investigated in the literature. First, ICL model trained with Imis strictly follows 7 it would struggle catastrophically with tasks from novel types that cannot be solved by any E S2... Second, the model learns an explicit optimal algorithm for tasks from types, would lack the distribution sensitivity necessary consistently handle tasks from seen types with data from varying distributions. The following results show that ICL is not "algorithm selection".
3.3. ICL CAN SOLVE TASKS FROM UNSEEN TYPE
We continue to consider the ICL model trained the above Qmix. but now introduce novel type of tasks for testing: radial distance tasks Drdd A task is generated by first sampling r E R from p(r), which defines the classification boundary. Then, {x(1) E Rdy are sampled from p(x). and labels are assigned as follows: (a) = [[x(+]] S r otherwise (0) 1. Note that we specifically control p(x) based on r to ensure that the labels are balanced within each task.
Such radial distance tasks cannot solved by following the "algorithm selection" approach, as radial distance task D cannot be learned by any 9 [MatchNet, ProtoNet, CNPs}. Formally, Vg 9 (pp((),p(z)) 2 y(l)) (x (6) D/(x("), (+))), (()] I, where represents the expected loss of making predictions through random guessing. Following the "algorithm selection" interpretation in (7). no g" can be determined. Even if an arbitrary g" were selected and used, its performance would improve with increasing number of examples.
However, we find that ICL model trained with Suix handle radials distance tasks effectively. While the exemplar tasks in the 4-6th columns in Figure [5] show that ICL model does solve them optimally, the increasing accuracy with growing number of examples, as shown Figure 4a) indicates that it effec- tively learns task-specific informa- tion sion-somehhing that an "algorithm selector" cannot achieve. We con- jecture that when g St is inclu- sive, the pre-tre pre-trained ICL could be generalized to diverse tasks, including those from novel types. This property likely contributes the success of LLMs by approaching the ideal of Heeaming-to-leant
Accuracy
ICL
675
JCL
Algorithm Selection
Algorihmm Selection
Example Number
Distribution Shitt
(a) Radial distance tasks.
(b) Under distribution shift.
Figure 4: Performance comparing between ICL and ideal "al- gorithm selection".
7

---

g with Published as a conference paper at ICLR 2025
3.3.2 ICL SHOWS DISTRIBUTION -ENSSIIVVEE GENERALIZABILITY
Another result reveals that the ICL model has limited generalizability, even on tasks from seen types, as it is sensitive to the data distribution. In contrast, an "algorithm selector," as an explicit optimal algorithm, would not exhibit such sensitivity. Figure [ shows the ICL's performance on hybrid tasks from seen type. The ICL model is trained with m from with input distribution p(r),, but tested on Siix with a shifted distribution p' (z)). The performance decreases noticeably as the distribution shift between training and testing (x-axis) increases, despite the tasks being from seen types. This result directly addresses our question, demonstrating that though ICL model trained with mix does learn data-dependent optimal learning algorithm, it is implicit. Consequently, the ICL model exhibits limited generalizability, as its meta-testing performance is sensitive to data distribution. This behavior reflects the generalizability characteristies of deep learning, now observed at the meta-level.
4 IMPROVING ICL MODELS BY TRANSFERRING TECHNIQUES FOR CLASSICAL DEEP NETWORKS
Above, we have seen that ICL. model can be seen as a data-dependent optimal meta-learner. However, the learned data-dependent optimal algorithm is fitting the training distribution, limiting generaliz- ability- a behavior similar to well-known characteristics of classical deep networks. There are many techniques to enhance classical deep networks' performance, e.g., contrastive learning (Khosla) ct al. 2020), meta-learning (Finn et al. 2017)) curriculum learning (Bengio et al. 2009), and data augmentation (Shorten & Khoshgoftaar 2019). We are motivated transfer exemplar techniques that are originally for deep networks to address the above problems to ICL here.
From a motivational perspective, classical deep networks and ICL models share fundamental chal- lenges. Both require inclusive training data for generalization, but training data is often limited in real-world. They also share common goals, such as improving generalizability, accelerating convergence, and enabling fast adaptation. Thus techniques designed to address these issues in classical deep networks may also be effective for ICL models. From a technical perspective, many techniques designed to achieve the above goals do not impose strict analytical restrictions on the loss function or model architecture, requiring only a differentiable supervision signal. And the inference process of ICL also only relies on greding-forwart be implemented the meta-level through the mapping from samples to tasks, sample-wise loss to task-wise auto-regressive loss, and epochs to episodes. Here, we discuss two exemplary practices: meta-level meta-learning, which significantly improves ICL performance on specific domains with very limited data for adaptation, and meta-level curriculum learning, which effectively accelerates the pre-training process.
Therefore, these techniques can at least
4.1 META-LEVEL META-LEARNINE
While general ICL models can be applied directly, adapting them to domain-specific data is common for building domain-specific intelligence. However, limited domain-specific data increases the risk of overfitting, a few-task problem akin the few-shot issue in supervised learning (Wang et al. 2020)_ illustrated in detail in Appendix G Techniques like LoRA (Hu et al.) (022) and prefix-tuning (Li &
Liang 202T) address this by efficiently adapting ICL models. Unlike existing methods that assume pre-trained general ICL model pre- rre-trained by (33) is given, we consider pre- treiirannnng specifically for adaptation, optimizing performance after few-task domain adaptation. This meta-level meta-leerring approach mimics few-task adaptation during pre-training by solving a bivee optimization problem. It practical as real-world vreiirann tasks are often naturally divided into semantic domains.
Consider domain distribution p(6). Each domain ó determines distribution of tasks ps(T) where domain specifcc task set S2 {D, 3 can be drawn. During pre-training, we manually split Ωg into two disjoint task sets: a training (support) task set 22 = {D+) and a validation (query) task set 52 {D, 1 Denote meta-level meta-learner as G(g. Ω A), i.e., a domain adapter adapting meta-learner g with task set Q. Denote a meta loss function evaluating (Di,, ) as emeea( (T,9). Meta-level meta-training is performing:
min Bp(()
Σ
e (T, G(o(; 0), })}].
(8)
8

---

Published as a conference paper at ICLR 2025
Specifically, to adopt meta-level meta-learning for improving the pre-training of the ICL model g(: = 9M (; MM), we implement MAML as the meta-level meta-learner: G(g(- 0), S A) = g(::0 Ve i Ereny (meta(T, g(: ()). which avoids designing additional Iearnable due to MAML's model-aggostie property. Alternatively, one can also design meta-level meta-learners based on metric or amortization-base approaches.
NA-ICL adpt. adpt NP-CI wio adpt. ICL wil adgt.
MP-ICL adpt. adst MP--CI weo wiyt. ICL wel sipi.
adpe.
302
MV' ICL w/o sopt.
Squared Ertrr
16
Number
Examata
(a) Given 64 tasks for adaptation. (b) Given 256 tasks for adaptation. (c) Given 1024 tasks for adaptation.
Figure 6: Performance of meta-trained ICL model and meta-level meta-trained ICL model on unseen domain, given a few tasks for adaptation.
We conduct experiments linear regression tasks, where the distribution of linear weights is Ps(T) = ((ss, Σs) with (Hu, Es) ~ p()). More details are provided in Appendix F We denote such meta-level meta-trained ICL model MC-ICL. After pre-training, we test on unseen domains drawn from p(6). Each domain provides 52 {D+}'-- for adaptation, and S2 {D+) - for performance evaluation. The performance is shown in Figure 6 Note that reasonable solutions include ICL w/ adpt, ICL w/o adpt, and M2-ICI adpt, while M--ICI w/o adpt serves only as an intermediate product of meta-level meta-learning. We find that M2-ICL w/ adpt outperforms both ICL w/ adpt and ICL w/o adpt. particularly when the number of adaptation tasks is very small (64, Figure 6(a)), while the advantage gradually decreases with the growth of task number (marginal with 1024 adaptation tasks Figure [(c). Meta-level meta-learning is effective for fast adaptation on few-task domain, like typical meta-learning's 's effectiveness for fast adaptation few-shot task. Note that, although the adaptation strategy in this experiment involves fine-tuning all parameters using gradient descent (i.e.. G is derived from MAML with inner updates as full-parameter fine-tuning), any differentiable adaptation strategy can replace the inner-update or be incorporated into other specifications of G. The comparison between ICL and M2-ICL isomorphic with the comparison between model trained using standard supervised learning and model meta-trui trained using MAML. Experiment results on cross-domain few-shot image classification is provided in Appendix H
4.2 MEVA-LEVEL CURRICULUM LEARNING
Curriculum meta-leannnn learning strategy is intuitive, as meta- should progressively learn tasks from simple to complex for better convergence (Bengio et al.. 2009). This approach has been shown to be effective for gradient-basse hased meta-learners (Chen al. 2021 SSergiadiss et al.) 2021). Here, investigate whether the curriculum strategy can enhance the meta-training of ICI.
We consider simple case:
ICL model learning linear regres- sion tasks, where task complexity is evaluated by the number of di- mensions. This approach, practiced by Garg et al. (2022). has not yet been explicitly investigated for its effect. With maximum dimension of 20, we train the ICL model on tasks with an increasing number of effective dimensions (denoted such baseline as CL-ICL, which curricu- lum is shown in Figure/((a while values in the remaining dimensions are set zero. The training loss (Figure 7(b) and performance
16
ICL
CLACI
12
CLICL
15
10
Step
Step
5
50k
100k
150k
200k
0
50k
100k
150k
200k
(a) Effective dimension.
(b) Training loss.
Figure 7: Training dynamics of ICL model with curriculum.

---

Published as a conference paper at ICLR 2025
comparison under limited training (Figure 8(a) demonstrate that this curriculum enables faster con- vergence. However, with sufficient training, the ICL models trained with and without the curriculum exhibit nearly identical training loss and testing performance (Figure This result suggests that, in this case, the curriculum strategy accelerates convergence but does not lead to better optimum.
RELATED WORKS
Here, we provide detailed discus- sion of the connections between this paper and the most relevant studies.
IC
It has been comprehensively inves-
Squored Eror
Saparde Eror
tigated (Akyürek et al., 2023 Bai
et al. 2023) that the explicit learn- ing algorithms that transformers can
learn in-context, including ridge gression, least squares, and Lasso on linear regression tasks. Building on their findings, we believe that transformers are capable of learn- ing numerous explicit learning algo- rithms in-context. Rather than focusing on identifying specific explicit algorithms under different settings, our work provides a broader and more abstract understanding of ICL with transformers. conceptualize ICL as data-dependent model that can express typical meta-learners. These meta-learners are capable of expressing wide range of explicit learning algorithms (Finn & Levine
Feomple Number
Fecmple Number
Testing @ x 10 episodes. Testing @ x 10 episodes.
Figure 8: Testing performance of ICI. model with curriculum.
2018 Zaheer et al. 2017) and can be applied to a variety of broad applications (Wu et al.) [2024] Yao al. 2024| Wang et al. 2024a). It has also been proven that ICL with transformers implement gradient descent on neural networks (Bai et al. 2023). which we leverage as an important tool to show that ICL can perform gradient-basee
meta-learming algorithms.
We adopt the definition of a learning algorithm from Kirsch et al. (2022) and build on their under- standing that ICL models function as general-purpose meta-learning systems with minimal inductive bias. They demonstrate that ICL models can do bearning-to-leat and a large number training tasks matters. We compare ICI. models with other meta-learners, proving the expressiveness of ICL. with transformers and revealing its learnability and the characteristics of the algorithms it learns.
CONCLUSION, LIMITATIONS AND DISCUSSION
Pre-training an ICL model is essentially learning-to-leams This paper compares ICI, models with typical meta-learners, suggesting strategies to enhance ICL. It shows that ICL with transformers can learn optimal algorithms in a data-dependent way, but generalizability remains an issue as the learned algorithm may fit the training distribution rather than generalize well. ICL models in meta- learning are similar to deep models in supervised learning, sharing characteristics like data-dependent optimality and generalizability challenges. We propose applying deep-learning techniques, such as meta-level meta-learning and curriculum learning, to improve ICL. These strategies could enhance domain adaptability and speed up convergence, providing insights into ICI. and foundation for more robust models.
This paper examines ICL with transformers without considering the order of examples. Future research could explore the impact of positional embeddings in transformers, alternative architectures like RNNs and SSMs (Gu & Dao 2023). and the convergence dynamics of pre-training ICL models, especially for real-world LLMs. Additionally, advanced techniques like contrastive learning and denoising could be adapted to the meta-level to further improve ICL, performance.
ACKNOWLEDGEMENT
Q. Yao's work is supported by National Natural Science Foundation of China (under Grant No. 92270106), Beijing Natural Science Foundation (under Grant No. 4242039), and CCF-Huawei Populus Grove Fund.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 technical report. arXiv preprint arXiv:2303.08711 2023.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In NewrIPS, pp. 45614 45650, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In ICLR, 2023.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In NeurIPS, pp. 37122-57221, 2023.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48, 2009.
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and L.LMs by learning to learn discrete functions. ICLR, 2024.
John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard Turner. Memory efficient meta-learning with large images. In NeurlPS, pp. 24327-
24339, 2021.
Tom B Brown. Language models are few-shot learners. In NeurlPS, pp. 1877-1901, 2020.
Yudong Chen, Xin Wang, Miao Fan, Jizhou Huang, Shengwen Yang, and Wenwu Zhu. Curriculum meta-learning for next POI recommendation. In KDD, pp. 2692-2702, 2021.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models implicitly perform gradient descent meta-optimizers. In Findings of the ACL, pp. 4005-4019, 2023.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariin for fast adaptation of deep networks. In ICML, pp. 1126-1135, 2017.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In ICLR, 2018.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? case study of simple function classes. In NeurlPS, pp. 30583-30598, 2022.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, pp. 1704-1713, 2018.
Khashayar Gatmiry, Nikunj Saunshi, Sashank Reddi, Stefanie Jegelka, and Sanjiv Kumar, Can looped transformers learn implement multi-step gradient descent for in-context learning? In ICML., pp. 15130-15152, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00 210:22312.00752, 2023.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366, 1989.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, IEEE TPAMI, 4499):51495516 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhin Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
11

---

Published as a conference paper at ICLR 2025
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. In ICML, pp. 19660-19722, 2024.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NewrIPS, pp. 18661-
18673, 2020.
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-leaming transformers. arXiv preprint DXXi::2222.04456 2022.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, volume 2, pp. 1-30. Lille, 2015.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, and Pin- Yu Chen. How do nonlinear transformers learn and generalize in in-context learning? In ICML, pp.
2024.
Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. In ACL, pp. 4582-4597 2021.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In ICML, pp. 19565-19594, 2023a.
Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards a mechanistic understanding. In ICML, pp. 19689-19729, 2023b.
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In ICLR, 2024.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. ICLR, 2019.
Michael E Sander, Raja Giryes, Taiji Suzuki, Mathieu Blondel, and Gabriel Peyré. How do trans- formers perform in-context autoregressive learning? In ICML, pp. 43235-432545 2024.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta.... hook. PhD thesis, Technische Universität München, 1987.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, pp. 4077-4087, 2017.
Emmanouil Stergiadis, Priyanka Agrawal, and Oliver Squire. Curriculum meta-learning for few-shot classification, arXiv preprint arXiv:2/ 2021.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, pp. 1199-1208, 2018.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to Learn, pp. 3-17. Springer, 1998.
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon S Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. In NeurIPS, pp. 71911-71947, 2023,
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. META-DATASET. A dataset of datasets for learning to learn from few examples. In ICLR, 2020.
A Vaswani. Attention is all you need. In NeurlPS, pp. 5998-6008, 2017.
12

---

Published as a conference paper at ICLR 2025
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18:77-95, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurIPS, pp. 3630-3638, 2016.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In ICML, pp. 35151-35174, 2023.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from a few examples: A survey few-shot learning. ACM Computing Surveys, 53(3):1-34 2020.
Yaqing Wang, Hongming Piao, Daxiang Dong, Quanming Yao, and Jingho Zhou. Warming up cold-start CTR prediction by learning item-specific feature interactions. In KDD, pp. 3233-3244, 2024a.
Zhijie Wang, Bo Jiang, and Shuai Li. In-context learning on function classes unveiled for transformers. In ICML, pp. 50726-50745, 2024b.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In NeurIPS, pp. 36637-36651, 2023.
Shiguang Wu, Yaqing Wang, and Quanming Yao. PACIA: Parameter-efficient adapter for few-shot molecular property prediction. In IJCAI, pp. 5208-5216, 2024.
Quanming Yao, Zhenqian Shen, Yaqing Wang, and Dejing Dou. Property-aware relation networks for few-shot molecular property prediction. /EEE TPAMI, 4613):4133-494 2024.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In NeurIPS, pp. 3391-3401, 2017.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. JMLR, 25 2024.
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint arXiv:2305. 10055/9944 /9420, 2023.
13

---

Published as a conference paper at ICLR 2025
A PRELIMINARIES
A.1 IN-CONTEXT LEARNING WITH TRANSFORMER
Input. Following existing works VVnn Oswald al. 2023 Ahn et al. 2023). we investigate ICL without positional embedding to study its learning-to- to-lerrm ability while ignoring the order of examples. Let x()) E IRd be a input, and y(i E R be the corresponding output. For each task there a task-specific function f- and dataset of the task DD that v(x(() y00) E DD, = St (x(()).
Labeled examples and query of task input together. Define the input matrix Zp:
x(1)
x(2)
x(") (())
x(a+1)]
Zo (z))) (22) z(m) z [n+1)]
E (((+))x(+1))
yy()) y(2)
where q EE R is the indicator of unlabeled query. ICL model is trained to output the prediction of y ( given Zo. with set of tasks (D+ {[x((, (y)) )= y- from training distribution fr x()),(()) p(x), generalize to unseen tasks.
Model Architecture. ICL is typically achieved by transformer, which are composed of stacked self-attention layers. Given Z E (xd+e) a single-head self-attention layer Attn smax is defined
smax Wt, (Z) W.Z smax Z WT W22).
(10)
where W., Ws. W. (R(++))x(d+e) are the (value, key and query) weight matrices, and smax(.) is the softmax operator which applies softmax operation to each column of the input matrix. Note that the prompt asymmetrie since the label for (=++1) excluded from the inpur.
An M-layer transformer, denoted as TFM. consists of stack of M self-attention layers and MI.P blocks. Formally, denoting by Zi the output of the 7th layer attention, we define
Z Za + oos (Attnp, (Z))) for l 0, 1,..., M 1,
(11)
where (-) represents feed-forward layers parameterized by an, which operates independently each column of the input. Given Zo, the prediction is obtained as
M-1 W.) = W. (,(+1)) y (1+1) 1) TF M (Zo: (Pt, Qi, on M
[ZM),(+1))
(12)
where [Zm]).n--1) is the (n + 1)-th column of ZM, and W, E IR (d++c) is the linear readout weight.
During training, given the distribution of tasks fr p(T), (+) ~ p(r), and loss function ((.--) (e.g.. cross-entropy), the parameters are optimized minimize the expectation of ruto-regressive loss of training tasks:
min Ex(r)a(())
Σ i=0
((g(++1) y(it1))]-
(13)
(P.QQ.:: A-- w,
N
A.2 META-LEARNING
Meta-learning is a methodology concerned with "learning-to-leam" algorithms. Define g(: 0) is a meta-learner that maps task dataset D, and a query input x(9) to its task-specific prediction. Typical meta-learning algorithms first learn an explicit model to a model from D, and then perform the prediction, i.c. (() g(x("), D: D)0) g' (D: (x(1)). For meta-training, given the training distribution p(T) and p(:).. from which the tasks [D, }f= are drawn, the goal is to learn g(: 0) that performs well on unseen tasks. Within each task, training set D ((rr,) b+,)) is used to provide supervised information, and a validation set Dral = {(x,,5 Br used to evaluate performance and optimize the meta-learner. The meta-training process is then performed
min (p()))(()) [ [N--
1
Σ
(14)
N2 imn+1
14

---

Published as a conference paper at ICLR 2025
B OPTIMALITY OF A LEARNING ALGORITHM
We claim ICI. model learns data-dependent optimal learning algorithms (DDOLAA), which is different and weaker than (true) optimal learning algorithm (OLA).
Formally, given a finite training set Drnin {(Z,, 3:3} where each sample is i.i.d.: (Ii, Me) ~ p(z, v). and a unseen testing set Dust [(T,, m2)} following the same distribution, the OLA is 9 = arg max, =((x:)))))((x.)) (Prob[g(z; Duain) yyl}. Which is to say learning algorithm can make the most "accurate" prediction given training set and unseen target input from the same distribution. It is possible to know the optimal learning algorithm with the priori of p(a,, y). For example, ordinary least squares is optimal for linear regression with Gaussian noise. In the paper, three types of tasks are generated by designed ways, i.c., known p(z, y) (Section 3.1. It is obvious that a MatchNet model with certain parameters (simply keeping all modules inside identical mappings) is the optimal learning algorithm for Sam. and so does ProtoNet for 2pn and CNPs for Sum- However, meta-learners have not access the true p(r, y). They only learn the function infer p(z. y) from Dusin = {x }/)} through meta-training which inevitably brings variance and bias, being (meta-training) data-dependent. So we denote that given certain meta-training set, the best that vandom-initialized initialized and meta-trained deep learner can do as the DDOLA. This could be empirically approximated by meta- veta-training a deep and madoominitializzo MatchNec/PoooNe//CNP with certain meta-training set (for the three task types respectively).
C PROOF OF THE META-LEVEL EXPRESSIVENESS OF ICL
Theorem C.1. V g E {ggd: grims Ipm: Jam 4 E R M E N < 0O, 3 0M E R (x(9), D)) = g(x(9), ((x()))D;)))
Proof Sketch The proof of is constructed by decomposing the functions of typical learners into M E N < oo conditioned steps, where each step can be achieved through one transformer layer with the following two basic tools:
1. Universal approximation property of multi-layer perceptron (MLP) (Horik et al. 1989): This property allows feed-forward layers to express wide range of functions amm aim In cach transformer layer, the feed-forward module operates independently on each column of the input matrix. This enables the transformer to perform wide range of sample-wise transformations every layer.
Orthonormal label embedding in I2 We use set of orthonormal vectors in 22 as embeddings for the categorical labels (including the query identifier), such as one-hot embeddings. This ensures that the attention weight matrix A E (nn+1))((++)) in the self-attention module of each transformer layer can label-aware. Label- awareness implics that {A E [[*+1)x(=-1) (y(1) =
(()) A (y()) = y()) + Ar, = Ar J. i.e. the interaction weight between ordered sample pair (i,j) depends only on their labels (y))),()) and can take any value in R*. This allows the learning algorithm achieve the behavior of a label-aware set function.
The decomposition into finite conditioned steps is specific to each theorem and is not necessarily unique. Next we provide the full proof.
C.I DETAIL SETTINGS
C.I.1 CLASSIFICATION TASK WITH ORTHONORMAL LABEL EMBEDDING
Classification task specifies the ICL's input in Section A. with y(t)) E (c1, c..... cce). where Ce the embedding vector of the c-th class. For these label embeddings, we can find 2C orthonormal vectors in 2C, {u; J2 such that:
11, ifi=j u TT, u 0, if
(15)
A simple choice of {u, |C is the set of 2C' one-hot vectors. We use {u; as the embeddings of (ce)C i.e., y(t) { and uc- the indicator of query q.
15

---

Published as a conference paper at ICLR 2025
C.1.2 SELF-ATTENTION WITHOUT SOFTMAX
In setting, consider self-attention layers that replace the softmax operation in (TO) with column-wise Ll-normalization. particular, ( is now approximated and reparameterized with weights == W, E gld+e)x(d+e) and Q = Wx Wa E [dd+e)x(dte) as:
Attnr.o (Z) = (z (Z)
(16)
A
Σ
AcJl #0
E3 A
where (oomm
Note that it conventional to omit certain
0,
A
non-linearities, such as the softmax operation, in self-attention layers to align transformers with explicit learning algorithms. While existing works often replace the softmax operation with 1 n
(Von Oswald et al.
2023 Ahn et al. 2023). the norm in (6) provides a closer approximation.
Since the proof for the gradient-based algorithm (3) follows trivially from the results of Bai al. (2023); Wang et al. 2024b). we focus on metric-based algorithms (4) and (5). as well as amortization- based algorithms 6). We prove that there exists an TF model with specific real-valued parameters that can perform these algorithms.
Note that, for simplicity in proving the expressiveness of ICL with transformer, focus the algorithm framework: we leave feature-level transformations with neural networks alone as they can occur both ICL model and conventional meta-lcarners, and enjoy the same universal approximation property (Hornik et al. 1989); we also do not consider the order of samples in D, omitting any se- quential models meta-learners and positional embeddings in ICL. Typical metric-based algorithms are thus categorized into to types: one is based on pair-wise distance ( e.g.. MatchNet; another one is based on distance with class prototypes (5) e.g., ProtoNet. And typical amortization-base algorithms are summarized that these exemplar set and inference functions can be implemented, more complex algorithms such feature-wiz wise transformations, interactions between samples, and advanced distance functions can be easily achieved through the recursive application of self-attention and weeewff layers.
function taking the query and the encoded set input 6 By proving
C.1.3 ICL CAN PERFORM WIRE METRIC- METRIC-BASED BASED ALGORITHMS
For pair-wise metric-based algorithms, take MatchNet for example, proving 12 can perform
y (a++1)
( x (n+1) > (1)
(17)
In fact, this case is relatively simple and can be implemented using single-layer transformer 0
[ 0
0
without relying on the two tools. One implementation is Qo =
Po
W
0 0 Note that though the output of TF would be A AE < x()) x((+1) > y()
U1 Ue
where À E IR is query-specific value, has the same classification result with
C.1.4 ICL CAN PERFORM CLASS-PROTOTYES METRIC-BASHR ALGORITTMS
For the second category of metric-based algorithms, we take ProtoNet for example, proving 22 can perform
((+11)
Σ -11 No y(
Σ x() x (n+11)
(18)
16

---

Published as a conference paper at ICLR 2025
This can be implemented by 3C 1 layer transformer achieving (2)=
Eom ||x [n(++) -Pel (cll in the following step-by-step functions:
x
x
(2)
x(m)
x(++1)
Zo =
(
(
y(m)
(19)
xx())
x((+1) p1
Z1
yr(i)
(20)
9
x(1)
x Pt
Z2
y(
9 ||x(++1) pı |
(21)
x(1
(m++1)
Z3 yy
9 ||x(n+1) 1) P1 p1lluct2
(22)
x(1)
((+1)) D2
Z4
(y())
9 ||x(==1) p l|uct2
(23)
x(0
x(++1) P2
Z3 =
y(()
+ | (m(+)) P1 uo+2 + ||x(=+1) p2 P2lluct3
(24)
x(1)
((*+1)
Z6 y(
+ ( - p1 uo+2 + ||x(n+1) 1221
(25)
(26)
((
x (x+1)
Za-3
(r(()
q - c0 ||x(++1) p: [(++++C)mod((C))
(27)
(1)
x(++1) pt
Za-2
(()
9 (1) pi | (M(+++))mool22
(28)
x(1)
x((+1)=
Za-1
(29)
(1(+)
q + Σ
||x(++1) pi
[+++++
(30)
0
0
and readout by W, =
Each step of function
uC+2 m(o+14C)mm
24
from Zı to Z141 can be implemented by one transformer layer, which would be proved later.
C.1.5 ICL CAN PERFORM AMORTIZATION-BASEE ALGORITHMS
Denote the set embedding 1 (x (0) as e Rdd,, As / in (6) can always be implemented by feed forward layers taking the concatenation of x(9) and as input, there exists learnable function h1 in Rdd Id and h2 in Rdd RR that
f((x(1)) Te,x 12(x(0) + hn(c)).
(31)
Thus, we prove that 12 can perform
(m++)) = 12(2x(1 (n+1) + hi( Σ ( (1)).
(32)
This can be implemented by layer transformer achieving the following step-by-step function:
(1) x(2)
x(m)
x(n+1)
Zo (()) y(2)
(33)
(n)
q
x x(++1) + hi En (x(1 (0)
Z1
((())
(34)
q
(1)
12(x(++1) + h1 Σ
[x
(y(00
Zo
((())
(35)
q
1
0
and readout by W,
Each step of function from Z, to Z can be implemented by one
0
0
transformer layer, which would be proved now.
17

---

Published as a conference paper at ICLR 2025
C.1.6 THE FUNCTION ONE TRANSFORMER LAYER
transformer layer (11) can perform a wide range of functions, can decompose it is composed of self-attention layer and feed-forward layers, where (i) self-attention (T6) with orthonormal label tokenization (1)) can achieve wide range of label-aware set operations. (ii) need-forwrd layer or:) in (IT) can learn any measurable functions in I2 Rd+ Here we prove how transformer layer can obtaín the above functions from Z to ZL+ The main idea a function can decomposed to three sub-steps: label-selecting which is achieved by A = 7 Q?, linear interaction achieved by P A), and non-linear transformation by ag(.) ) needed.
Label-Aware Attention. In one self-attention layer, equation (16), first each column in Z refer to other columns through attention weights A 2 QZ. A E R(n+1)x(x++ is selecting interaction objectives and weighting interaction weights. We use label-aware to describe {AA E [g(++))((++)) (y y(1)) A (y(i) y' + Aij Av.y) ). i.e., the interaction weight between ordered sample-pair (i.j) only depends their labels ,y()),))))) (including unknown label q), and can be arbitrary value in IR.
y(l))
With orthonormal label embedding in R A is label-aware, thus achieve label-aware interaction. For example, to achieve (19 to 20
we require (c + 1)² conditions about A:
A Ag.e
Aa.- = 0, i {2,3, ,C
A Ag.cc 0, i [1,2, ".
(36)
Ac,e; 0, i,j e (1,,
Ag.9 =
[0 0
As A ZTQZ and A,, only related to y()), y(),, we have Q
where E R22xx0
L
Equation (36) gives (C + 1)a linear equations about L:
u Lug =
u, Laig = 0, E [2,3, .. .C}
2 Lut; 0, i (1,2, ... ,0} ")
(37)
u Luj 0, i,j E f1,2.....0
u Lug =0
Proposition C.2. Vc > 1, Equation
(37) has solutions in 2CxC
Proof. Denote X Uj
Iu. u.2u, , 20 1 E
R4C²
L1,1, ... L1,20, L2c.c] e R40² then u, Lu, = (u, 2 u₁) TL.
37
UÉ A. where
U [u; 8 új for for i.j (1,2, , 13JT [(C++))xx4C*
(38)
2c
(u, 8 u,)) T @ su,))
> t=1
U up uy
(39)
[u] ugy) )(
up
(40)
1=1 (u] uuyTyy) ( (u, ux))
(41)
ifi if
[1,
s' j = j'
(42)
10 v i
18

---

Published as a conference paper at ICLR 2025
rank(U)) (C 1)²
(43)
[U, A] (((+)))((a+++))) , ->> (an S (C 1C²
(44)
44
rank(U, (U. runk(U)) +C+ 4C
(45)
Equation UL = Á has solutions in RAC²
Equation (37 has solutions in 2C200
Note that for any function from Z to Z1+1, the number of conditions about A < (2C') Thus for any label-aware function from Zi to Z441, requires label-ware A and find linear system of equations UL Å, that has solutions in R446", as the proof rank((U, rank (U) 4c² is without loss of generalizability.
Linear Interaction. After obtaining desired A, (A) is performed as norm, is a better approximation of softmax than #,2 and also required to deal with inconsistent label number in our classification tasks. Then all columns in interact with the others linearly through ""
[-1
Still taking
20 as example, after obtaining desired satisfying
36
0
0 can
achieve the function.
Non-Linear Transformation. In (11) (-(+) is feed-forward layers that function on each column of Z independently. Thanks to the universal approximation property (Hornik al. 1989). it can approximate any measurable function in d2 dd to any desired degree of accuracy. Thus,
feature-level non-linear transformation from Z₁ to ZL+1 could turn to de (-). For example, 19 to
(20) does not require non-linearity so it can be implemented as o==) = z. For (20) to (1 one
implementation is (=) [0, [[[ [===a:d++a [C++]] T Note that in this step, the does not hold strictly,
but can be approximated by MLPs with eтror > O. We use "=" to mean such approximation for simplicity, as the error can be arbitrary small.
In conclusion, each step from Z: to ZI-1 can be implemented using a transformer layer. Typical metric- and amortization-basee meta-learning algorithms 5 66 can be implemented with ICL. More complex models following the same set functions can also be performed by ICI with additional recursion of transformer layers, whose proof is trivial. Moreover, as it has been proved that ICL can perform gradient-based algorithms (3) ICI. can exactly perform conventional handcrafted meta- learning algorithms.
D GENERATING TASKS
Here, we present the task generation process for Isim- Ipa and Jam- The tasks designed in specific forms such that they are linearly separable (() Rd, enabling 2D visualization to observe the behavior of ICL.
For wise metric-based algorithms Jsims we generate a task by sampling x Nc support samples
{x()) E Cx from distribution p(r) and randomly assign them with labels y = c, making
C' x No supports exactly contains Nc label for each 1,2, C. Then the remaining samples (x(() Ne
No=CxNo+1 are sampled from distribution Pa, and assigned with labels y()) arg max, -j=1yb0= < x() (()) >. A typical meta-learner, MatchNet, can learn the optimal classifier. case is shown in Figure (()) where each point corresponds to a (1) I22 and different
labels are assigned with different colors.
For class-proootype metric-based algorithms 9pr. we generate a task by sampling C prototypes {pe E R4)C from p(r). Then sample (x(i) E R from Pas and assign labels by y = argmin, I|p. - x0 The corresponding optimal classifier is ProtoNet. A case shown in Figure
(b))
19

---

Published as a conference paper at ICLR 2025
(a) Pair-wise metric-hased.
(b) Class-prototype metric-based.
(c) Amortization-based.
Figure 9: Examples of three types of tasks.
For amortization- algorithms 9am we pre-define a partition R. {Re) as decision range. We generate a task by sampling E Rd from p(r). Then sample fx(*) E R from P=(u) = (p, Σ), and assign labels by y = where >d 1=1 [x(1) sAle E Ωe. The corresponding optimal classifier is CNPs. A case shown in Figure (c)
E MORE EMPIRICAL RESULTS
E.1 ICL MODEL TRAINED WITH SINGLE TYPE OF TASKS
Figures
and show more cases support that ICL model learns MatchNet, ProtoNet, CNPs
on S2sm $2 ppt. S 2am respectively.
(a) ICL-1.
True-1.
(c) ICL-2.
(d) True-2.
(e) ICL-3.
(f) True-3.
Figure 10: Comparison of ICL's predictions with true labels on pair-wise metric-based meta-testing tasks, with the ICL model trained on a single task type.
(a) ICL-I.
(b) True-1.
(c) ICL-2.
(d) True-2.
(e) ICL-3.
(f) True-3
Figure 11: Comparing ICL's predictions and true labels on class ploss-prototype metric-basee tasks, with the ICI. model trained single task type.
(a) ICL-1.
(b) True-1.
(c) ICL-2.
(d) True-2.
(c) ICL-3.
(f) True-3.
Figure 13: Comparing ICL's predictions and true labels on pair-wise metric-based tasks, with the ICL model trained mixed task types.
20

---

Published as a conference paper at ICLR 2025
(a) ICL-I.
(b) True-1.
(c) ICL-2.
(d) True-2.
(e) ICL-3.
(l) True-3,
Figure 12: Comparing ICL's predictions and true labels on amortization based tasks, with the ICL model trained single task type.
(a) ICL-1.
(h) True-1.
(c) ICI.22
(d) True-2.
(c) ICL-3.
(f) True-3.
Figure 14:
Comparing ICL's predictions and true labels on class-prototype metric-based tasks, with the ICL model trained mixed task types.
(a) ICL-I.
True-1.
(c) ICL-2.
(d) True-2.
(e) ICL-3.
(f) True-3.
Figure 15:
Comparing ICL's predictions and true labels amortization- based tasks, with the ICL model trained on mixed task types.
E.2 ICL MODEL TRAINED WITH MIXED TYPE OF TASKS
Figures 13
14 and 15 show cases to support that ICL model learns data-dependent optimal learning
algorithm S2mix
[Saim Serii 0am
F
EXPERIMENT DETAILS
Our code is provided attpps///egttabb..o///oo//////
21

---

Published as a conference paper at ICLR 2025
data point (x.
training
testing
The setting of learning (single task)
meta-training
meta-testing
The setting of meta-learning
The setting meta-level meta-learning
meta-level meta-training
meta-level meta-testing
Figure 16: The problem setting of meta-level meta-learning.
G ILLUSTRATION OF META-LEVER META-LEARNING
Here, we provide further illustrations of the problem setting and the algorithm procedure of the proposed meta-level meta-lcarning, supplemening Section 4.1 The problem setting is illustrated in Figure 16 Conventional single-task learning solves problems on single task, which consists of set of data points for training and another set for testing. Typical meta-learning addresses problems domain, which includes set of tasks for meta-training and another set for meta-testing. Meta-level meta-learning operates on a collection of domains, each containing multiple tasks, and involves set of domains for meta-level meta-training and another set for meta-level meta-testing.
Next, we illustrate the process of training an M2-ICI. model. The meta-level meta-training process, with MAML as the meta-level meta-learner, provided in Algorithm [ Note that the meta-level meta-learner G(; A) in Equation (8) does not have to be MAML. We take MAML G(: A) because
of its madel-agnostic property, which allows us to avoid designing additional learnable parameters A. However,
design customized G(; A) meta-learners, such metric-based or amortization- based methods.
Algorithm Training M2-ICL
Input: Training domain distribution p(6), ICL model g( 0).
1: while Not converge do
2: Sample a domain d ~ p(6).
3: Sample tasks (6) to form task sets 52 and s2
4: for Every task 5 E 52 do
5: Calculate task loss Emeca(
((:0))() (T:ea(T:((;0)))
N. DNO E(g(+12), (0044,((111))) by 12
6: end for
7: Update 0g = Vo [22
Ereny laeia (notaF..ghh (T.((=;)))
8: for Every task t E 52yal do
9: Calculate task loss Eaeeaa (T, 0(:),()) N, EN- (e((i+)),,((+11)) by 12
10: end for
Update 0 + Ve [
Ereny emora g(:)0))).
12: end while
H META-LEVEL META-LEARNING FOR CROSS-DOMANT FEW-SHOT IMAGE CLASSIFICATION
We investigate the effective of meta-level meta-lcarning on real-world few-shot image classification problem. We use META-DATASET (Triantafillou al. 2020) for training. Because it contains
22

---

Published as a conference paper at ICLR 2025
multiple datasets inside each sample many few-shot classification tasks, thus can be naturally divided into multiple domains to perform the meta-level meta- veea-training
Following standard settings, used the training sets of ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, and Fungi during training. For testing. we used unseen datasets such as Traffic Signs, MSCOCO, and additional datasets like MNIST, CIFARIO 10, and CIFAR 100. Each dataset is treated as a domain for meta-level meta-learning. We considered 5-way 5-shot tasks at the meta-level and 8/16/32 tasks-fr--ddaptation per domain the meta-meta-level. The sampling of classes and images to form tasks, as well as the sampling of tasks-for-adaptation within a domain, was random.
We consider the following baselines:
ICL. w/o adpt: The standard meta-learning setting, where meta-training is performed on all tasks without distinguishing between datasets. During meta-testing, no domain adaptation is performed, meaning that the 8/16/32 tasks are not utilized.
The meta-training process is identical to that of ICL w/o adpt. While dur- ing meta-testing. the model adapts using 8/16/32 domain-specific tasks by fine-tuning all parameters (step = 5, learning rate 0.0001, batch size 8/16/32).
ICL w/ adpt:
M2-ICL: Meta-level meta-training an ICL model following the method introduced in Sec- tion 4.1 The domain adaptation process, i.e., the inner-loop of meta-level-MAMe is configured as step=5, 1=0.0001, with 16 tasks-for-ane tasks-for-adaptation per domain. During testing, given 8/16/32 domain-specifie tasks, the same adaptation process is applied.
Our implementation builds the ICL model with 8-layer transformer (without positional encoding), where the input features are 512-dim extracted by a ResNet (i.e., resnet-18). Though the model is relatively toy, is enough to verify the effectiveness of meta-level meta- lett-learnnng for improving ICL in real-world application: the method pipeline is generalizable and one can replace them with models with more advanced architectures or for other applications.
Table 1: Image classification accuracy (%) on META-DATASET, benchmark for cross-domain few-shot image classification.
Method
Traffic Signs
MSCOCO
MNIST
CIFAR1O
CIFAR 100
Average
ICL w/o adpt
45.4
35.5
88.1
65.2
55.9
38.02
ICL w/ adpt (8 tasks)
41.9
35.1
76.4
64.9
55.5
53.76
ICL w/ adpt (16 tasks)
43.3
36.2
78.5
66.0
56.3
56.06
ICL w/ adpt (32 tasks)
46.1
36.5
83.2
66.8
58.3
58,18
M (8 tasks)
45.9
39.4
86.6
67.4
57.2
59.30
M*-ICL (16 tasks)
47.5
40.6
88.9
68.0
S7.9
60.58
M*-ICL (32 tasks)
52.6
44.1
91.0
69.4
59.2
63.26
The results are provided in Table We find that the M--ICL significantly outperforms ICL w/o or w/ adpt with any tasks. Specifically, comparing with ICL w/o adpt (standard meat-training and testing), adapting the ICL model with 8 tasks badly harms the performance due to overfitting, and with 16 tasks also do harm, while 32 tasks shows marginally improvement. However, adapting the M2-ICL model with only 8 tasks is enough to surpasses the average performance, and the growing number of significant improvement. ICL w/o adpt, ICL w/ adpt (32 tasks) and M2IICL (8 tasks) have comparable performance. This show the proposed meta-level meta- meta-learning is very effective to improve the few-task domain adaptation ability.
tasks for adaptation brings
23


Reviews:
[{"review_id": "7IUaRkEvaW", "reviewer": "Reviewer_CiDY", "summary": "The paper begins by framing the rise of large language models (LLMs) and their use of in-context learning (ICL) to perform diverse tasks without fine-tuning. This sets up ICL as a unique form of \"learning to learn,\" contrasting with traditional meta-learning models. A theoretical foundation is provided, showing that ICL, particularly with transformers, can express and replicate the behaviors of all main meta-learning approaches (gradient-based, metric-based, and amortization-based). The paper tests ICL’s ability to learn optimal algorithms in three task types, showing that it can indeed achieve data-dependent optimal performance on simple, homogeneous tasks. However, when tasked with mixed types, ICL models exhibit implicit (distribution-sensitive) learning rather than general, explicit algorithm selection. This distinction introduces the core limitation: limited generalizability. To address generalizability and efficiency issues, the authors suggest transferring established deep-learning techniques into the meta-learning realm. They propose meta-level meta-learning for domain adaptability (training ICL models to quickly adapt to new domains with limited data) and meta-level curriculum learning to accelerate pre-training.", "questions": "- How would you define or quantify the \"optimality\" of the ICL model’s learning algorithm in real-world settings with diverse and complex task distributions?\n- ICL demonstrates limited generalizability on tasks from seen types, and is sensitive to the data distribution, why this behavior (distribution-sensitiev) is called as \"implicit\"? How the term \"implicit\" is defined in this paper?\n- Does ICL prioritize simpler or more complex tasks, and what strategies might be implemented to balance performance across tasks of differing complexity levels? Given the challenges of defining a task \"difficulty\" hierarchy in real-world scenarios, how would you propose constructing an effective curriculum for complex, real-world datasets?\n- Does increasing the size or diversity of the training dataset systematically improve generalization across unseen distributions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a theoretical foundation for understanding ICL models as learning algorithms, proving their expressiveness and comparing them to traditional meta-learning algorithms.\n- By comparing ICL with various meta-learning techniques, it offers insights into how and why ICL might outperform traditional meta-learning methods due to its flexibility in the hypothesis space.\n- The paper highlights the critical issue of generalizability in ICL models and offers potential solutions to improve it.", "weaknesses": "- The extensive theoretical background may limit accessibility for practitioners interested in practical applications, as certain concepts (e.g., implicit vs. explicit optimality) are dense and could benefit from simpler, more intuitive explanations.\n- The proposed solutions, meta-level meta learning and meta-level curriculum learning, are promising but lack empirical evaluation on complex, real-world tasks where curriculum ordering could be less clear-cut. Researchers are suing ICL since it can save us from training tons of parameters, although propose method can improve ICL’s performance on specific domain with very limited data for adaptation and transferring deep-learning techniques to the meta-level is a good point, the proposed method lack its practicability.\n- There is a vague treatment of data dependency in \"optimal\" algorithms. The paper mentions data-dependent optimality without fully clarifying how this dependency impacts performance across tasks. The authors do not delve into specific criteria or metrics for measuring optimality across distributions, making it challenging to objectively evaluate their claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbxH5DARk3", "reviewer": "Reviewer_APrh", "summary": "This paper investigates ICL model from a learning-to-learn perspective, examining its expressiveness, learnability and generalizability as a meta-learner. It theoretically demonstrates that ICL, when integrated with a transformer, is expressive to perform various existing categories of meta-learning algorithms. The paper also reveals that while the ICL model can learn data-dependent optimal algorithms, but it is not \"algorithm selection\". Finally, it proposes training ICL models using a task-level meta-learning framework and with curriculum.", "questions": "Could the authors elaborate on whether the proposed methods can be adapted for use with real-world datasets? If so, what modifications or considerations would be necessary to accommodate the complexities of real-world data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The exploration of the ICL model from a learning-to-learn perspective is thorough and yields significant insights, particularly in enhancing our understanding of ICL's role in large transformers. Notably, the clarification that \"ICL is not algorithm selection\" challenges previous interpretations and adds depth to the academic discussion on this topic. \n2. The clarity and logical flow in Sections 3 and 4 are particularly commendable, making complex findings accessible and substantiating the solidity of the research.", "weaknesses": "1. The introduction section could be improved for better accessibility. Key concepts such as \"expressiveness,\" \"learnability,\" and \"generalizability\" should be clearly defined early. For instance, integrating these definitions in the introduction section would enhance comprehension and engagement from the outset. Additionally, Figure 1 needs a clearer explanation to effectively convey its intended message. \n2. Another concern is the validation of the proposed methods, which currently relies solely on simplistic synthetic data. This limitation raises questions about the practical applicability of the methods in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yFYlAqhy3f", "reviewer": "Reviewer_prfg", "summary": "Summary:\nThis paper studies in-context learning (ICL) models from the perspective of meta-learning with solid theoretical analysis and experiments. \n\nContributions:\n1.It theoretically proved that the hypothesis space of ICL encompass the hypothesis spaces of typical meta-learners.\n2. it experimentally verified that ICL with transformer learns the optimal algorithms by designing different types of meta-learning tasks, and several other aspects.\n3.It proposed new techniques including meta-level meta-learning and meta-level curriculum learning to improve ICL’s performance and convergence, respectively.", "questions": "1.Please elaborate why M^2-ICL without adaptation sometimes works worse than ICL\n2.For meta-level curriculum learning, how would you define the complexity of other tasks, such as classification tasks? Could you add some more experiments with regard to few-shot classification tasks?\n3.For task generation, what is the definition of p_f, and how to obtain p_f for different tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.The paper is well-motivated, well-written and easy to follow.\n2.The topic discussed in this paper is interesting and worth studying.\n3.The proposed assumptions are proved and confirmed with solid theoretical analysis and experiments.\n4.The contribution of this paper is solid, please refer to the summary of contributions.", "weaknesses": "1.The paper only focuses on ICL with transformer, ICL with other deep architectures are also worth exploring \n2.Typo: line 482 tunie", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dPzyVTdXRQ", "reviewer": "Reviewer_bz7Z", "summary": "This paper investigates the meta-learning capabilities of in-context learners (ICLs) by comparing them to state-of-the-art meta-learning algorithms. The authors demonstrate how ICL can learn optimal algorithms in various scenarios and propose a strategy to enhance generalization by incorporating techniques from supervised learning. They also explore the impact of applying curriculum learning during the training phase.", "questions": "See Weaknesses part.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "*  The observation that different meta-learning methods generalize best on their respective task sets is interesting. The paper shows that ICL, when trained on a mixture of task sets, can outperform individual meta-learners.\n* The study of generalization across tasks is valuable for understanding the capabilities of ICL in various domains.", "weaknesses": "*  The paper is poorly written, with numerous typos and a lack of clarity, making it difficult to follow. A complete revision is necessary.\n*  The related work section needs significant improvement. Rather than simply listing various meta-learning studies, it should focus on situating this work within the broader literature, especially by incorporating more relevant references on in-context learning and model generalization.\n*  Several key references are missing in the text, including citations for MatchNet, ProtoNet and CNP in line 244.\n*  There are incorrect assumptions present throughout the paper. For example, lines 82–85 incorrectly claim that transformers are inherently permutation-invariant, and line 453 mistakenly implies that all in-context learners are pre-trained, whereas studies such as [1] and [2] demonstrate otherwise.\n* The main text and the appendix feel disconnected, almost as if they belong to separate studies. Mathematical proofs, even if valuable, bring a little contribution if left in the appendix.\n*  The experimental design is weak, and there is no reference to standard literature or pipelines for task creation. The authors should either adopt state-of-the-art experimental methodologies or provide a strong justification for deviating from established practices.\n*  More details about the task generation process  should be included in the main text, not only in the appendix section. This would give readers a better understanding of the experimental setup and strengthen the paper’s presentation.\n*  The results presented in Figure 7, which show that adapting a model to a different distribution leads to higher squared error, seem trivial. Furthermore, the meta-meta-learner does not outperform in a meaningful way, so the contribution of this section is unclear.\n*  The experiments with curriculum learning are insufficient to demonstrate its effectiveness in improving in-context learning. A more robust and comprehensive evaluation is needed to validate these claims.\n*  What is the meaning of Figure 1? What guarantees the orthogonality among the three methods?\n*  In the first part of the paper, the authors claim that ICL can act as different types of meta-learners (gradient/metric/amortization-based) and learn the \"optimal\" algorithm. However, the notion of \"optimal\" is vague and needs to be better defined within the context of the study.\n\n\n\nReferences: \\\n[1] Sewon Min, Mike Lewis, Luke Zettlemoyer, & Hannaneh Hajishirzi. (2022). MetaICL: Learning to Learn In Context.\\\n[2] Kirsch, Louis, et al. \"General-purpose in-context learning by meta-learning transformers.\" arXiv preprint arXiv:2212.04458 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)