Learning to Learn with Contrastive Meta-Objective
Authors: Shiguang Wu
Yaqing Wang
Yatao Bian
Quanming Yao
Venue: NeurIPS 2025 oral ()
Keywords: Meta-Learning
Contrastive Learning
Primary area: general_machine_learning

Abstract: Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans.
	Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's fast learning.
	This is achieved by contrasting what meta-learners learn, i.e., model representations.
	The proposed ConML is evaluating and optimizing the contrastive meta-objective under a problem- and learner-agnostic meta-training framework.
	We demonstrate that ConML integrates seamlessly with existing meta-learners, as well as in-context learning models, and brings significant boost in performance with small implementation cost.

Learning to Learn with Contrastive Meta-Objective
Shiguang Wu', Yaqing Wang'*, Yatao Bian', Quanming Yao 1,4+
Department of Electronic Engineering, Tsinghua University
"Bijjing Institute of Mathematical Sciences and Applications
Department of Computer Science, National University of Singapore
State Key laboratory of Space Network and Communications, Tsinghua University wsg23@mails tsinghua. edu. cn, wangyaqing@bimma. ybi ancnus. edu. sg, qyaoaaOtsinghua. edu. cn
Abstract
Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans. Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's 's fast learning. This is achieved by contrasting what meta-learners learn, i.e., model representations. The proposed ConML evaluating and optimizing the contrastive meta-objective under problem- and leanner-agnostic meta-training framework. We demonstrate that ConMI. integrates seamlessly with existing meta-learners, well as in-context learning models, and brings significant boost in performance with small implementation cost.
Introduction
Learning to learn, also known as meta- meta-learning [38, 41], is powerful paradigm designed to enable learning systems to adapt quickly to new tasks. During the meta-traing meta-training phase, a meta-learner simulates adaptation (learning) across a variety of relevant tasks to accumulate knowledge on how to learn effectively. In the meta-testing phase, this learned adaptation strategy is applied to unseen tasks. The adaptation is typically accomplished by the meta-learner, which, given a set of task-specific training examples, generates predictive model tailored to that task.
As the objective of meta-learning is learn a meta-learner generalize well to unseen tasks where few labeled examples are given, the most conventional objective in meta-training follows the natural idea "train as you test" [46] to minimize the validation loss, by splitting each task into training set (support set) to which the meta-lcarner would be adapted to, and a validation set (query set) to evaluate the adapted model. Beyond "train as you test", people also have introduced regularization to the meta-training objective to improve generalizability. like supervision from stronger models [52, 13, 54], or injecting global information into each task [49]. All these works are under/with the same mini--btth episodic training framework: sampling batch of tasks in each episode to obtain an episodic loss to minimize.
The mini-batch episodic training framework is universal, and naturally gives the information about task identity, which can serve additional supervision for meta-training for generalizability. Inspired by the intrinsic property of human's 's fast learning ability: alignment and discrimination [9, 23, 11]. we hope meta-learner itself should be able to tell if different datasets are from the same task or different tasks by exploiting task identity. A good learner possesses alignment ability
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

model space
H
IRa
mapping
e align
ha1
subset 1
ea
contrastive
task
meta- objective
ha2
subset 2
g(i O)
discriminate
task
hp
models
eb
meta- meta-learner
Figure I: ConML is performing contrastive learning in model space, make the meta-learner itself able to align information from the same task (alignment) while discriminate different tasks to improve generalizability (discrimination).
to align different partial views of certain object, which means they can integrate various aspects or perspectives information to form a coherent understanding [10]. This means a meta-learner should learn similar models from different datasets of the same tasks even if the data are few or noisy, benefiting the meta- veta-testing performance through being robust to the given labeled data. Discrimination involves distinguishing between similar stimuli to respond appropriately only to decisive inputs. This means a meta-leamer should learn different models from different tasks even if some of their inputs are similar, benefiting meta-testing performance through generalization to diverse tasks.
In this paper, propose ConML, modifying the conventional mini-batch episodic meta-training with additional contrastive meta-objective to improve alignment and discrimination abilities meta- learner. Similar to how contrastive learning contrasts unlabeled samples by identity, ConML contrasts the outputs the meta-learner based on task identity. Positive pairs consist different subsets of the same task, while negative pairs come from different tasks, with the objective of minimizing inner-task distance (alignment) and maximizing inter-task distance (discrimination) We design cheap and straightforward ways to obtain model representations for different types of meta-learners.
ConMI. distinguishes itself by being universal: it problem-agnostic. as is based-on mini-batch episodic training where task-identity are intrinsic information; and it is learner-agnostic, as we design toy-ttoommleentt mapping functions from model to representations for different meta-learners. Additionally, it is efficient in that it requires no additional data or retraining. Existing approaches have also leveraged task-level alignment or contrastiveness as additional supervision for improved meta-learning. However, they are based on either problem-specific knowledge [52, 54, 49] or learner- specific knowledge [18, 13]. Thus, they can be improved by exploiting the problem- and learner- agnostic task identity through incorporating with ConML. Our contributions are:
We propose to exploit task-identity as additional supervision in meta-training by emulating human cognitive alignment and discrimination abilities.
We extend contrastive learning from the representation space in unsupervised learning to the model space in meta meta-learning, by designing mapping functions from models to representations for various types of meta-learnett
We empirically show the proposed ConML universally improves the performance of various meta- learning algorithms from different categories with small implementation cost. Furthermore, we show that ConMI. can also improve in-context learning (ICI.) as its training also follows the paradigm learning to learn.
2 Preliminaries: Learning to Learn
learning to learn, which is also known as meta-learning, focuses on improving the learning algorithm itself [38]. We focus on the most general meta-learning setting. Formally, let g(; 0) be a meta-learner

---

that maps a dataset D to model h. i.e., - Let L(D;)) h) represent the loss when evaluating model ht on dataset D using a loss function fly, 3) (e.g., cross-entropy or mean squared error). Given a distribution of tasks p(T) for meta-training, where each task corresponds to dataset D,= ((r75 J= 1=1 the objective of meta-learning is train g(: 0) to generalize well to unseen task r' sampled from p(r'). During meta-testing, give an unseen task r' with labeled dataset Di, (training set or support set) to g(: 0) to generate ft, which is tested by another set from the same task Did (validation set or query set), i.e., evaluated by (DD"; (D*,,;
In meta-training the meta-learner g(: 8) is optimized through a series of episodes, each consisting of a batch b of B tasks, and obtains an episodic loss Le to minimize. The form of Le can be various, while we take the most typical validation loss as example to illustrate our method. Splitting each D, into a training set D = { Br.)) |==1 and a validation set Di { : Vr,i) (..)))=== -= the meta-training objective is minimizing Br= C(Dra) 9 (D 0)). The mini-batch episodic training with validation loss is outlined in Algorithm I. Note that ConML relies on the mini-batch episodic framework, which is general. ConML does not rely on how the specific meta-learner measures Le inside each episode. Here we take the representative validation loss as example and will discuss other forms in Section 5.
Different meta-learners implement their own specific functions within O(; Popular meta-learning ap- proaches can be broadly categorized into the follow- ing types [5]: (i) opiimization-basel approaches [2, 15, 31J, which focus on learning better opti- mization strategies for adapting to new tasks; (ii) Metric-based approaches [46, 39, 40), which lever- age learned similarity metrics; and (iii) Amortization- based approaches [17. 36, 4], which aim to learn a shared representation across tasks, amortizing the adaptation process by using neural networks to di- rectly infer task-specific parameters from the training set; (iv) Furthermore, the emerging ICI. ability in large language models (LLMs) can also be viewed as the consequence of meta-learning [16, 1J, and ICL model is meta-learner with minimal inductive bias [53], so we will also use meta-learner g to express the function of ICL. Details reformulating ICL model as meta-learner to incorporate ConML with are in Section 3.3.
Algorithm 1 Mini-Batch Episodic Training (with Validation Loss).
while Not converged do
Sample batch of tasks b (T)) for All r E
Get task-specific model h, = gDD;; 0);
Get validation loss L(D))), N.
end for
Get episodic loss
Le = B >v L(DF) (DD" Update @ by 0 - A VoLe.
end while
Meta-Learning with ConML
Now, we introduce our ConML which equips meta-learners with the desired alignment and discrimi- nation ability via task-level contrastive learning.
3.1 A General Framework
To enhance the alignment and discrimination abilities of meta-learning, we draw inspiration from Contrastive Learning (CL) [32, 8, 50]. CL focuses on learning representations that are invariant to irrelevant details while preserving essential information. This is achieved by maximizing alignment and discrimination (uniformity) in the representation space [50]. While most existing studies focus on sample-wise contrastive learning in the representation space via unsupervised learning [20, 43, 8), we extend CL. to the model space in meta-learning.
Specifically, we introduce contrastive meta-objective by tasks-level CL in the model space, where alignment is achieved by minimizing the inner-task distance (i.c., the distance between models trained on different subsets of the same task), and discrimination is achieved by maximizing the inter-task distance (i.e., the distance between models from different tasks). Such alignment and discrimination together form the contrastive meta-objective optimize meta-learner. The detailed procedures of ConML are introduced below.
Obtaining Model Representation. To train the meta-learner y. the inner-task distance d and inter- task distance dout are measured the output space of g. also referred as the model space H. A practical approach is to represent the model h = g(D; 0) e H as fixed-length vector E Rd and
3

---

then compute the distances using an explicit distance function ((-,.) (e.g., cosine distance). To form a learner-agnostic framework, we introduce a projection function de ' > IR to obtain the model representations e (h). The details of H. and &i will be specified in Section 3.3.
Obtaining Inner-Task Distance. Alignment is achieved by minimizing inner-task distance. During meta-training, the combined dataset D U Diaal contains all the available information about task The meta-leamer is expected to produce similar models when trained on any subset A of this dataset. Moreover, models trained on subsets should resemble the model learned from the full dataset DD!D For VK S D U Drail we expect E = e,, where e = (g(x;; 0)), e = (g Di 0)) The inner-task distance dia for each task T is computed as:
= Σ Óle et); s.t. KR Th (D U DD
(1)
where {= are K subsets sampled from D U DDal using specifie sampling strategy Te. In each episode, given a batch of task containing B tasks, the overall inner-task distance is averaged as dm = Ereb
Obtaining Inter-Task Distance. Discrimination achieved by maximizing inter-task distance. Since the goal of meta-learning is to improve performance unseen tasks, it is crucial for the meta-learner g to generalize well across diverse tasks. Given the natural assumption that different tasks require distinct task lusk-specifie models, it is essential that g can learn to differentiate between tasks -.s. possess strong discrimination capabilities. To enhance task-level generalization, we define the inter-task distance dotr, which should be maximized to encourage g to learn distinet models for different tasks. Specifically, for any two tasks + r' during meta-training, we maximize the distance between their respective representations, e and e... To make this practical within the mini- episodic training paradigm, we compute d across a batch of tasks sampled in each episode:
d = (L/B(BLBBD) 1)) ErebEr o(e
Training Procedure. ConML optimizes the combination of the original episodic loss La and contrastive meta-objective Lue dn dou..
Algorithm 2 Meta-Training with ConML (with Valida- tion Loss).
while Not converged do Sample a hatch of tasks b ~ (r).
LCMMM = L.e + ALe
(3)
for All 7 E b do
&Sample Ky from (D.; U Di for E {1 (1... K);
The meta-training procedure with ConML is in Algorithm 2. Note ConML is agnos- tic to the form of Le so here take the typical validation loss as example. Com- pared Algorithm 1, ConML introduces additional computations for (g (D; D)) a total of K + 1 times per episode. How- ever, & is implemented as a lightweight function (e.g., extracting model weights), and g(D 0) is already part of the stan- dard episodic training process, with mul- tiple evaluations of (DD; 8) being paral- lelizable. As a result, ConML incurs only a little extra cost in computation (detailed analysis is in Appendix A).
+Get model representation e² ((g(ks:)))
(Get model representation e = u D. 0));
+Get inner-task distance de by (1):
Get task- speciifi model h- = 6))
Get validation loss L(D))) h.): end for
+Get dd= - dtt and d by (2); Get loss LC by (3);
Update 0 by end while
Vol.
"f" indicates additional steps introduced by ConML to Algo- rithm 1.
3.2 Provable Benefits for Generalization
Here, we provide another perspective understand how ConML helps meta-learning. It is provable that a meta-learner which minimizes L. has lower generalization error upper- bound than any other meta-learners. This means ConML serves samples in Dyal in meta-testing, that can be plugged-in any meta-learners.
'safeguard' for the worst case of error due finite
Following [28], the excess risk of a meta-leamnr g(: 0) is defined as:
Acp()) (0)= Er EDI ()) (z), min E, ET-)()) min ((r.))) el (b(())))) v1).

---

Table Specifications of integrating ConML with different meta meta-learners.
Category
Examples
Meta-learner ((;
Model representation (D(())))
Optimization
MAML,
Update model weights 6 VeL(D; h(; 0))
Updated model weight i - VoL(D;;((())
-based
Reptile
Metric -based
ProtoNet,
Build classifier with ba label أ)
Concatenate
MatchNet
EaveD, f(z;; 0)
P3
Amortization
CNPs,
Map D to model weights by (D;
Output of hypernetorr (D;
-based
CNAPs
In-context
In-context
Task-specific prediction for is given by sequential model ((
g( a): where u is dummy input
learning
lcarning
where Hg(o) is the hypothesis class of h given g(: The value of AC)(r)()) > O, means the difference between expectation of validation loss between g(; 0) given finite n examples per task, and the best we can find given g and p(r). First, we find an upper bound Up(r) (0) for (0).
Lemma 1. Denote (0) = C1 / suppllllsi ((xp))
g y) 0)) 1 + Cz. There exists positive constants C1, C2 not related with 6, satisfying V0, A (0) - (0).
Given contrastive meta-objective Le defined above, with mild assumptions and choice of á then have the following theorem:
Theorem 1. Yp(T), (02.) = ming (0), where 02_ arg ming
This means the contrastive meta-objective can exactly serve as surrogate objective of the worst-case meta-testing performance, as described in the above theorem. Note that this holds for any p(r) and g. which indicates the problem- and learner-agnostic benefit of ConML. The proof is in Appendix B.
3.3 Integrating with Typical Meta-Learners
ConML is universally applicable to enhance meta-learning algorithm that follows episodic training. It does not depend on specific form of g or L. and can be used alongside other forms of task-level information. Next, we provide the specifications of and (g 0)) to obtain model representations for implementing ConML. We illustrate examples across different categories of meta meta-learning algorithms, including optimization-basee summarized in Table 1. Appendix C provides the detailed procedures for integrating ConML with various meta-learning algorithms.
metric-based, amortization- based and ICL. They are
With Optimization BBase.. The representative algorithm of optimization-baser meta-learning is MAML, which meta-learns an initialization from where gradient steps are taken learn task-specific models, i.e., gD; 0) h(; 0 - VeL(D; h(; 0))). Since MAML directly generates the model weights, we use these weights as model representation. Specifically, the representation of the model learned by given dataset D is: (bgg 8)) A VeL h[: f)), certain optimization- siinizzation-basedd meta-learming algorithms, such as FOMAML [15] and Reptile [31], use first-order approximations of MAML and do not strictly follow Algorithm 1 to minimize validation loss. Nonetheless, ConML can still be incorporated into these algorithms as long as they adhere to the episodic training framework.
Metric-based algorithms are well-suited for classification tasks. Given a dataset DD for an N-way classification task, these algorithms classify based the distances between input samples {f.; h..ee, J.M and their corresponding labels, where f(; O) is meta-learned encoder and D, represents the of inputs for class j. We represent this metric- based classifier by concatenating the mean embeddings of each class in label-aware order. For example, ProtoNet [39] computes the prototype Cj. which is the mean embedding of sam- ples in each class: C₁ = |D, E(SWMEED f(z;;)) The classifier hy then makes predictions as ply = j x) exp(-d(](z; (),z)) ΣS. depends D through {c,1,- and their corresponding labels, the representation is specified as ()) (cl/e2]... [cN], where [-I-] denotes concatenation.
With Metric-Basee
0), cy)).. Since the outcome model hr
With Aatorimmoatin-aaaaa Based. Amortization- based approaches meta-learns a hypernetwork H 0) that aggregates information from D to task-specific parameter a, which serves as the weights for the
5

---

main-network h, resulting in a task-specific model a); For example, Simple CNAPS [4] uses hypernetwork to generate small set of task-specific parameters that perform feature-wise linear modulation (FiLM) on the convolution channels of the main-network. In ConMI,, we represent the task-specific model lo; (x) using the task-specific parameters a, i.e., the output of the hypernetwork ():): (D((D; 0)) O))
With In-Conlexi Learning (ICL). An ICL model makes task-specific prediction by ,(D se]: 0). where
g is sequential model and D is the sequentialized (prompt), 101,,1, Vi........... Im, 9m The details are in Appendix D. Note that ICL does not specify an explicit output model h(x) = (D; (2);;
instead, this procedure exists only implicitly through the feeding-forward of the sequence model. Thus, obtaining the representation ()(( 0)) by explicit model weights of h is not feasible for ICL. To represent what g learns from D, design to incorporate D with dummy imput u, which functions as probe and its corresponding output can be readout as representation:
(g(; 0)) = u((),,):))
(4)
where is constrained example, for training ICL model on linear regression tasks we can choose 1, and in pretraining of LLM we can choose u ="what is this task?". The complete algorithm of ConML for training an ICL model is in Appendix C.
in the same shape as I, and has consistent value in an episode. For
4 Empirical Studies
We provide empirical studies to understand the effect of ConML on synthetic data, which shows that learning to learn with ConML brings generalizable alignment and discrimination abilities. Code is avaliable at httph::////// //github. com/LARS cttpaab..ccooA--rreerraahhhl ConML.
4.1 Few-Shot Image Classification Performance
To show ConML brings learner-agnostic improvement, we integrate ConML into various meta- learners and evaluate the meta-learning performance on few-shot image classification problem follow existing works [46, 15, 4J. We use two few-shot image classification benchmarks: minilmageNet [46] and mageed [35], evaluating on 5-way 1-shot and 5-way 5-shot tasks.
We consider representative meta-lcarning algorithms from different categories, including optimization- based: MAML [15], FOMAML [15], Reptile [31]; metric-based. MatchNet [46], ProtoNet [39]; amortization-based: SCNAPs (Simple CNAPS) [4]; and the state-of- tate--of-thh-ar the-art ICL-based few-shot learner: CAML [14]. Note that for CAML, ConML only effect the meta veta-training of the ICL mode, not the pretraining of Vit feature extractor. We also incorporate ConML with meta-learners with
Table 2: Meta-testing accuracy (%) ) on minilmageNet and tieredImageNet.
minilmageNet
NireddIaggeNet
Category
Alguritim
Objective
5-way 1-shot
waw Shoh
5. 5-way 1-shot
5-way 5 s.shot
48.75 + 1.25
64.50 + 1.02
51.30 + 1.31
68.25 +: 0.98 72.94 0.98
MAML
ConML
56.25 0.94
67.37 0.97
58.75 + 1.45
Optimization- Based
FOMAML.
48.12 +: 1.40
63.86 1: 0.95
51.44 + 1.51
68.32 +: 0.95
ConMLL
57.64 1.29
68.50 0.78
58.21 + 1.22
73.26 0-78
Reptile
49.21 0.60
64.31
4 1.62
w/ ConMIL
52.82 + 1.06
67.04 17.04:000 0.81
55.01 + 1.28
70.15 + 1.00
43.92 1.03
56.26
18.74 1.06
61.30 0.94
Metric-
MatchNet
w/ ConML
48.75 + 0.88
62.04 : 0.89
53.29 + 1.05
67.86 t 0.77
48.90 + 0.81
65.69 + 0.96
52.50 + 0.96
71.03 + 0.74
ProtoNet
w/ ComML
51.03 0.91
67.35 - 0.72
54.62 + 0.79
73.78 0.75
Amortization
SCNAPs
53.14 1: 0.88
70.43 1: 0.76
62.88 1 1.04
79.82 1. 0.87
55.73 0.86
71.70 0.71
65.06 + 0.95
81.79
Context Learning
CAML.
96.15 0.10
96.57 1. 0.08
95.41 1: 0.10
98.06 0.10
w/ ConMI
97.08 +: 0.10
98.92 0.08
96.56 + 0.09
98.23 0.05
51.33 0.73
68.16 0.59
54.90 + 0.89
72.51 + 0.81
MELR
Other Objective
w/ ConMLL
53.56 + 1.02
70.04 0.95
57.06 + 0.90
74.21 t 0.78
64.80 0.20
81.65 0.14
69.37 0.23
85.36 0.16
LastShot
w/ ConML
66.24 +: 0.72
83.29 + 0.45
71.82 + 0.70
87.05 + 0.49

---

improved meta-trainin objective discussed in Section 5, including: MELR [13] and LastShot [54]. We evaluate the meta-learning performance of each algorithm in its original form (w/o ConML) and after incorporating ConMI. into the training process (w/ ConMI.. The implementation of ConMI. follows the general procedure described in Algorithm 2 and the specification for corresponding category in Section 3.3.
Table 2 shows the results on minilmageNet and rieredImageNet respectively. We uses common
configuration for ConML's 's hyperparameter for all meta-learners: task batch size B = 32, inner- task sampling K 1, and The (D= U ) D Φ(a, b) 1 - * bll and À 0.1. Other hyperparameters related to model architecture and training procedure remain consistent with the original meta-leamnrs'' This demonstrates boosted performance can be brought even without specific hyperparameter tuning for different meta-learners. The performance improvement demonstrates that ConMI. offers universal improvements across different meta-learning algorithms. Note that performance between different algorithms are not comparable. We also show ConML's consistent benefit on different sizes of backbones in Appendix F.
4.2 Cross-Domain Few-Shot Image Classification Performance
To show that ConMI. is problem- agnostic, we provide learner-agnostic improvement
Table 3: Cross-domain results on META-DATASET (accu- racy (%)).
large-scale cross- domain few-shot image classification problem, obtained on META-DATASET [44]. Table 3 shows the results. The backbone and setting of P>M>F [22] is different with the other baselines [44], so they
Baseline
MatchNet
ProtoNet
fo-MAMI
Mo-Proto-MMML
P>MsF
ConML
wlo
w/
w/o
w/
w/o
w/
[woo
w/
woo
w/
ILSVRC
45.0
50.5
52.3
45.55
[49.5
54.3
77.0 78.6
Omniglot
52222 54.6
[59.9
61.2 555.5
63.7
|63.3
69.8
191.7 93.3
comparable across base-
Aireraft
[48.9
51.5
[53.1 54.9
556.2
64.9
[55.9
61.5
|89.7 91.1
lines. ConML. is introduced with the same setting as Section 4.1 (inner-task sampling K = and n. (DD U ) D., ó(a, b)
Birds
62.2
66.8
68.7
68.9
63.6
69.9
668.6
68.6
922.9 94.0
Textures
64.1
66.5
[6800
72.3
66.4
69.4
86.9 87.5
la//|allll (cosine distance) and X = 0.1.). Note that for P>M>F, ConML is integrated into the meta-training phase, and all other phases remain unchanged. As shown, ConML brings consistent improvement.
Quick Draw [2.88
46.7
[88.9
50.0
433.9
48.5
|51.5
53.1
800,2 83.3
Fungi 33.9 36.4
[39.7 40.9
322.
40.6
|39.9
43.7
788.2 80.1
VGG Flower 80.1
84,9
85,2
88.0
|81.7
90.4
|87.1
91.0
|95.7 96.8
Traffic Signs (4..8
49.5
47.1
48.6
|50,9
52.2
|48.8
51.5
89.88 94.0
MS COCO 33.99 40.1
[1.0
42.4
335,3
43.5
43.7
48.9
664.9 68.4
4.3 Model Analysis
We show ConML does not require much efforts on tuning hyperparameters. Furthermore, better performance can be obtained through hyperparameter optimization for specific meta-learners. In this Section we show the impact of key ConML settings: (1) the number of subset samples K, which influences the model's complexity, and (2) the contrastive loss, including the distance function ø, the weighting factor X, and the use InfoNCE as replacement for (din dduu).
4.3.1 Effect of the Number of Subset Samples K
Table 4 presents the results of varying the number of subset samples K. Starting from K = observe moderate performance growth as K increases, while memory usage grows linearly with K. Notably, there is significant discrepancy in both performance and memory (approximately ~ 2x) between the configurations without ConML and with K = 1. However, K has a negligible impact on time efficiency, assuming sufficient memory, as the processes are independent and can be executed in parallel.
4.3.2 The Design of Contrastive Loss
Here, we explore various design factors of the contrastive loss. ConML optimizes the following objective: LCaMM.. = L. + XL.. In the previous sections, to highlight our motivation and perform a decoupled analysis, we used the naive contrastive loss L. din got with the natural cosine distance o(I, y). Here, we consider distance function é Euclidean distance, contrastive loss L.
7

---

in the form of InfoNCE [32], varying contrastive weight À in a wide range. More details are in Appendix G.I.
Table 4: The effect of subset sampling number K.
513-
35.0
w/o
K-1
16
32
Acc.(St)
48.75
56.25
56.08
57.59
157.3
5500
MAML w/ ConML
415
(MR)
2801
| |301
| 4103 | (5831
P0LL Faporiad
Crafs
Time (relative)
1x 1.1x
|
1.1x
| 1.1x| 1.1x
425
Coell
Faporee
000
ProtoNet
Acc,(%))
48.90
51.00
| |52.04] | 522344 |52.48
(a) MAML w/ ConML. (b) ProtoNet w/ ConML.
w/ CouML
Mcm.MB)
7955
|4116 [15175|
| 19943
26449
Time (relative)
1x
1.2x
1.2x 1.2x 1.2x
Figure 2:
The effect of distance function o, con- trastive loss form Le. contrastive weight A.
Figure presents the results. We observe that ConML. can significantly improve the performance of meta-learners across considerable range of X, though setting À too high can lead to model collapse by overshadowing the original meta-learning objective. The choice distance function varies between algorithms, with some performing better with specific functions. Additionally, InfoNCE outperforms the naive contrastive strategy, offering greater potential and reduced sensitivity to hyperparameters.
These findings suggest that we may not have yet reached the full potential of ConML, and there are several promising directions for further improvement. For instance, refining batch sampling strategies to account for task-level similarities or developing more advanced subset- subset-sampling methods could enhance performance further [26, 47, 48]. We also notice that the matching between the chosen distance metric and model representation is the key success. We find that Euclidean distance performs much better than cosine in ProtoNet, since ProtoNet makes classification with Euclidean distance, and ConML contrasts the classifier's weights describing the model's behavior more precisely. Although cosine works generally, it would be interesting to tailor distance metrics for various parameter types (e.g.. classifiers, MLPs, CNNs, GNNs, Transformers). It is also worth noting that a more delicate choice of distance metric is implied by Theorem 1. See details in Appendix B.
In Appendix E. we provide empirical results under synthetic dataset to understand (i) learning to learn with ConML brings generalizable aligmment and discrimination abilities; and (ii) alignment enhances fast-adaptation and discrimination enhances task-level generalizability.
4.4 ICL Performance
Following [16), we investigate ConML on ICL by learning to learn synthetic functions including linear regression (LR), sparse linear regression (SLR). decision tree (DT) and 2-layer neural network with ReLU activation (NN). We train the GPT-2 [34]-like transformer for each function with ICL and ICL w/ ConML respectively and compare the inference (meta-testing) performance. We follow the same model structure, data generation and training settings [16]. More implementation details are provided in Appendix G.2.
1.5
1.5
1.5
1.5
sawedden
1.0
1.0
1.0
souredd
8.
0.5
0.0
0.0
0.0
10
20
10
15
20
20
20
30
Invenbeett eccemples
Incontect eccemples
Inseerbeot esamppless
inconteect eccmroless
(a) LR.
(b) SLR.
(c) DT.
(d) NN.
Figure 3: Varying the number of in-context examples during inference of ICL.
Figure 3 shows the performance, where ICL w/ ConML always makes more accurate predictions than ICL w/o ConML. Table 5 shows the two values to show the effect ConML brings to ICL: Rel. Min. Error is ICL w/ ConML's 's minimal inference error given different number of examples, divided by ICL's; and Shot Spare is when ICL. w/ ConMI. obtain an error no larger than ICL's minimal error, the difference between the corresponding example numbers. One can observe significant improvement.
8

---

Table 5: Relative minimal enror (Rel. Min. Error) and spared example number to reach the same (Shot Spare) comparing ICL w/ and w/o ConML.
Function
(10 shot)
SLR (10 shot)
shot)
(40
(max prompt
Rel. Min. Error
0.12 0.09
0.19 # .06
0.81 + 0.12
0.74 0.19
Shot Spare
4.68 + 0.45
-3.94 + 0.62
44222 1.29
111:25 + 2.07
The effect of ConML on ICL is without loss of generalizability to real-world applications like pretrained LLMs.
5 Relation with Existing Works
Task-Identity Meta-Learning There are existing works attempt to leverage task-identity in- formation into meta-learning, but no "counterpart" for ConML. We discuss them in two categories: (i) The first category using problem-specific information, while ConML uses problem-agnostic information thus be plugged-in these methods and brings improvement. They primarily focusing on few-shot image classification problem [12, 19, 33], and require static pool of base classes for meta-training and class-level alignment [51, 13, 54, 42, 49]. These problem-specific approaches limited by their focus on few-shot classification and cannot effectively handle dynamic or diverse tasks, nor can they generalize other meta-learning problems beyond classification. As such, they are not directly comparable with ConML. However, ConML can be integrated into these methods. Though they introduce new objectives other than validation loss by additional modules or steps. but they all work under/with the general mini-batch episodic training, either by replacing the steps to obtain L. in Algorithm with their steps to obtain episodic loss, or introduce additional steps outside Algorithm 1. We demonstrate in Section 4.1 that incorporating ConML leads to performance gains. (ii) The second category includes works that are learner-specific but not problem-specifte For example, [18] and [27] explore contrastive representations for neural processes. However, their methods are tightly coupled with specific meta-learners that involve explicit model representation vectors, which can be seen as special cases of ConML within coortization-basee meta-learners.
Contrastive Learning with Meta-Learning Some studies involve both meta-learning and con- trastive learning key components, but they are not directly related ConML. [30] reformulates contrastive learning through meta-learning for better unsupervised learning, while [55] proposes an optimization-basee meta-learner inspired by contrastive Hebbian learning in biology, which is not related to the contrastive learning used in unsupervised learning. [25] introduces contrastive set representations for unsupervised meta-learning but does not integrate them with the general meta-lcarning framework or model.
6 Conclusion, Limitations and Discussion
In this work, we propose ConML, a universal, learner-agnostic contrastive meta-learning framework that emulates the alignment and discrimination capabilities integral to human fast learning, achieved through task-level contrastive learning in the model space. ConML can be seamlessly integrated with meta-training procedure of existing meta-learners, by modifying the conventional mini-bated episodic training, and we provide specific implementations across wide range of meta-learning algorithms. Empirical results show that ConML consistently and significantly enhances meta-learning performance by improving the meta-learner's fast-adaptation and task-level generalization abilities. Additionally, we explore in-context Icarning by reformulating within the meta-lcarning paradigm, demonstrating how ConML can be effectively integrated to boost performance.
The primary contribution of ConML is offering universal framework built and on the general meta- learning setting and training procedure, to reflects the inherency of alignment and discrimination as meta-training objective and the efficacy of learning to learn with contrasting model representation. The cost of ConML is additional training cost, as dicussed in Section 4.3.1, which is moderate but indelible under such framework. The current implementation of ConML is relatively primitive, as discussed in Section 4.3.2, there are many directions for further improvement, such as optimizing sampling strategies, task-scheduling. refining the contrastive strategy and tailoring model representations and distance metrics.

---

Acknowledgment
Wang is sponsored by Beijing Nova Program. Q. Yao is supported by National Natural Science Foundation of China (under Grant No. 92270106) and Beijing Natural Science Foundation (under Grant No. 4242039). Bian supported by the National University of Singapore SoC (grant no: A-0010308-00-00).
References
[1] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In ICLR, 2023.
[2] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurlPS, 2016.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
[4] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In CVPR, 2020.
[5] John Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann, Sebastian Nowozin, and Richard Turner. Memory efficient meta-learning with large images. In NeurIPS, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurIPS, 2020.
[7] Tadeusz Caliñski and Jerzy Harabasz. dendrite method for cluster analysis. Communications in Statisticx-Theory and Methods, 1974.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[9] Zhe Chen. Object-based attention: A tutorial review. Attention, Perception, & Psychophysics, 74.
[10] Brian Christian. The alignment problem: How can machines learn human values? 2021.
[11] Stella Christie. Learning sameness: Object and relational similarity across species. Current Opinion in Behavioral Sciences, 2021.
[12] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. NeurlPS, 2020.
[13] Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. MELR: Meta-learning via modeling episode-level relationships for few-shot learning. In ICLR, 2021.
[14] Christopher Fifty, Dennis Duan, Ronald G Junkins, Ehsan Amid, Jure Leskovee, Christopher Ré, and Sebastian Thrun. Context-aware meta-learning. In ICLR, 2024.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In ICMI, 2017.
[16] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. In NewrIPS, 2022.
[17] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In ICML, 2018.
10

---

[18] Muhammad Waleed Gondal, Shruti Joshi, Nasim Rahaman, Stefan Bauer, Manuel Wuthrich, and Bernhard Schölkopf. Function contrastive learning of transferable meta-representations. In ICMI., 2021.
[19] Markus Hiller, Rongkai Ma, Mchrtash Harandi, and Tom Drummond. Rethinking generalization in few-shot classification. NeurIPS, 2022.
[20] R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Maroii-Mrr Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
[21] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 1991.
[22] Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In CVPR, 2022.
[23] John E Hummel. Object recognition. Oxford Handbook of Cognitive Psychology, 2013.
[24] Louis Kirsch, James Harrison, Jaseha Sohl-Dickstein, and Luke Metz. General- purpose in- context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022.
[25] Dong Bok Lee, Seanie Lee, Kenji Kawaguchi, Yunji Kim, Jihwan Bang, Jung- Woo Ha, and Sung Ju Hwang. Sepf-super set representation learning for unsupervised meta- meta-learming. In ICLR, 2023.
[26] Chenghao Liu, Zhihao Wang, Doyen Sahoo, Yuan Fang, Kun Zhang, and Steven CH Hoi. Adaptive task sampling for meta-learning, In ECCV, 2020.
[27] Emile Mathieu, Adam Foster, and Yee Teh. On contrastive representations of stochastic processes. In NewrlPS, 2021.
[28] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. JMLR, 2016.
[29] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. MetalCL: Learning to learn in context. In ACL, 2022.
[30] Renkun Ni, Manli Shu, Hossein Souri, Micah Goldblum, and Tom Goldstein. The close relationship between contrastive learning and meta-learning. In ICLR, 2021.
[31] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1803.v8800.222 0.999, 2018
[32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807. 007.033748 2018
[33] Rashindrie Perera and Saman Halgamuge. Discriminative sample-guided and parameter-efficient feature space adaptation for cross-domain few-shot learning. In CVPR, 2024.
[34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
[35] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for supi-supe few-shot classification. In ICLR, 2018.
[36] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurIPS, 2019.
[37] Peter Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics. 1987.
11

---

[38] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
[39] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
[40] Flood Sung, Yongxin Yang. Li Zhang. Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. CVPR, 2018.
[41] Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to Learn. 1998.
[42] Pinzhuo Tian and Yang Gao. Improving meta-learning model via cota-contrastive loss. Frontiers of Computer Science, 2022.
[43] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[44] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset datasets for learning to learn from few examples. In ICLR, 2020.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[46] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In NeurlPS, 2016.
[47] Jingyao Wang, Wenwen Qiang, Xingzhe Su, Changwen Zheng, Fuchun Sun, and Hui Xiong. Towards task sampler learning for meta-learning. IJCV, 2024.
[48] Qi Wang, Yiqin Lv, Yixiu Mao, Yun Qu, Yi Xu, and Xiangyang Ji. Robust fast adaptation from adversarially explicit task distribution generation. In SIGKDD, 2025.
[49] Ruohan Wang, John Isak Texas Falk, Massimiliano Pontil, and Carlo Ciliberto. Robust meta- representation learning via global label inference and classification. IEEE TPAMI, 2024.
[50] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICMI., 2020.
[51] Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In ECCV, 2016.
[52] Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning model the tail. In NeurlPS, 2017.
[53] Shiguang Wu, Yaqing Wang, and Quanming Yao. Why in-context learning models are good few-shot learners? In ICLR, 2025.
[54] Han-Jia Ye. Lu Ming, De-Chuan Zhan, and Wei-Lun Chao. Few-shot learning with strong teacher. IEEE TPAMI, 2022.
[55] Nicolas Zucchet, Simon Schug, Johannes Von Oswald, Dominic Zhao, and João Sacramento. A contrastive rule for meta-learning. In NewrlPS, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this work, we propose ConML, a universal, learner-agnostcc contrastive meta-learning framework that emulates the alignment and discrimination capabilities inte- gral to human fast learning, achieved through task-level contrastive learning in the model space. ConML can be seamlessly integrated with meta-training procedure existing meta-learners, by modifying the conventional mini-batch episodic training, and we provide specific implementations across a wide range of meta-lcarning algorithms. Empirical results show that ConML consistently and significantly enhances meta- meta-lcarning performance by improving the meta-learner's fast-adaptation and task-level generalization abilities. Addition- ally, explore in-context learning by reformulating it within the meta-learning paradigm, demonstrating how ConML can be effectively integrated to boost performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations of the work performed by the authors, as summarized in the last paragraph of the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
13

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes].
Justification: For each theoretical result, the paper provides the full set of assumptions and a complete (and correct) proof, involving assumption and results in Section 3.2 and proof in Appendix B.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The experiment setting and methods are fully described.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
14

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code is provided at https /tt///gitttt http:////gggtubb...co/ooo///////
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
cublic/guides httgg/idde//ooeeuuuiiieeeeeeeees
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: /tips.cc/pppi hwwpbblicgguideessee http://ggggieee//eeuuuuiiiissss
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Most key details directly mentioned and discussed, while the others be figured out in the provided code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
15

---

Most results are provided with standard derivation. Exclusions are Table 3 that requires too much source we could not afford do multiple times, and neglecting for paper space limitation at Table 4.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No].
Justification: Table 4 has shown the detailed relative consumption comparing the proposed method and standard method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurbl: cc cctpublic/tthiceeccccccpopppp cetuidelineds
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
16

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA],
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Justification:
17

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:[NA]
18

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ //neurips...../ cctp.ccc///oiiia e2///on/erenc0s/22255/////////// for what should or should be described.
19

---

A
Complexity Analysis
We compare the relative complexity of computing the original meta-objective and the additional contrastive objective introduced by ConML.
A.1
ICL
For ICL model like LLM, ConMI does not obtain model representation by explicit model parameters, but by simply adding an additional token to the forward-pass (u in (4)). Which means pretraining LLM with ConML only requires K/ (K: subset sampling number, nı: average sentence length, typically K/n << 1) times computation comparing with pretraining a LLM without ConML, regardless of the model size.
A.2 Typical Meta-Learnness
For typical meta-learners, denote the model representation has parameters, i.e., e E Rd. We discuss about the complexity of the original computation path à = gg(D; computation path h = 0) - vh(h) - L. introduced by ConMI. We consider giving a single input sample in 1-d vector, the complexity On-Ce calculate the loss L. E(h(z).)) y), and the complexity O₂ (n-) to calculate Le d(1b(h), (hh)).
L. and the additional
For optimization-based, e.g., MAML, we have d = 10] = 1/). We consider /t as a I-layer MLP, with each average layer size (I/1/T)1/2 * (1/1/11///2 With a single input sample, = 0l (00(0((/////////) (1h1/1)3/2), On- = =((d) + O((4)) = O(I)). While I << [ have On-cc > Op-
For metric-based, e.g., ProtoNet, 0 corresponds the parameter in feature extractor like CNN. is the final classifier which makes prediction by Euclidean distance, which can be viewed as linear classifier with parameter size in N * |4//N. d equals to the N (ways per task) times the dimension of the embedding of a each sample |h//N, = [h]. A sample x E RM//N. We have On-L. = O(K)/NN, OR- = ((d) + ((d) = ((hl).
With [M] >> N. On-L, > Oh-
For cetootiizationbbbedd.. parameters generated by hypernetwork Ho( |d] Ok-+g(h)=C.. O(d) + O(d) (g)). Consider layers in main-network modulated by HD) feature-wisely, the q =
Simple CNAPs. Denote q as the dimension of task task-adaptive
y /hl/l, OK-L. = Oll * (I/l/1)3/2) + O(g/t). With 1 << |//. we have O On > 0
To summarize, for ICL model like LLMs, the complexity to pretrain with ConML is atk = times the complexity to pretrain without ConML. For typical meta-learners, the additional introduced objective in ConML is comparably less than the complexity of the original meta-training objective, which verifies the empirical computation cost presented in Table 4 in main text.
20

---

B Provable Benefits for Generalization
Here we first provide the proof of Lemma I which shows Up(r) (0) is upper bound of the excess risk of meta-learning (O), and then the proof of Theorem I which shows minimizing contrastive meta-objective is minimizing (0).
We need two preliminary results:
Lemma (Upper Bound from (28]). VO, (0p(r) < (0) where
@ (p)))(0) = Ble ()) C2.
with Ci, C2 O, dg
4C2 = dg
Lemma 3 (Universal Approximation of MLP from [21]). Let R + be a non-constant, bounded, and continuous function. Let K be a compact subset of IR.. The set of real-valued continuous functions on K is denoted by C(K). For any function tolerance > 0, there exists an integer N (the number of neurons in the hidden layer), and real constants Us E TR and vectors wi E T2nh for 1..... N, such that we can define the MLP output
C(K) and for any error
function F K - R as:
N F(x)
Σ
u0 (wfx x b.)
1=1
which satisfies Yo K and [f(x) - F(x) <d.
B.1 Proof of Lemma
Proof. We use the Universal Approximation of MLP to bound the & in (p)r)(0).
For the model space H (recall h g(D; 8) H.) is closed and bounded by the meta-leamnr's
hypothesis, which is a compact set on IR |Ri,, we can apply Theorem 3 on in & (0), There exists two-layer MLP F with nonlinear activation function d. Then
VXg E H.
(xx))) F((x)) <,
(5)
where X, represents the parameter vector learned by g(; 0) from sample (x, v).
As Xg E H. F(gg) IR and ||F((x))l (1x)(X)) || || by definition, we have
N
N
Bwi, ". UN E R pl
Σ
| < N, F(X,) = Σ (wwk, X,))
As can be arbitrarily small by selecting large enough N, we approximately rewrite (5) as (X2g)) F(X,) which is not completely rigorous but no harm to our proof. We have
Erpplr) ((
v): v): = Ex
[ [a((w
where P (a(0)() is the distribution of X, output by meta meta-learnr g(: f) on defined p(T), x,
Let the activation function a has Lipschitz constant Le, and (0)) = 0. We have
Ex PXxPP(t
Σ Ca((wi, (M(WW,X,0) x,)) ST2 |w 2 Ex Pap [1
wk
x
|wal
<1
sup Ex- a (1,, X) ||v|
So we have
$ pp)) Ci Plewe
C2
I C1
L2N sup E (() |M||
= (0)
So (0) is upper bound of he the excess risk of a meta-learner (0)
21

---

B.2 Proof of Theorem 1
We need assuming vD, (D; 0)] 1, and choosing the distance function & Le to be Φ(a. b) a-b [allib for dt and Φ(o, b) -( Ta lbb )² for dout, We will discuss about the rationality of such
assumption and choice after proof.
Proof. On the one hand, we have
Up(r)(0) =G,
C1,
L2N sup E Erngtt ((((())))) C2
= Ci
Ci
12N sup Ex-Poooxin x)2] + C2 x.
1101151
L
ViV
+ C2.
HH]
where the minimum of (0)
L2N
U (0) = Cip
+ C2
[7|
achieved and only if
ExxP we [e, = |H
sup ||
which is achieved if and only if
r( H
X = 1, 1, Po()) (X)
RI
2T
ie., X uniformly distribute on the unit sphere in RIH)
On the other hand, by definition we have La =d dot
(Xriss XTw)
(Xr,
=Fx,,n~Pa Paaaa-aaaaa X N.w
||x, |||xx,111 )2)
(Xy Xr.nn)
(X, X,,)
-En-Palett) Palereeeaaa
+
)21
|| X
|x X
Tiny X
(X,, X,1) 131.
-ErPP Patteeamn.......Paee.a.eee (
E U(
1 Xx,ee
1x+11|x++111
For arbitrary subset sampling strategy Te, we have
min 2 1 ||
[H]
where the minimum of Ce
C = 1 [7(]
is achieved if and only if
( Xr.ni X
1,
| 7.x X 7,00
(X,, Xxr)
1
EirPatee [(
)2
X xX
HH
22

---

which is achieved if and only if
I(2 [K4
M|||| = 1, Pgco
P.0........pm)))) (r) = (X),
2T DK 2
where 6, is the Dirac-delta function centered on T. Then ||| 1,
T) dr
l|rl|==1
I( [Hl]
8, (X) dr
||-1|=1 2T
r |H
MI
2x
Combining both hands, we have
02 = arg min Co(g(:0),
0
I'(
(24)
= *|XX|
2T 1H
L2N
+ (f2))
V
C2 = min (p)
[76]
Now we discuss the implication of assuming VD, (D]( = 1, and choosing the distance function a- in L. to be b) |6 for d and o(a, b) = - )a for dout, [Ig(D;0)]I 0) 1 can be viewed regularization model weights of h that prevents trivial solution and enhances generalization.
The choice of is more inspiring: comparing with ordinary cosine distance for both dd and dout above form modifies dout, For maximizing dout, if we also choose ordinary cosine distance p(a, اল given a pair of tasks, it is optimized if and only if the two tasks are "opposite".
i.e. still coupled, while the modified - (1aif1/)) )a is maximized if and only if the two tasks "orthogonal", i.e., decoupled. The latter form preferred be more reasonable. However, this understanding has only come to our mind after our work, so we have not implemented in our experiments. This could be studied as future direction.
23

---

C Specifications of Meta-Learning with ConML
Here, we provide the specific algorithm process of representative implementation ConML, including the universal framework of ConML (Algorithm 3), the most efficient implementation of ConMLwith K and The ( U Dyal D (Algorithm 4), training ICL model with ConML (Algorithm 5), MAML w/ ConML (Algorithm 6), Reptile w/ ConML (Algorithm 7), SCNAPs w/ ConML (Algo- rithm 8), ProtoNet w/ ConML (Algorithm 9).
Algorithm 3 ConML.
Input: Task distribution p(T), batch size B, inner-task sample times and sampling strategy Tx- while Not converged do Sample a batch of tasks b ~ p (T). for All E b do
for k: = 1, 1,2,.... .... K do
Sample Kj from nx (Dr U D
Get model representation ex = (g(:: 0));
end for
Get model representation et = (D( U DD
Get inner-task distance det by (1);
Get task-specific model h (DD 0);
Get validation loss C(Dral h,);
end for
Get d
= Erer Ereb dtth and d by (2);
Get loss LCoaMI. by (3);
Update by 8 VoL.
end while
Algorithm 4 ConML (K- 1).
Input: Task distribution p(T), batch size (inner-task sample times K and sampling strategy T (D" U ) = D
while Not converged do
Sample a batch of tasks b ~ (T).
for All E b do
Get task-specific model h, = (D 0), and model representation e.fh = <b([k=))))) Get model representation et = (D (
Get inner-task distance df by (1);
Get validation loss L(DYal: h;));
end for
Get ddn
= 1 ErEb d and dmt by (2);
B Get loss LCoMM.. by (3);
Update by 0 < 0 VoL.
end while
24

---

Algorithm 5 ICL with ConML (ICL w/ ConML).
Input:
Task distribution p(r), batch size B, inner-task sample times K and sampling strategy Te. dummy input u (probe). while Not converged do
Sample a batch of tasks b ~ (T).
for All r E b do
for k: = 1,2, K do
Sample from ne (D+);
Get e,t = (( u/: 0):
end for
Get e = g((D,, u):
Get inner-task distance di by (1);
Get task loss 1
m Ei f(yr.+1,, ((D Fr,4+1]:0))
end for
Get d
Erec di and d by (2);
Get episodic loss L.e Ereb 1 m-1 Elyrit ((0 m 2i=
Update 9 by 8 VolLe A(d ))
end while
Algorithm 6 MAMI. w/ ConMI.
Input: Task distribution p(r), batch size B, inner-task sample times K 1 and sampling strategy
Th
while Not converged do Sample a batch of tasks b pB(T)) (T).
for All 7 E b do
for 1: 1,2, K do
Sample from n, (D", U Di
Get model representation ex VoL(kui ha);
end for
Get model representation e = VoL(D) u Dyal, ho).
Get inner-task distance di" by (1);
Get ssecifcc model hg-v.<(Pp;0))
Get validation loss L(Dyal, he so
end for
Get dd
= 1 Ersb E d and dd" by (2);
Get loss LComML by (3);
Update A by 0 <- 0 VoL.
end while
Algorithm Reptile w/ ConML.
Input: Task distribution p(r), batch size B. (inner-tase task sample times K and sampling strategy
Ty (D" U Di D while Not converged do Sample batch of tasks b ~ BT)
for All 7 E b do
for k 1, 2,.. K do
Sample Λ& from n, (Dr);
Get model representation ex 0 VoL(kui ha);
end for
Get model representation et = VoL(D. Dial he).
Get inner-task distance d by (1);
end for
Get di
ErEb die and our by (2);
Update 9 by 0 t 0 1, Ereole; f) Ao d=
end while
25

---

Algorithm 8 SCNAPs w/ ConML.
Note: Here twe corresponds to the feature extractor fo: Hg corresponds to the task encoder Js in [4].
Input: Task distribution p(T), batch size B, inner-tase task sample times K and sampling strategy Tx- Pretrain h with the mixture of all meta-training data;
while Not converged do
Sample a batch of tasks b ~ p (T).
for All 7 E b do
for k = 1, 2........ .....KK do
Sample from nN (D" U )
Get model representation ex* = Hb(#));
end for
Get model representation e = (DD : Dyal);
Get inner-task distance di by (1);
Get specifpe model by FiLM h Ho
Get validation loss L(DYal: hy);
end for
-TEr Get loss LCoMMI. by (3);
Get n
ErEb die and d by (2);
Update 0 by 0 0 VoL.
end while
Algorithm 9 ProtoNet w/ ConML (N-way classification).
Input: Task distribution p(r), batch size B, inner-task sample times K 1 and sampling strategy
Tx
while Not converged do Sample a batch of tasks b ~ (T).
for All b do
for 1: = 1,2. K do
Sample Kı from मw (D) DYal);
Calculate prototypes Cj = s Lmm fo(x;) for = 1,...
Get model representation N [c1|02] ... CNN]:
end for
Calculate prototypes сソ- D. Σ Wi, fo(2:) for j 1,...,, ,N;
Get model representation et = [c1 [cx]]
Get inner-task distance di by (1);
Get task-specific model h/c [alleel)))ew) which gives prediction by ply j x)
ezpl c
Σy eapl [ap))__[((())...)))
Get validation loss C(Dyal, lellew))))))
end for
Get d
= = dt and nm by (2);
Get loss LcOMML by (3);
Update θ by 0 - 8 V.C.
end while
26

---

D ICL with ConML
D.I ICL
ICL is first proposed for LLMs [6], where examples in a task are integrated into the prompt (input- output pairs) and given a new query input, the language model can generate the corresponding output. This approach allows pretrained model address new tasks without fine-tuning the model. For exam- ple, given "happy->poxitive: aud->negative; blue->", the model can output "negative", while given "green->cool; yellow->warm; blue->" the model can output "cool". ICL has the ability to learn from the prompt. Training ICL can be viewed as learning to learn, i.e., meta-learning [29, 16, 24j. More generally, the input and output are not necessarily to be natural language. In ICL. sequence model Te (typically transformer [45]) is trained to map sequence [x1. V1, I2, y2, ... #m--1 1, 9m Fm] (prompt prefix) prediction ym. Given distribution /' of training prompt t, then training ICL follows an auto-regressive manner:
min P() i=0
E(1,,+1, To To Ut.1," ,(,,41)))
(6)
It has been mentioned that the training of ICL can be viewed an instance of meta-learning [16. 1] as To learns to learn from prompt. It has been pointed out that ICL model is meta-learner with minimal inductive bias [53]. In this section first formally reformulate To meta-learner g(: 0), then introduce how ConML can be integrated with ICL.
D.2 A Meta-learning Reformulation
Denote a sequentialized ID as D where the sequentializer is default to bridge p(T) and P(I). Then the prompt [rr,1, Vr.1, ; Fr.m: Vr.mm can be viewed as Djr which is providing task-specific infor- mation. Note that ICL does not specify an explicit output model h(x) (D: 0) (x); instead, this procedure exists only implicitly through the reeding-forward of the sequence model, i.e., task-specific
prediction is given by g(DD, z²: Thus we can reformulate the training of ICL (6) as:
min ETp m E.T ((v+,i4,, ((Pr, 07,4+];;
The loss in (7) can be evaluated through episodic meta-training, where each task in each episode is sampled multiple times to form D and D. evaluate the episodic loss Le in an auto-regressive manner. The training of ICL thus follows the episodic meta-training (Algorithm 1), where the validation loss with determined D and Dial. L(Dral, (DD"; 8)), is replaced by loss validated in the auto-regressive manner: 1 > E(yr,41., ((D, 7. xr] 0)).
D.3 Integrating ConML with ICL
Since the training of ICL could be reformulated as episodic meta-training, the three steps to measure ConML proposed in Section 3 can be also adopted for ICL, but the first step to obtain model representation (g((D, 8)) needs modification. Due to the absence of an inner learning procedure for predictive model for prediction h(x)) = ((D (z), representation by explicit model weights of h is not feasible for ICI.
To represent what learns from D, we design to incorporate D with dummy input u, which functions as probe and its corresponding output can be readout as representation:
(()(D:0) u()))
(8)
where u is constrained be in the same shape x, and has consistent value in an episode. The complete algorithm of ConML for ICL is in Appendix C.For example, for training a ICL model on linear regression tasks we can choose u 1, and in pretraining of LLM we can choose ="what is this task?". From the perspective of learning to learn, ConML encourages ICL to align and discriminate like it does for conventional meta-learning while the representations to evaluate inner- and inter- task distance are obtained by probing output rather than explicit model weights. Thus, incorporating ConML into the training process of ICL benefits the fast-adaptation and task-level generalization ability. From the perspective of supervised learning, ConMI. is performing unsuper- vised data augmentation that introduces the dummy input and contrastive objective as additional supervision to train ICL.
27

---

E Experimental Results on Synthetic Data
We begin by conducting experiments synthetic data in controlled setting to explain: (i) Does ConML enable meta-learmen to develop alignment and discrimination abilities? (ii) How do alignment and discrimination boost meta-learning performance? We take MAML w/ ConML as example and investigate above questions with few-shot regression problem following the same settings in [15]. Each task involves regressing from the input the output of a sine wave, where the amplitude and phase of the sinusoid are varied between tasks. The amplitude varies within [0.1, 5.0] and the phase varies within [0. .). This synthetic regression dataset allows to sample data and adjust the distribution as necessary for analysis. The implementation of ConML follows a simple intuitive setting: inner-task sampling K and T (Dr U ) = Dir Φ(a, b) = --1| all [o]l (cosine distance) and À 0.1. The meta-leaner is trained on meta-training distribution with amplitudes uniformly distributed over 10.1, 5), and each training task has a fixed N = 10. For Figure 4(f), the meta-learner is tested on tasks with amplitudes uniformly distributed over [0.1 + 8, 5 + 6), where 8 is shown on the z-axis.
13000
30000
Number
400
MAML d'
MAHL wi d
200
MAMAL deo
0.2
MAMM wi diS
CorMiL
ConMi
Distance
(a) Model MAMI.
distribution of (b) Inner-task distance distribution.
(c) Varying test shots.
350000
Rotatian 0--
350000
MAMIL
MAML
10000
MNML ml
161ML w/ dM
510ce
MAMI W/ ConML
MM/L w/ ConML
Dillnnee
Distribution Shift
(d) Model distribution of MAML (e) Inter-task distance distribution. w/ ConML.
(f) Varying test distribution.
Figure 4: Evaluation of ConML on synthetic few-shot regression.
Learning to learn with ConML brings generalizable alignment and discrimination abilities. If optimizing d and d does equip meta-leamer with generalizable alignment and discrimination, MAML w/ ConML can generate more similar models from different subsets of the same task, while generating more separable models from different tasks, though these tasks are unseen. This can be verified by evaluating the clustering performance for model representations from unseen tasks. Figure 4(a) and 4(d) visualize the distribution of these models, where each point corresponds to the result of a subset and the same color indicates sampled from the same task. We randomly sample 10 different unseen tasks. For each task, we sample 10 different subsets, each containing N = 10 samples. Using these 100 different training sets D as input, the meta-learner generates 100 models. It can be obviously observed MAML w/ ConML performs better alignment and discrimination than MAML. To quantity the results, we also evaluate the supervised clustering performance, where task identity is used as label. Table 6 shows the supervised clustering performance of different metrics: Silhouette score [37] and Calinski-Harabnn
index (CHI) [7]. The results indicate that MAML with ConML significantly outperforms standard MAML across all metrics. These findings confirm that training with ConMI. enables meta-learners to develop alignment and discrimination abilities that generalize to meta-testing tasks.
28

---

Table 6: Meta-testing performance (MSE) on few-shot regression problem and clustering performance (Silhouette and CHI) of model representations.
Method
MSE (5-shot)
MSE (10-shot)
Silhouette
CHI
MAML
||
.677 .038
.068 + .002
.107 + .060
31.6 + 2.5
MAML w/ ConML
||
.399 = .010 .040 = .001
.195 = .062
39.2 =: 2.6
Alignment enhances fast-adaptation and discrimination enhances task-level generalizability. We aim to understand the individual contributions of optimizing d' (alignment) and d (discrim- ination) to meta-learning performance. conventional unsupervised contrastive learning, both positive and negative pairs necessary avoid learning representations without useful information. However, in ConML, the episodic loss Le plays fundamental role in "learning to learn," while the contrastive objective serves additional supervision we consider two variants of ConML: MAML w/ d which optimizes L. and dmn MAML w/ dout which optimizes Le and d Figure 4(b) and 4(e) visualize the distribution of d and dod respectively, where the dashed lines mark mean values. We randomly sample 1000 different unseen tasks, with 10 different subsets (each containing N 10 samples) per task. These subsets are aggregated into single set of N = 100 to obtain e for each task. Smaller d' means better alignment and larger d means better discrimination. We can find that the alignment and discrimination abilities are separable, generalizable, and that ConML effectively couples both. Figure 4(c) shows the testing performance given different numbers of examples per task (shot). The results indicate that the improvement from alignment (MAML w/ d is more pronounced in few-shot scenarios, highlighting its close relationship with fast-adaptation. Figure 4(f) shows the diitr--distritution testing performance. As the distribution gap increases, the improvement from discrimination (MAML w/ dd is more significant than from alignment (MAML w/ dd"), indicating that discrimination plays a critical role in task-level generalization. ConML leverages the benefits of both alignment and discrimination.
enhance alignment and discrimination. Thus,
29

---

Experimental Results Obtained Using Different Backbones
In the main text, we have used the following backbones in experiment: Conv4: MAML, FOMAML, Reptile, MatchNet, ProtoNet; ResNet12: MELR, Lastshot; ResNet18: SCNAPs; ViT-base: CAML.
To study the effect of ConML different backbones, compare MAML w/o and w/ ConML, reporting the minilmageNet 5-way 1-shot accuracy. We specifically demonstrate the effect of equipping the model with ConML using the change in accuracy (A Acc).
Table Meta-testing accuracy (%) on minilmageNet 5-way 1-shot, using different backbones.
Backhone
Conv4
Conv6
ResNet12
ResNet18
MAML w/o ConMI.
48.7
50.9
57.2
56.3
MAML w/ ConML
56.2
57.8
64.5
64.9
A Acc
+7.5
+6.9
+7.3
+8.6
Table 7 shows the results. As the network deepens, A Acc shows little change. Although deeper networks generally achieve higher baseline accuracy, making further improvements challenging. ConML consistently enhances performance even outperforming shallower architectures, For instance, while ResNet18 (which may be overly deep for 1-shot MAMI. on minilmageNet) generalizes worse than the shallower ResNet12 without ConMI., ConMI. boosts ResNet18 s by a significant Acc (+8.6%), surpassing ResNet12 with ConMI. This suggests that deeper networks can better leverage ConML's alignment and discrimination capabilities.
30

---

G Implementation Details
G.1 Model Analysis
ConML optimizes the following objective: LC = Le + ALe, where Le is the episodic loss, and Le is the contrastive loss. In the previous sections, highlight our motivation and perform a decoupled analysis, we used the naive contrastive loss L. d dm with the natural cosine distance (ba,,) v) = z'y
| Here, we also considered a manually bounded Fuclidean distance o(2, y) sigmoid( vl)). Beyond the simple contrastive loss, we incorporate the InfoNCE [32] loss for an episode with a batch b containing B tasks. The contrastive loss defined as C. log
--eA exp(- -D...) where D o(et, In this case,
treat negative "distance" as "similarity." For the similarity metric in InfoNCE, we experiment with both cosine distance v(x, | and Euclidean distance o(x, ||x yll-
xr
G.2 ICL
We implement ICL w/ ConML with K = 1 and nxx (x "1,".. Rn Un.) [x1 ".. F U1 To obtain the implicit representation (4), we sample from standard normal distribution (the same with æ's distribution) independently each episode. Since the output (4) a scalar, i.e., representation E R, adopt distance measure Φ(a, b) o((a - b) where o(-) is sigmoid function to bound the squared error. À = 0.02. Note that the learning of different functions (LR, DT, SLR, NN) share the same efficient and straightforward settings about ConML above, which shows ConMI. can bring ICL universal improvement with cheap implementation.
We notice that during training of LR and SLR 1 5, which happens to equals to the dimension of the regression task. This means sampling by Tu would results in the minimal sufficient information to learn the task. In this case, minimizing d" is particularly beneficial for the fast-adaptation ability, shown as Figure 3(a) and 3(b). This indicates that introducing prior knowledge to design the hyperparameter settings of ConML could bring more advantage.
31


Reviews:
[{"review_id": "UEKaMIhvFK", "reviewer": "Reviewer_VjCq", "summary": "The authors present ConML, a new way to include a contrastive objective based around model representations as task identity to improve meta-learning methods. Results are obtained across typical few-shot image classification benchmarks (both within and cross-domain) for a number of popular approaches, and a set of synthetic in-context-learning tasks – consistently leading to improvements compared to baseline methods.", "questions": "**[Q1]:** “Inner-task sampling K=1” used for few-shot image classification results; If I understand this correctly, the authors only use *one* sample subset in addition to the original one for the contrastive objective;   \n$\\textrightarrow$  This feels very weak and could easily provide almost no additional information, in case a sample that is close to the class-mean would be chosen (especially because only the training data seems to be used).   \n$\\textrightarrow$  Nevertheless, the results indicate it does have quite a significant effect. $\\textrightarrow$  I’d like to know the authors’ experiences on this, and potential insights for which cases or applications one might need to choose a higher K.\n\n**[Q2]:** How did the authors choose the representation of each ‘model identity’ in the model space (including ‘mapping function’), e.g. the set of updated parameters for optimisation-based methods? Is the ‘essential/influential minimal set’ that explicitly individualises the models always a good starting point (e.g. param for optim-based, steering-vector for hypernetworks, etc.)? Some details, like concatenation of prototypes for metric-based ones, don’t necessarily seem as obvious;   \n$\\textrightarrow$  Some advice/guidance might be interesting to the reader in case they want to extent this approach beyond the methods used in this work.  \n\n**[Q3]:** Less a question but more a weakness: The grammar of the manuscript needs further improvement, and I’d highly recommend the authors check and refine many of the passages. I am aware that this is language-dependent, but given that many tools and/or LLMs are quite well-suited for this task, it shouldn’t be too hard to fix.\n   \nSome examples (not exclusive!):    \n- l.29: “All these works are all under..” -> remove one ‘all’\n- l.30: “the the same ” -> remove one ‘the’\n- l.42f: “from different task even some of their..” -> from different taskS even IF some..\n- l.104: “Details […] is provided” -> ‘are’ provided\n- l.137: ‘tau’ after “task” is missing\n- l.185: “can exactly serves” -> ‘can exactly serve’ \n- and several more… (including many missing articles); \n\n--- \n\n**Further minor comments and suggestions:**  \nSince the experiments are all targeting few-shot applications (especially classification), it could be worth extending the literature review/related works to include other few-shot works that use contrastive objectives (some of which use meta-learning in their fine-tuning stage) -- such as:  \n- Using a contrastive objective to learn class prototypes treated as anchor points to create task-adapted embedding spaces *[Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning, Perera et al., CVPR2024]* \n- Using intra- and inter-task similarity to determine important areas in the exemplar images to aide few-shot classification, optimised via meta-finetuning *[Rethinking Generalisation in Few-Shot Image Classification, Hiller et al., NeurIPS2022]*\n- Demonstrate the use of the contrastive method simCLR to enable discrimination between dataset samples while avoiding negative effects due to the supervision signal for few-shot scenarios *[CrossTransformers: Spatially-Aware Few-Shot Transfer, Doersch et al., NeurIPS 2020]*\n- And likely others..\n\n---\n\n**TL;DR:** I think the paper provides an interesting view on meta-learning that differs from others, and the experimental results provide evidence regarding its effectiveness as well as general applicability -- hence my rating. (I do, however, expect the authors to rework/improve the language/grammar of the manuscript!)", "limitations": "Some limitations are briefly included within (e.g. 4.3.2), but no 'proper' discussion. I'd recommend the authors think about potential limitations their approach could face to provide the reader with some insights into where potential difficulties might arise!", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n**Originality & Significance:**   \n- The motivation of using a contrastive objective is rather straight-forward and hence easy to understand, and the paper makes a clear effort to guide the reader along step-by-step\n- While contrastive objectives have been used frequently in the few-shot domain (including methods that leverage meta-learning approaches, with some mentioned in the references section), the authors propose a different strategy that is novel and seems to consistently improve upon baselines\n\n**Quality:**  \n- Results provided for popular few-shot benchmarks, including both in-domain and cross-domain classification\n- A range of additional results and insight provided in the appendix, e.g. ConMLs impact on different architectures\n\n**Clarity:**  \n- The paper is mostly easy to follow, and the use of algorithmic pseudocode further supports a clear interpretation of the underlying concepts\n\n## Weaknesses \n- The choice of how a ‘model identity’ is represented is somewhat obscure and simply provided as ‘defined’ without much explanation or reasoning/insight – could be improved to provide a basis for readers that might want to transfer the concept to different methods\n- A significant number of typos and grammatical errors that should be corrected", "was_revised": true, "final_justification": "Having re-read parts of the paper, the authors' rebuttal and the other reviewers' questions, I still believe this paper is a valuable addition to the community -- especially since meta-learning hasn't received as much attention lately, and the authors provide an interesting new way of thinking about meta-learning in the context of various different few-shot methods. \n\nI do agree with the concerns of reviewer *Kgc2* in terms of memory increase (which is a concern), and potentially the robustness (although one or very few datasets not showing major improvements is, at least to me, more related to honesty w/o excessive finetuning than to lack in robustness, given that the authors demonstrate their findings on many different datasets). \n\n$\\rightarrow$ However, I do think this works contributes a valuable addition as stated above, and will therefore stick with my initial rating and recommend acceptance.\n\n*P.S.:* Checklist requires major rework, as detailed in the *Paper Formatting Concerns* section.", "rebuttal": "We are very glad, and appreciate it a lot to meet such a detailed and professional review. \n\n## Q1. Inner-task sampling times K=1\nFirst we want to emphasize the meaning of K=1. Though the reviewer might have understood it correctly, the expression in question \"only use one sample subset\" might be a little confusing. It does not mean that the subset contains K-shot per class. It means in each episode, we sample K subsets from, and to contrast with, the complete set ($D_\\tau^{tr}\\cup D_\\tau^{val}$), following certain sampling strategy $\\pi_\\kappa$. With the specification in line 254, in each episode we are contrasting model representation obtained from $D_\\tau^{tr}$ with $D_\\tau^{tr}\\cup D_\\tau^{val}$.\n\nWe acknowledge the concern about the potential weakness of using only one subset (K=1) for the contrastive objective. In our experiments, Our choice of K=1 was motivated by computational efficiency. However, we agree that increasing K could provide more robust supervision. We observed no significant performance gains when increasing K from Table 4. We infer this is because the large meta-training episode (iteration/step) number, generally over $10^4$, with randomly sampled tasks/subsets, mitigates and compensates for the effect of K in each episode. We would need a larger K for meta-training with only fewer episodes.\n\n## Q2. Model identity representation\nThe ‘essential/influential minimal set’(EMS) that explicitly individualises the models is a very nice summarization! \nFor meta-learners with very a large EMS, we can selectively choose a subset.\nFor instance, in large MAML models that update all parameters, limited computational budgets may restrict us to contrasting only the parameters of the last few layers—this approach still achieves comparable performance. \nHowever, using parameters solely from earlier layers fails, likely because the final layers capture more task-specific information.\n\nWe believe that the alignment between the EMS's role in the meta-learner and the chosen distance metric is crucial for success.\nFor example, we can find that euc distance performs much better than cosine in ProtoNet (Fig.3), \nsince ProtoNet classifies using Euclidean distance and ConML contrasts the classifier's weights—capturing the learned representations more accurately. \nAlthough cosine works generally, it would be interesting to tailor distance metrics for various parameter types (e.g., classifiers, MLPs, CNNs, GNNs, Transformers).\n\n## Q3. Grammar and typos\nWe apologize for the grammatical errors and typos. We will thoroughly proofread the manuscript and use LLM-based tools to ensure clarity and correctness in the revised version.\n\n## Q4. Related works\nWe will expand the related works section to include the suggested contrastive learning papers (Perera et al., Hiller et al., Doersch et al.).\n\n## Q5. Proper discussion of limitations\nWe will modify to add a formal \"Limitation\" section for proper discussion, including current limitation mentioned in sec 4.3.2 sec 6.\n\n## Q6. Checklist\nThanks for your kind reminding. We will modify and expand the justifications."}, {"review_id": "y9MKw5XLw2", "reviewer": "Reviewer_euqW", "summary": "This paper focuses on enhancing the alignment and discrimination capabilities of meta learners via a task-level contrastive meta-objective. Unlike traditional contrastive learning in representation space, the proposed method applies contrastive learning directly in the model space, which is a novel and insightful idea. Empirical studies show consistent performance improvements across multiple meta-learning algorithms and settings, which validates the effectiveness of the proposed ConML.", "questions": "- In Table 1, the model representations for optimization-based methods are high-dimensional model weights, which could make similarity calculations computationally intensive. It would be helpful if the authors could comment on the resulting computational overhead and whether it poses practical limitations.\n- Additionally, there seems to be a discrepancy between the description of the meta learners provided in line 198 and the information presented in Table 1. Clarification regarding this inconsistency would improve the clarity of the paper.\n- As described in line 204, metric-based algorithms are naturally suited for classification tasks, yet their performance reported in experiments is lower than other categories. What might explain this gap?", "limitations": "- The paper does not explicitly clarify the relationship between in-context learning and meta-learning, which could be elaborated for better conceptual clarity.\n- More references should be considered:\n\n[1] Robust Fast Adaptation from Adversarially Explicit Task Distribution Generation. KDD2025.\n\n[2] Data augmentation for meta-learning. ICML2021.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- Universal and agnostic to both problem types and meta-learner architectures.\n- Successfully applies to in-context learning, extending its impact beyond conventional meta learning.\n- Provides theoretical guarantees showing tighter generalization error bounds when optimizing the contrastive objective.\n\n**Weaknesses:**\n- Some expressions are overly casual and could be made more precise or formal (e.g., “people also…” in line 27; “obtaining …” in lines 123, 130,139).\n- Lacks evaluation of robustness under noisy or corrupted data, which is important for meta-learning generalization.\n- Episodic learning inherently provides task-identity information. However, since tasks lie in a distribution, when task differences are subtle or overlapping, could ConML degrade performance or overfit?", "was_revised": true, "final_justification": "The responses addressed my concerns, and I will raise my score from 3 to 4.", "rebuttal": "Thanks a lot for your valuable comments and suggestions.  \n\n## Q1. Lack of evaluation of robustness under noisy or corrupted data\nThis is a very good point, as ConML is intended for meta-learning generalization. \n\nWe have provided the effect of ConML under distribution shift in Appendix E.\n\nHere, to further evaluate the effect of task interference, we conduct experiments where a portion of tasks during meta-training was 'poisoned' by randomly disturbing the labels of all samples (support and query) independently. \nAfter meta-training with 10/20%-poisoned task set, we report meta-testing results on 5-way 1-shot MiniImageNet, and calculate relative performance difference (Relative Diff) as (Clean − Poison)/Clean. Results below show that the relative performance decrease is significantly smaller when using ConML. This shows that ConML improves the meta-learning robustness under noisy or corrupted data.\n\nAcc (\\%) | Clean Acc | 10%-poison Acc (Relative Diff.) | 20%-poison Acc (Relative Diff.)\n-|-|-|-\nMAML w/o ConML  | 48.7 |45.5(-6.5%)| 37.8(-22.4%)\nMAML w/ ConML  | 56.2 | 54.7(__-2.6__%)| 51.9(__-7.6__%)\n\n## Q2. ConML's effect when task differences are subtle or overlapping\n\nWe would like to discuss by decomposing ConML into optimizing inner- and inter-task distance.\n\nOptimizing inner-task distance benefits generalize to smaller support-size (as shown in Appendix E), would not likely to be effect by task-distribution.\n\nOptimizing inter-task distance helps generalize to ood tasks (as shown in Appendix E). If the meta-training task differences are subtle or overlapping, generalizing to ood tasks becomes more important. This is because if not testing on ood tasks, we can use supervised learning (maybe a little extreme, but the effect of meta-learning itself degrades).\n\n## Q3.Computational overhead for optimization-based meta-learners.\nIn our experiments,\nwe only use the weights in the last layer (linear classifier) as representation of optimization-based meta-learners.\n\nThis is a very rational concern. We choose model representation by the ‘essential/influential minimal set’ (EMS) of parameters that explicitly individualises the models. And for optimization-based meta-learners, they are inner-updated model weights.\nFor meta-learners with very large EMS, we can selectively choose a subset.\nFor instance, in MAML models that inner-updates all parameters, limited computational budgets may restrict us to contrasting only the parameters of the last few layers—this approach still achieves comparable performance. \nWe also found using parameters solely from earlier layers fails, likely because the final layers capture more task-specific information. We would add illustration in our manuscript.\n\n## Q4.Discrepancy between MAML and other optimization-based meta-learners\nIn Section 3.3, we select one representative method for each category of meta-learners as example to illustrate how to get model representation. The principle of choosing the 'essential/influential minimal set' is shared in the category.\n\nIn line 198, we used MAML as example to show the principle for optimization-based is contrasting inner-updated model weights, which also applies to FoMAML and Reptile in Table 1. We have also provided detail algorithm procedure for incorporating ConML with each specific meta-learner in Appendix C.\n\n\n## Q5. Metric-based algorithms' performance\nWe assume the fact that Metric-based algorithms slightly down-perform optimization-based ones on miniImagenet, is likely due to their reliance on fixed embeddings functions which may not adapt as flexibly. Note that in Table 1, all metric- and optimization-based algorithms use the same Conv4 backbone as feature extractor, which is a relatively small model that would benefit from task-specific adaptation. Optimization-based ones adapt the backbone but metric-based ones fix it inside each task.\n\nIndeed, metric-based naturally suited for classification tasks.\nThis can also be observed by the strong performance of P>M>F in Table 3, which uses a more deep feature extractor and protonet as classifier. Note that it departures from standard protonet by additionally fine-tuning using support set to achieve task-specific adaptation, which might be an evidence of above explanation.\n\n## Q6. Explicitly clarify the relationship between in-context learning and meta-learning.\nThanks for your suggestion. In fact, we have discussed the relation between ICL and meta-learning and reformulate training ICL model as meta-training in Appendix D.\nApart from these, we have also noticed\na recent paper [1] exactly answers this with \"ICL model is meta-learner with minimal inductive bias\", we would add this to our manuscript.\n\n[1]Why In-Context Learning Models are Good Few-Shot Learners? ICLR2025\n\n## Q7. Related works\nWe find the two works provided by the reviewer are highly relevant and very good works. We certainly would add them to references."}, {"review_id": "CigfqIZl3s", "reviewer": "Reviewer_PV71", "summary": "The paper 'Learning to Learn with Contrastive Meta-Objective' introduces a contrastive loss in the model representation space. This loss is used to minimize the intra-task distances while maximize the inter-task distances. The goal of this loss is for better task identification during meta training. The contrastive loss can be straightforwardly applied to different paradigms meta-learning methods including gradient based, metric based, amortized and ICL. Numerical experiments seem to suggest promising improvements over the baseline when combining various meta-learning algorithm with this contrastive loss, with non-significant computation overhead.", "questions": "1. line 198 and line 200: why is $\\theta$ only updated by ONE gradient step? In general, MAML optimize its inner problem by multiple steps sgd. \n2. line 196-line 231: it would add to the clarity if the authors could write clearly whether $\\mathcal{D}$ is the training set or validation set or both. \n3. line 664-line 668 is repeated.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. This paper is well-written and well-organized. Notations and definitions are clear, which makes the paper easy to read. The proposed contrastive loss makes intuitive sense. \n2. The contrastive loss can be directly applied to different types of meta-learning methods including optimization-based, metric-based, amortization-based and ICL, almost in a plug-and-play manner. This loss function is also problem-agnostic. Previous works are either problem-specific or learner-specific. \n3. The authors have performed many numerical experiments with promising results and similar running time as the vanilla methods. Ablation studies on $\\lambda$ and $K$ are also provided. \n\nWeaknesses: \n1. The main contribution of the work is proposing a the contrastive loss $\\mathcal{L}_c$, which in most experiments takes the form $d^{in} - d^{out}$. To better understand this loss, the authors have explored Euclidean distance $\\phi$ and $\\mathcal{L}_c$ in the form of InfoNCE. Based on results from Figure 2, it seems that for a vanilla $\\mathcal{L}_c$, Euclidean distance is more stable than cosine similarity for large $\\lambda$. Could the authors provide more intuition on this phenomenon? \n2. Also based on Figure 2, $\\mathcal{L}_c$ based on InfoNCE consistently beats vanilla $\\mathcal{L}_c$. Can this observation be generalized to other experiments in this work?", "was_revised": true, "final_justification": "All my questions and concerns have been addressed by the authors. I will keep my positive rating.", "rebuttal": "Thanks for you appreciation and valuable comments. We especially appreciate your detailed reading.\n## Q1. Distance metric and contrastive loss\nWe would like to discuss the distance metric and contrastive function respectively.\n\nFor the choice of distance metric, we believe that the matching between the chosen distance metric and model representation is the key to further success. The model representation is a set of model parameters,\nwhose role in the meta-learner need to be considered.\nFor example, we can find that Euc distance performs much better than cosine in ProtoNet (Fig.3), \nsince ProtoNet makes classification with Euclidean distance, and ConML contrasts the classifier's weights — describing the model's behavior more precisely. \nAlthough cosine works generally, it would be interesting to tailor distance metrics for various parameter types (e.g., classifiers, MLPs, CNNs, GNNs, Transformers).\n\nFor the contrastive function,\nInfoNCE generally obtains better performance than the naive $d^{in}-d^{out}$ (Figure 2), which we assume to be generalizable.\nWhile InfoNCE exploits same information as the naive loss does. \nThe performance difference might be explained with the difference of gradient magnitudes, where InfoNCE matches the term of lower bound of mutual information between model representations. \nIn most parts of this paper we use the naive $d^{in}-d^{out}$, to illustrate ConML more straightforwardly, and to decouple $d^{in}$ $d^{out}$ for detail investigation (Appendix E). As we have discussed in the manuscript, we mainly intend to provide the idea and investigate the role of contrastive meta-objective, and there are still many directions to further improve the performance, e.g., contrastive function, distance metric, inner-(subset) and inter-(batch)task sampling strategy.\n\n## Q2. inner-update steps in MAML in line 198 and 200\nWe acknowledge the concern appreciate the reviewer's carefulness. \nIn line 198 and 200, we use $\\theta-\\nabla_\\theta L$ just to show that the function of MAML as a meta-learner, is achieved by gradient descents. We only explicitly write one step for notation simplicity. ConML's framework allows, and our experiments in fact do, multiple steps inner-update for optimization-based meta-learners like MAML. \nSuch simplified notation can be found in a lot of related papers, even in \"Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks\" (MAML) itself (sec 2.2).\n\n## Q3. $D$'s identity in line 196-231\nThanks for your kind suggestion. The $D$ in sec 3.3 can be various. For standard mini-batch episodic training (Algorithm 1), all $D$ in this part is $D^{tr}$. For ConML, considering the Algorithm 2 we introduced in sec 3.1, $D$ can be $D^{tr}$, $D^{tr}\\cup D^{val}$ or $\\kappa_k$.\nSection 3.3 is intended for specifying the general framework of ConML in sec 3.1 to different meta-learners. We hope readers consider this part based on above information, as a modularized part.\n\n## Q4. Appendix line 664-line 668 is repeated.\nWe sincerely apologize for the error and thank you again for your detailed reading. We will modify it in manuscript."}, {"review_id": "nS5Q3C9Yzh", "reviewer": "Reviewer_Kgc2", "summary": "This paper proposes a general meta-learning framework called ConML (Contrastive Meta-Learning), which aims to improve the generalization performance of meta-learners by simulating the \"alignment\" and \"discrimination\" abilities of human learning. The core idea is to introduce an additional \"contrastive meta-objective\" in the training phase of meta-learning. Specifically, ConML performs contrastive learning in the \"model space\": on the one hand, it achieves \"alignment\" by minimizing the distance between models learned from different data subsets under the same task; on the other hand, it achieves \"discrimination\" by maximizing the distance between models learned from different tasks. The author demonstrates that ConML is a problem- and learner-agnostic framework that can be seamlessly integrated into a variety of existing meta-learning algorithms (such as optimization-based, metric-based, and amortization-based methods) as well as the emerging In-Context Learning (ICL).", "questions": "1. Regarding the model representation of ICL (Section 3.3): Can you elaborate on the criteria for choosing the \"virtual input\" $u$? Have you tried different $u$, and how does its choice affect the final performance? Are there any specific design considerations behind keeping $u$ consistent within an episode but sampled independently across episodes?\n\n2. Regarding the theoretical analysis (Section 3.2): The proof outline in Appendix B relies on the assumption that minimizing $\\mathcal{L}_c$ makes the distribution of model representations $P_g(\\theta)$ symmetric. Can you provide more intuition on why this assumption is reasonable for sufficiently expressive models and how it relates to practical implementations of contrastive loss?\n\n3. In cross-domain experiments (Table 3), ConML shows very little to no performance improvement on some datasets (e.g., fo-Proto-MAML on Birds). Is there a potential explanation for why ConML works better in some domains than others? Does it have to do with the diversity or similarity of tasks in these datasets?", "limitations": "Yes. The authors explain the limitations of their study, and their assessment of potential societal impacts meets the standards for basic research of this type.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Extending the idea of ​​contrastive learning from the traditional \"feature space\" to the \"model space\" of meta-learning is a novel and insightful idea.\n2. The highlight of ConML is its versatility. The authors clearly show how to apply the framework to different categories of meta-learning algorithms and provide specific implementation specifications in Section 3.3 and Table 1.\n3. The paper provides a theoretical perspective in Section 3.2, proving that minimizing the contrast loss helps to reduce an upper bound on the generalization error.\n4. The paper is well-written and logically clear.\n\n\n### Weaknesses\n1. The paper claims that ConML only brings a \"small additional computational cost.\" However, as seen in Table 4, after enabling ConML (K=1), the memory usage increases significantly (for example, MAML increases from 1331MB to 2801MB, more than 2 times). Although the relative time increase is not significant (about 1.1 times), the jump in memory is not negligible, especially when the model is larger or the K value is higher.\n2. The model analysis in Figure 2 shows that the performance is quite sensitive to the choice of $\\lambda$, distance function (Euclidean/cosine), and loss form (InfoNCE/naive form). Therefore, the statement that \"not much tuning effort is required\" may be overly optimistic.\n3. The approach to defining a model representation for ICL using a \"dummy input\" $u$. However, its theoretical foundation is somewhat weak. How sensitive is the final performance to the choice of $u$? The paper gives examples of $u$=1 or $u$= \"what is this task?\" but there is no in-depth analysis or justification for this choice. \n4. If there is overlap or even similarity between tasks, it is not reasonable to directly regard them as negative samples. For example, task a (class 1 2 3) and task b (class 1 3 4).\n5. Learning-to-learn is very general. As usual, learning-to-learn methods are also validated on reinforcement learning.", "was_revised": true, "final_justification": "The rebuttal has addressed my question to some extent. However, my rating remains the same primarily due to the concern about memory cost: 4: Borderline accept.", "rebuttal": "Thanks a lot for your valuable comments and suggestions. We find most of them very objective, indicating the reviewer's accurate understanding.\n\n## W1. Memory consumption\nIndeed, ConML introduces additional memory consumption which grows about linearly with K. We want to emphasize that $K=1$ is good enough all our experiment, and the memory consumption (about 2 times) is at the same scale with w/o ConML.\n\n## W2. Hyperparameter sensitivity\nOur statement \"not much tuning effort is required\" is because all main results (Table 2 & 3) are obtained with the shared and primitive hyperparameters (lin 254), not even the best results we have witnessed (only corresponding to the red line in Figure 2). And in a large range of choice of hyperparameter, ConML brings improvement over baseline (gray line in Figure 2).\n\n## W3. $u$ for ICL with ConML\nPlease refer to Q1.\n\n## W4. Task distribution Problem\nwe think it is a very rational point and were taking it as future directions. \nAs we have discussed in the manuscript, we mainly intend to provide the idea and investigate the role of contrastive meta-objective, and there are still many directions to further improve the performance, e.g., contrastive function, distance metric, inner-(subset) and inter-(batch)task sampling strategy.\n\nThe mentioned problem involves the inter-(batch)task sampling strategy we have discussed. Optimizing inter-task distance with the current random task batch sampling can effect sub-optimally in scenarios pointed out by the reviewer. It just works statistically.\nThis might be improved by advanced task batch sampling strategy with task-level diversity/difficulty/similarity awareness and schedule, which have been studied in literature, e.g., [1,2,3].\n\n[1]  Adaptive task sampling for meta-learning. ECCV2020\n\n[2] The effect of diversity in meta-learning. AAAI2023\n\n[3] Towards task sampler learning for meta-learning. IJCV2024\n\n## W5. Broader scope in Meta-learning\nThanks for your kind advice. We will try to apply ConML on other meta-learning scenarios including RL as future work.\n\n## Q1. Choosing the \"virtual input\" $u$ for ICL.\nFirst, we want to illustrate why $u=1$ meets ConML's goal of additional supervision by task identity without labeled data for generalization. \nLet's consider ICL for regression tasks $y=wx$, where the output of Eqn(4) is expected to be $wu$. ConML expect same $w$ can be learned from different subsets of the same task, and distinguishable $w$ can be learned from different tasks. However, $w$ is intractable in ICL models, so ConML use the output of Eqn(4) as an alternative, which is exactly $w$ with $u=1$. Note that above goal only requires $u$ to be constant, where $u=1$ is only an example and choosing $u$ to be any constant is feasible.\n\nNow let's discuss our consideration of keeping $u$ consistent within an episode but sampled independently across episodes, while keeping $u$ consistent globally (e.g. $u=1$) already meets the goal of ConML. Note that in ConML algorithm, the procedure of calculating and optimizing inner- and inter-task distance is complete in each episode, independent to other episodes. So there is no constraint on $u$ to be consistent across episodes. From the principle of \"Occam's Razor\", we set $u$ free across episodes by sampling independently in each episode. From another perspective, technically we can not define a optimal constant as $u$, and keeping $u$ globally consistent might result in model collapsing with large $\\lambda$.\n\nEmpircally, we have evaluated setting $u=1$ under the same metric as Table 5. We provide results in the following table (the relative gain of $u=1$ over independent $u$ across episodes is in bracket). We find that though not significant, such independent design makes general improvement.\n\nFunction | LR | DT\n-|-|-\nRel. Min. Error  | 0.35 (-17%) |0.62 (-23%)\nShot Spare  | -3.86 (-18%) | -3.80 (-10%)\n\n## Q2. Why minimizing $L_c$ makes $P_g$ symmetric\nThanks for your detailed reading! There might be a little misunderstanding. \nTechnically, it is not an \"assumption\" but a solution to an optimization problem, which can be formally described as $argmin_{p(\\tau),p(x|\\tau)} E_{i,j,k \\sim p^3(\\tau), x_\\tau\\sim p(x|\\tau)}[\\frac{x_{i,1}^\\top x_{i,2}}{||x_{i,1}||||x_{i,2}||}-\\frac{x_j^\\top x_k}{||x_j||||x_k||}]$. And it is easy to see the solution is $\\forall p(\\tau), s.t. p(\\tau)=p(-\\tau), p(x|\\tau) =\\delta(\\tau)$. Then in the manuscript $P(X)=p(x|\\tau)p(\\tau)$ is symmetric.\n\n## Q3. Explanation of performance on Meta-Dataset.\nWe understand this concern may rise from similar reasons with weakness 4. We agree with such potential problem as discussed in \"A reply to Weakness\". But we think it does not holds here considering the setting of Meta-Dataset. Following the common setting of Meta-Dataset, we collect all meta-training tasks in ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, VGG Flower datasets for meta-training, then test on meta-testing tasks from both aforementioned datasets (in-domain) and additional Traffic Signs and MS COCO datasets (ood) to perform cross-domain meta-learning. That is to say, we are using the same commonly meta-trained learner when testing on different datasets. So the mentioned reason could not explain well. \n\nWe apologize for currently we could not explain the failure of ConML on fo-Proto-MAML testing on Birds. We will analyze this further to see if this keeps occurring with different hyperparameters and vision models if necessary."}]

Author rebuttal:

Meta-review: This paper proposes ConML, which learns models for a variety of tasks by performing meta-learning based on the idea of contrastive learning. In particular, by using task identity as a new source of supervision and carrying out contrastive learning not in representation space but in model-parameter space, the authors succeed in theoretically deriving an excess risk bound that is independent of the concrete problem setting and the learner of the meta learning. As a result, the theory applies in a unified way not only to common meta-learning paradigms such as optimization-based, metric-based, and amortization-based methods, but also to the so-called in-context learning (ICL). The numerical experiments showed the consistent improvement by the proposed contrastive approach.

While most reviewer concerns were largely resolved through discussion, the robustness of the empirical results and the additional computational cost introduced by the proposed method remained primary points of focus. Regarding robustness, the authors provided additional experiments across various datasets. As for computational cost, a concern remains that memory overhead grows rapidly with the number $K$ of subsampled datasets used to evaluate distances between models within the same task. However, the authors report from experiments that $K=1$ yields sufficiently good results, which serves as a reasonable counterargument.

The idea of performing contrastive learning in model-parameter space is highly interesting and offers a new perspective that bridges contrastive learning and meta-learning. Consequently, the paper provided the universally applicable risk bound and presented results that extend to the increasingly prominent ICL. Considering this as a meaningful contribution to the NeurIPS community, I recommend the paper for an oral presentation.
Decision: Accept (oral)