Title: Learning to Learn with Contrastive Meta-Objective
Venue: NeurIPS 2025
Accepted: yes
Keywords: Meta-Learning

Contrastive Learning
Abstract: Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans.
	Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's fast learning.
	This is achieved by contrasting what meta-learners learn, i.e., model representations.
	The proposed ConML is evaluating and optimizing the contrastive meta-objective under a problem- and learner-agnostic meta-training framework.
	We demonstrate that ConML integrates seamlessly with existing meta-learners, as well as in-context learning models, and brings significant boost in performance with small implementation cost.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a general meta-learning framework called ConML (Contrastive Meta-Learning), which aims to improve the generalization performance of meta-learners by simulating the \"alignment\" and \"discrimination\" abilities of human learning. The core idea is to introduce an additional \"contrastive meta-objective\" in the training phase of meta-learning. Specifically, ConML performs contrastive learning in the \"model space\": on the one hand, it achieves \"alignment\" by minimizing the distance between models learned from different data subsets under the same task; on the other hand, it achieves \"discrimination\" by maximizing the distance between models learned from different tasks. The author demonstrates that ConML is a problem- and learner-agnostic framework that can be seamlessly integrated into a variety of existing meta-learning algorithms (such as optimization-based, metric-based, and amortization-based methods) as well as the emerging In-Context Learning (ICL).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Extending the idea of ​​contrastive learning from the traditional \"feature space\" to the \"model space\" of meta-learning is a novel and insightful idea.\n2. The highlight of ConML is its versatility. The authors clearly show how to apply the framework to different categories of meta-learning algorithms and provide specific implementation specifications in Section 3.3 and Table 1.\n3. The paper provides a theoretical perspective in Section 3.2, proving that minimizing the contrast loss helps to reduce an upper bound on the generalization error.\n4. The paper is well-written and logically clear.\n\n\n### Weaknesses\n1. The paper claims that ConML only brings a \"small additional computational cost.\" However, as seen in Table 4, after enabling ConML (K=1), the memory usage increases significantly (for example, MAML increases from 1331MB to 2801MB, more than 2 times). Although the relative time increase is not significant (about 1.1 times), the jump in memory is not negligible, especially when the model is larger or the K value is higher.\n2. The model analysis in Figure 2 shows that the performance is quite sensitive to the choice of $\\lambda$, distance function (Euclidean/cosine), and loss form (InfoNCE/naive form). Therefore, the statement that \"not much tuning effort is required\" may be overly optimistic.\n3. The approach to defining a model representation for ICL using a \"dummy input\" $u$. However, its theoretical foundation is somewhat weak. How sensitive is the final performance to the choice of $u$? The paper gives examples of $u$=1 or $u$= \"what is this task?\" but there is no in-depth analysis or justification for this choice. \n4. If there is overlap or even similarity between tasks, it is not reasonable to directly regard them as negative samples. For example, task a (class 1 2 3) and task b (class 1 3 4).\n5. Learning-to-learn is very general. As usual, learning-to-learn methods are also validated on reinforcement learning.\n\nQUESTIONS:\n1. Regarding the model representation of ICL (Section 3.3): Can you elaborate on the criteria for choosing the \"virtual input\" $u$? Have you tried different $u$, and how does its choice affect the final performance? Are there any specific design considerations behind keeping $u$ consistent within an episode but sampled independently across episodes?\n\n2. Regarding the theoretical analysis (Section 3.2): The proof outline in Appendix B relies on the assumption that minimizing $\\mathcal{L}_c$ makes the distribution of model representations $P_g(\\theta)$ symmetric. Can you provide more intuition on why this assumption is reasonable for sufficiently expressive models and how it relates to practical implementations of contrastive loss?\n\n3. In cross-domain experiments (Table 3), ConML shows very little to no performance improvement on some datasets (e.g., fo-Proto-MAML on Birds). Is there a potential explanation for why ConML works better in some domains than others? Does it have to do with the diversity or similarity of tasks in these datasets?\n\nLIMITATIONS:\nYes. The authors explain the limitations of their study, and their assessment of potential societal impacts meets the standards for basic research of this type.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper 'Learning to Learn with Contrastive Meta-Objective' introduces a contrastive loss in the model representation space. This loss is used to minimize the intra-task distances while maximize the inter-task distances. The goal of this loss is for better task identification during meta training. The contrastive loss can be straightforwardly applied to different paradigms meta-learning methods including gradient based, metric based, amortized and ICL. Numerical experiments seem to suggest promising improvements over the baseline when combining various meta-learning algorithm with this contrastive loss, with non-significant computation overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This paper is well-written and well-organized. Notations and definitions are clear, which makes the paper easy to read. The proposed contrastive loss makes intuitive sense. \n2. The contrastive loss can be directly applied to different types of meta-learning methods including optimization-based, metric-based, amortization-based and ICL, almost in a plug-and-play manner. This loss function is also problem-agnostic. Previous works are either problem-specific or learner-specific. \n3. The authors have performed many numerical experiments with promising results and similar running time as the vanilla methods. Ablation studies on $\\lambda$ and $K$ are also provided. \n\nWeaknesses: \n1. The main contribution of the work is proposing a the contrastive loss $\\mathcal{L}_c$, which in most experiments takes the form $d^{in} - d^{out}$. To better understand this loss, the authors have explored Euclidean distance $\\phi$ and $\\mathcal{L}_c$ in the form of InfoNCE. Based on results from Figure 2, it seems that for a vanilla $\\mathcal{L}_c$, Euclidean distance is more stable than cosine similarity for large $\\lambda$. Could the authors provide more intuition on this phenomenon? \n2. Also based on Figure 2, $\\mathcal{L}_c$ based on InfoNCE consistently beats vanilla $\\mathcal{L}_c$. Can this observation be generalized to other experiments in this work?\n\nQUESTIONS:\n1. line 198 and line 200: why is $\\theta$ only updated by ONE gradient step? In general, MAML optimize its inner problem by multiple steps sgd. \n2. line 196-line 231: it would add to the clarity if the authors could write clearly whether $\\mathcal{D}$ is the training set or validation set or both. \n3. line 664-line 668 is repeated.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on enhancing the alignment and discrimination capabilities of meta learners via a task-level contrastive meta-objective. Unlike traditional contrastive learning in representation space, the proposed method applies contrastive learning directly in the model space, which is a novel and insightful idea. Empirical studies show consistent performance improvements across multiple meta-learning algorithms and settings, which validates the effectiveness of the proposed ConML.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Universal and agnostic to both problem types and meta-learner architectures.\n- Successfully applies to in-context learning, extending its impact beyond conventional meta learning.\n- Provides theoretical guarantees showing tighter generalization error bounds when optimizing the contrastive objective.\n\n**Weaknesses:**\n- Some expressions are overly casual and could be made more precise or formal (e.g., “people also…” in line 27; “obtaining …” in lines 123, 130,139).\n- Lacks evaluation of robustness under noisy or corrupted data, which is important for meta-learning generalization.\n- Episodic learning inherently provides task-identity information. However, since tasks lie in a distribution, when task differences are subtle or overlapping, could ConML degrade performance or overfit?\n\nQUESTIONS:\n- In Table 1, the model representations for optimization-based methods are high-dimensional model weights, which could make similarity calculations computationally intensive. It would be helpful if the authors could comment on the resulting computational overhead and whether it poses practical limitations.\n- Additionally, there seems to be a discrepancy between the description of the meta learners provided in line 198 and the information presented in Table 1. Clarification regarding this inconsistency would improve the clarity of the paper.\n- As described in line 204, metric-based algorithms are naturally suited for classification tasks, yet their performance reported in experiments is lower than other categories. What might explain this gap?\n\nLIMITATIONS:\n- The paper does not explicitly clarify the relationship between in-context learning and meta-learning, which could be elaborated for better conceptual clarity.\n- More references should be considered:\n\n[1] Robust Fast Adaptation from Adversarially Explicit Task Distribution Generation. KDD2025.\n\n[2] Data augmentation for meta-learning. ICML2021.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present ConML, a new way to include a contrastive objective based around model representations as task identity to improve meta-learning methods. Results are obtained across typical few-shot image classification benchmarks (both within and cross-domain) for a number of popular approaches, and a set of synthetic in-context-learning tasks – consistently leading to improvements compared to baseline methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n**Originality & Significance:**   \n- The motivation of using a contrastive objective is rather straight-forward and hence easy to understand, and the paper makes a clear effort to guide the reader along step-by-step\n- While contrastive objectives have been used frequently in the few-shot domain (including methods that leverage meta-learning approaches, with some mentioned in the references section), the authors propose a different strategy that is novel and seems to consistently improve upon baselines\n\n**Quality:**  \n- Results provided for popular few-shot benchmarks, including both in-domain and cross-domain classification\n- A range of additional results and insight provided in the appendix, e.g. ConMLs impact on different architectures\n\n**Clarity:**  \n- The paper is mostly easy to follow, and the use of algorithmic pseudocode further supports a clear interpretation of the underlying concepts\n\n## Weaknesses \n- The choice of how a ‘model identity’ is represented is somewhat obscure and simply provided as ‘defined’ without much explanation or reasoning/insight – could be improved to provide a basis for readers that might want to transfer the concept to different methods\n- A significant number of typos and grammatical errors that should be corrected\n\nQUESTIONS:\n**[Q1]:** “Inner-task sampling K=1” used for few-shot image classification results; If I understand this correctly, the authors only use *one* sample subset in addition to the original one for the contrastive objective;   \n$\\textrightarrow$  This feels very weak and could easily provide almost no additional information, in case a sample that is close to the class-mean would be chosen (especially because only the training data seems to be used).   \n$\\textrightarrow$  Nevertheless, the results indicate it does have quite a significant effect. $\\textrightarrow$  I’d like to know the authors’ experiences on this, and potential insights for which cases or applications one might need to choose a higher K.\n\n**[Q2]:** How did the authors choose the representation of each ‘model identity’ in the model space (including ‘mapping function’), e.g. the set of updated parameters for optimisation-based methods? Is the ‘essential/influential minimal set’ that explicitly individualises the models always a good starting point (e.g. param for optim-based, steering-vector for hypernetworks, etc.)? Some details, like concatenation of prototypes for metric-based ones, don’t necessarily seem as obvious;   \n$\\textrightarrow$  Some advice/guidance might be interesting to the reader in case they want to extent this approach beyond the methods used in this work.  \n\n**[Q3]:** Less a question but more a weakness: The grammar of the manuscript needs further improvement, and I’d highly recommend the authors check and refine many of the passages. I am aware that this is language-dependent, but given that many tools and/or LLMs are quite well-suited for this task, it shouldn’t be too hard to fix.\n   \nSome examples (not exclusive!):    \n- l.29: “All these works are all under..” -> remove one ‘all’\n- l.30: “the the same ” -> remove one ‘the’\n- l.42f: “from different task even some of their..” -> from different taskS even IF some..\n- l.104: “Details […] is provided” -> ‘are’ provided\n- l.137: ‘tau’ after “task” is missing\n- l.185: “can exactly serves” -> ‘can exactly serve’ \n- and several more… (including many missing articles); \n\n--- \n\n**Further minor comments and suggestions:**  \nSince the experiments are all targeting few-shot applications (especially classification), it could be worth extending the literature review/related works to include other few-shot works that use contrastive objectives (some of which use meta-learning in their fine-tuning stage) -- such as:  \n- Using a contrastive objective to learn class prototypes treated as anchor points to create task-adapted embedding spaces *[Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning, Perera et al., CVPR2024]* \n- Using intra- and inter-task similarity to determine important areas in the exemplar images to aide few-shot classification, optimised via meta-finetuning *[Rethinking Generalisation in Few-Shot Image Classification, Hiller et al., NeurIPS2022]*\n- Demonstrate the use of the contrastive method simCLR to enable discrimination between dataset samples while avoiding negative effects due to the supervision signal for few-shot scenarios *[CrossTransformers: Spatially-Aware Few-Shot Transfer, Doersch et al., NeurIPS 2020]*\n- And likely others..\n\n---\n\n**TL;DR:** I think the paper provides an interesting view on meta-learning that differs from others, and the experimental results provide evidence regarding its effectiveness as well as general applicability -- hence my rating. (I do, however, expect the authors to rework/improve the language/grammar of the manuscript!)\n\nLIMITATIONS:\nSome limitations are briefly included within (e.g. 4.3.2), but no 'proper' discussion. I'd recommend the authors think about potential limitations their approach could face to provide the reader with some insights into where potential difficulties might arise!", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}