Unsupervised Meta-Learning via In-Context Learning
Authors: Anna Vettoruzzo
Lorenzo Braccaioli
Joaquin Vanschoren
Marlena Nowaczyk
Venue: ICLR 2025 Poster (2025)
Keywords: meta-learning
unsupervised learning
in-context learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We introduce a novel unsupervised meta-learning approach that reframes meta-learning as a sequence modeling problem and leverages the generalization abilities of in-context learners.

Abstract: Unsupervised meta-learning aims to learn feature representations from unsupervised datasets that can transfer to downstream tasks with limited labeled data.
In this paper, we propose a novel approach to unsupervised meta-learning that leverages the generalization abilities of in-context learning observed in transformer architectures. Our method reframes meta-learning as a sequence modeling problem, enabling the transformer encoder to learn task context from support images and utilize it to predict query images. 
At the core of our approach lies the creation of diverse tasks generated using a combination of data augmentations and a mixing strategy that challenges the model during training while fostering generalization to unseen tasks at test time. 
Experimental results on benchmark datasets showcase the superiority of our approach over existing unsupervised meta-learning baselines, establishing it as the new state-of-the-art. Remarkably, our method achieves competitive results with supervised and self-supervised approaches, underscoring its efficacy in leveraging generalization over memorization.

Published as a conference paper at ICLR 2025
UNSUPERVISED META-LEARNING VIA IN-CONTEXT LEARNING
Anna Vettoruzzo" Halmstad University, Sweden anna vettoruzzo@hh.ss
Lorenzo Braccaioli*
University of Trento, Italy
lorenzo www.caailcl..................... ieunitn.
Joaquin Vanschoren
Marlena Nowaczyk
Eindhoven University of Technology, Netherlands j. wwwschoren@eue.
Halmstad University, Sweden marlenal maxieenalToooaccryeeem..........
ABSTRACT
Unsupervised meta-learning aims to learn feature representations from unsuper- vised datasets that can transfer to downstream tasks with limited labeled data. In this paper, we propose novel approach to unsupervised meta-learning that lever- ages the generalization abilities of in-context learning observed in transformer ar- chitectures. Our method reframes meta-leariing as sequence modeling problem. enabling the transformer encoder to learn task context from support images and utilize it to predict query images. At the core of our approach lies the creation diverse tasks generated using combination of data augmentations and a mixing strategy that challenges the model during training while fostering generalization to unseen tasks test time. Experimental results benchmark datasets showcase the superiority of our approach over existing unsupervised meta-learning base- lines, establishing it as the new state-of-the aart.. Remarkably, our method achieves competitive results with supervised and self- approaches, underscoring efficacy in leveraging generalization over memorization.
INTRODUCTION
Meta-learning, or learning-to-learn, enables models to accumulate knowledge from multiple tasks, allowing rapid adaptation and generalization to new tasks (Vettoruzzo et al.) [024 Vanschoren 2019). Traditional meta-learning approaches typically rely on labeled data to construct tasks during meta-training. However, collecting large labeled datasets in real-world applications is challenging and often impractical. Unsupervised meta-learning (UML) methods address this issue by leveraging unlabeled data to learn transferable feature representations, enabling adaptation to new tasks with limited labeled data (Vettoruzzo et al. 2024).
Various approaches have been proposed to address the UML problem Hssu et al. 2018 Jang et al, 2022 Khodadadeh et al. 2019 Kong et al. 2021 Lee al. 2022 2020). However, UML still faces several challenges. Existing UML methods often rely on simple data augmentations to construct the training tasks, while following the standard meta-learning task sampling pipeline for evaluation. This results in a significant difference between training and testing tasks, limiting generalization and often requiring fine-tuning on the test domain. Furthermore, existing UML approaches typically assume that the training and test datasets belong to the same domain. In our framework, we loosen this assumption resulting in a more challenging setting that necessitates better model generalization compared to usual meta-learming applications. We refer to this as the cross cross-domain scenario.
In this paper, we propose novel approach to UML that addresses these challenges by leveraging in-context learning within a transformer architecture (Dong et al. 2022 Min et al.) (022). In- context learning allows the model to use the context provided by a sequence of input-output pairs to make predictions new input data. Inspired by recent advancements in large language models (LLMs) Wei et al. 2022 Brown et al. 2020
Liu et al. 2022), we formulate meta-learning as
"Equal contributions.

---

Published as a conference paper at ICLR 2025
Fixed feature extractor f
Unear 5) (n
Support set
Querr set
Transformer encoder Me
augmmnt
mbdup
cample es
Azy 2)tiss
f(c (..))))) ((so ((C(" (+e (1. (
Drrats
[xi
Learned class encoder
Figure Visualization of CAMeLU (with 3-way 5-shot tasks). The left side illustrates the task creation mechanism, where N samples are drawn from an unlabeled dataset D Dtrain- Each sample
In augmented K times obtain (sp) " strategy inspired by mixup (Zang al. 2018) 8) is utilized for generating the query set by using an augmented version of nn, i.e., In.j. The same pseudo-label n E [1, N is assigned to all data generated from the sample In. On the right side, the so-created task is fed into the transformer encoder for predicting the query input. Inspired by CAML (Fifty et al. 2024), the transformer encoder processes demonstrations created by concatenating features from fixed pre-trained feature extractor and a learned class encoder. The symbol denotes the unknown query label that the transformer encoder aims to predict.
sequence modeling problem, where task is seen as a non-causal sequence of support images and an unknown query image. The support set is treated as the context utilized by the model to predict the class of the query image. We call our approach CAMeLU, which stands for Context-A Awre Meta- Learning in Unsupervised scenarios. Central to our approach is a novel task creation mechanism that enables the generation of a large number of different tasks from unlabeled dataset. Drawing inspiration from the natural decision process of learning by analogy (Winston 1980). we construct tasks that closely resemble the structure of those encountered during inference. Specifically, we use combination of different data augmentation techniques based on basic image manipulations (Shorten Khoshgoftaar, 2019) for generating the samples the support set. Conversely, a strategy similar to mixup (hang et al. 2018) is employed to generate query images by combining a support element, after applying a distinct augmentation function, and an image randomly sampled from the training dataset. This process ensures that the query contains sufficient information from the support image to be classified as the latter, while introducing diversity by blending them. Consequently, query images appear distinct from their corresponding support images while still belonging to the same class, better mimicking the tasks seen at test time and hence enhancing generalization. Following task creation, support and query images encoded using fixed pre-trane pre-trained feature extractor. The resulting latent representations are aggregated into a sequence and passed as input to a transformer encoder along with their label encodings. The transformer encoder learns extract contextual information from support images and predict the query image in a single pass, eliminating the need for the fine-tuning step during inference. An overview of our approach is visualized in Fig. [
Throughout extensive experiments we demonstrate the effectiveness of the proposed approach to generalize to new tasks in real-time. Particularly, CAMeLU outperforms other UML baselines across several datasets, establishing itself as the state-of-the-art in the field. It also achieves com- parable results to its supervised counterpart and to SSI. approaches. While the latter requires fine- tuning on the test domain, CAMeLU obtains comparable performance with a single forward step, highlighting its applicability to real-time applications. Furthermore, by recasting the meta-learning phase as in-context learning within transformer architecture, we improve efficiency, ensuring the whole training and inference phase can be executed with a consumer device with 8GB VRAM.
The main contributions of this paper are follows:
We introduce CAMeLU, novel UML method that leverages in-context learning within transformer architecture, reframing meta-leaming as a sequence modeling problem.
We propose a novel task creation mechanism that generates diverse few-shot tasks from unlabeled datasets using a combination of data augmentations and mixing strategy. This
2

---

Published as a conference paper at ICLR 2025
ensures better alignment between training and testing tasks, thus improving generalization performance.
We demonstrate that CAMcLU outperforms existing UML baselines across five datasets, without the need for fine-tuning to the test domains.
We investigate the ability of CAMeLUU to generalize across various datasets, including those significantly different from the training data.
2 RELATED WORK
Unsupervised meta-learning. Meta-learning is a well-studied field in the machine learning com- munity due to its ability to enable models to quickly adapt to tasks with limited labeled data. Pi- oneering work in the field (Finn al. 2017 Snell al. 2017 Vinyals et al.) 2016 Mishra al.) 2018 Sung et al. 2018) considers the scenarios where large labeled dataset is available for meta- training. a challenging requirement in real-world applications. UML addresses this challenge by extracting meaningful information from unsupervised data that can be transferred to downstream tasks with limited labeled data. Different techniques have been explored in the literature to construct diverse tasks. CACTUs (Hsu et al.) 2018) applies clustering in the embedding space and assigns the same pseudo-label to all images in the same cluster. Other methods focus on generating synthetic samples, either using data augmentations, as in UMTRA (Khodadadeh et al. 2019), or leveraging interpolation in the latent space of a generative model (Khodadadeh et al. | 2020). Differently, Meta- GMVAE (Lee et al. 2020) and Meta-SVEBM (Kong et al. 2021) use variational autoencoders and memory-based models for pseudo-label generation. Recent methodologies have also incorporated SSL techniques (Doersch et al. 2015) into UML methods. In particular, Set-SimCI.R (Lee al.) 2022) builds on top of the SimCLR (Chen et al. 2020) approach and reframes meta-learning as a set-level problem, while PsCo (Jang et al. 2022). inspired by MoCo (He et al.) 2020), utilizes a mo- mentum encoder and a queue of previous samples to improve pseudo-labeling and construct diverse tasks for UML applications. Similarly, BECLR (Poulakakis-Dakiz Daktylii Jamali- Ranali-Rad] 2024) intro- duces an approach for unsupervised few-shot learning by proposing a constrastive representation learning framework, instead of meta-learning.
Data augmentation. Several UML approaches rely on data augmentation to construct the training tasks (Khodadadeh et al. 2019 Lee et al. 2022] Jang et al., 2022). However, traditional transforma- tions such as rotation, translation, cropping, resizing. and flipping (Shorten & Khoshgoftaar 2019) might generate images that are too similar to the original ones, ending up in tasks with low in-class variability between the support and query images. This creates a problem when the model needs to generalize to test tasks, where the guery data are different instances than the support ones, not only augmented versions of them. In this paper, we addressed this limitation by generating query images using a strategy inspired by mixup (Zhang et al., 2018) to enhance model generalization. Similarly to mixup, which performs linear interpolation of the feature vectors at the pixel level, other strate- gies based mixing images comprise CutMix ( (Yun et al. 2019), PatchMix (Liu et al. 2021). and Manifold Mixup (Verma et al. 2019).
In-context learning. In-context learning refers to the ability to perform a new task via inference alone by conditioning on few input-output pairs and making predictions for new inputs (Dong
et al. 2022). Although typical of LLMs (Devlin et al.) 2019 Radford et al.] 2019 Touvron et al.
2023 this ability has also been explored in different fields, such in-painting (Bar et al. 2022
Zhang et al. 2024), image segmentation (Butoi et al. 2023). and notably meta-learning (Chan et al.
2022 Singh et al. 2024; Kirsch al. 2022 Fifty et 2023 2024 Min et al. 2022). Recent methods, such as Chan et al. (2022) and Singh et al. (2024), examine the emergence of in-context
learning abilities from a data distribution perspective, extending these insights to images. GPICL
(Kirsch et al.] (022)) further demonstrates that transformers can be meta-trained as general-purpose in-context learners, while CAML (Fifty et al., 2024) adapts this concept to non-causal sequence modeling problems. Building on these advancements, ling the unsupervised meta-learning problem. Specifically, we introduce a novel task creation mech- anism that, together with an in-context learner, enables learning directly from an unlabeled dataset. This approach differentiates our method from prior in-context learning techniques, aligning it with the unique requirements of UML.
work takes a different direction by tack-
3

---

Published as a conference paper at ICLR 2025
3 PROPOSED APPROACH
Our proposed approach, Context-Aware Meta-Learning in Unsupervised scenarios (CAMeLU), leverages the in-context learning ability of transformers address the challenges of UML. These challenges include the need to construct meaningful tasks from unlabeled data and the requirement for models to generalize effectively to new tasks during inference. CAMeLU consists of two phases that are intertwined during the model training. Initially, tasks are automatically constructed from an unlabeled dataset utilizing a combination of two strategies. Subsequently, we reformulate the meta-learning framework sequence modeling problem, aiming to harness the in-context learn- ing capability of a transformer. This enables the model to extract context from the support sam- ples and predict the unknown query samples without requiring any fine-tuning during the inference phase. The combination of these two phases is essential and guarantees good generalization per- formance without labeled information. Transformers excel at modeling dependencies and capturing relationships between support and query samples, which is particularly beneficial in few-shot learn- ing scenarios. The novel task creation mechanism complements this by constructing diverse and challenging pseudo-tasks, effectively preparing the model for the complexities of target tasks. We delve into the two phases in Sect. 3.1 and Sect. 3.2 respectively.
3.1 TASK CREATION
Central to our proposed approach is the task creation mechanism. In meta-learning, task Ti corre- sponds to data generating distribution T, ((()))) and consists of data from N distinet classes. The data sampled from each task is divided into a support set, D( containing K train- ing examples per class, and a query set, D( At meta-test time, only the support set (sp) of task Tnew ~ Dreat is labeled and used to fine-tune the model and make accurate predictions on the unlabeled query set. Contrary supervised meta-learning, tasks in UML are only available at test time, while a large unlabeled dataset Dirain is available during training. The main goal is to extract prior knowledge from this unlabeled dataset that can be generalized to a target task, Tnow Drest, during inference. A critical aspect of UMI. approaches lies in the task creation mechanism to create tasks from Dtrain. which must ensure that the constructed training tasks reflect the structure of those encountered during testing, thereby facilitating effective generalization to novel tasks at test time. To do so, we employ two distinct strategies for constructing the support and query sets of each task.
For the support set, we randomly sample N images from Dtrain under the assumption that they belong distinct categories, as shown Fig. []] This assumption reasonable when N << C, where C' denotes the total number of classes in Dirain, which is satisfied using a large training dataset. If we assume that all samples are equally distributed among the classes, i.e., m samples per class, the probability that two or more samples are in the same class is equal to
()) P 1 (C.m) ((C- 1) ..C (CC- N N =1 C.. C..m........... (C (C-m- - N)!
NN- (C m))!
For example, the probability for a 5- classification on the ImageNet-964 dataset used in our experiment is around 0.01, which is negligible. To emulate the K-shot scenario typical of meta- learning tasks, we augment each of the N images K times, with an augmentation function A sampled from a predefined set transformations A, and we assign the same pseudo-label n E [1, N] to all data generated from the same sample In- Specifically, for each image Fn. K augmentation
functions are applied obtain (.p) A (z,,) with As. A and h = 1...., K. One requirement of A& is that the function must preserve class membership. i.c., In E c Ax (z,) E c, for c E C.
7.,
Although this property cannot be directly verified due to the lack of class information in the training
set, it is reasonable to assume that it holds by selecting transformations that minimally alter the
image content.
For the query set, we employ a different approach. We demonstrate in Appendix A.5 that simply applying data augmentations sampled from A is not sufficient for creating query set resembling those in test tasks. At test time, the query set samples are different instances belonging to the same N classes encountered in the support set, not augmented versions of the support samples. However, since Derain is unlabeled, we need a strategy to create new samples with the same implicit classes as those in the support set. For each query image ..(or) that want to generate, randomly select
4

---

Published as a conference paper at ICLR 2025
an image 2n from the ones sampled for the support set generation and apply augmentation function Aj ~ A. possibly different from the one used for the support generation. We then propose strategy inspired by mixup, where combined the augmented image Fn.j" = Az( and an image sampled from Dtrain according to:
2//T)) Az (1 A)2n..
where À ~ Beta(a, B) with a == 1. = and À E (0, 0.5), and 2 ((r)) is assigned the same label as the support samples generated from T'p- By merging a small proportion of new image 2j into Tn.j. we enhance diversity in the query set with respect to the images in the support set. This strategy enforces the model to extract robust features and effectively generalize to scenarios where query images differ from the support samples, as commonly encountered at test time.
This task-creation mechanism can be seen as a task augmentation strategy (Yao et al. 2021 Rajen- dran et al. 2020) that allows the generation of large number (almost infinite) of diverse tasks. This is particularly useful for meta-lcarning and in-context learning applications where the model needs to acquire knowledge from multitude of tasks to generalize to unseen tasks sampled from different domains.
Differences with mixup. While the strategy used for generating the query images draws inspira- tion from the mixup strategy proposed in ZZang et al. (2018), there are some substantial differences. develop new data augmentation strategy to expand the number of training
The aim of mixup examples and diversify the data distribution used for training, thereby enhancing the robustness and generalization of neural networks. In CAMeLU, the primary objective of merging images is to en- courage the model to learn even in scenarios where only fraction of the class context is present in the image. In CAMeLU, À is sampled from uniform distribution (obtained with a Beta distri- bution with embedded into ) is less than 50%, thus ensuring that the assigned label is consistent with the class of the support images generated from IG Indeed, we assign the same label n to -r(T² forcing the network to learn to retrieve information in x; ily, mixup creates new examples by interpolating both images and labels at the scope of limiting memorization over the training distribution.
1, 1) in (0, 0.5), guarantecing that the amount of information from 2j that is
((rr that is related to the category of æn. Contrar-
3.2 IN-CONTEXT LEARNING METHOD
Following task creation, we rephrase the meta-lcarning framework non-causal sequence mod- eling problem, where the order of the examples does not entail a causal relationship. Inspired by recent developments in LLMs (Garg al., 2022 Li et al. [023] Devlin et al., 2019 Radford al.) 2019 Touvron et al. 2023), we treat each task as prompt, where the support embeddings, together with the learned projected labels, form the demonstration context, whereas the query represents the classification problem that the network is required to solve. A model is said to in-context learn task
it can approximate g!(r) for a new query input ((((r) by conditioning on a sequence SJ,j containing in-context (support) examples and one query input defined as follows:
S. = (11 (+)))) 91 (sP)), )...., (#NK VNK),
(sp) (ep), (gr))
j j= i.
with Q the number of query samples to classify and NK the total number of context (support) samples. Formally, Mo can in-context learn task T, if it can predict (ar) with an average error
E
(MMo(S
<E
(3)
(9r) in T,, and (gr) E N1. where é the loss function, S, is the sequence associated to "
To achieve this, we design a model comprising three components: (1) a feature extractor fys (2) class encoder 9s. and (3) transformer encoder with linear projection layer top, i.e.,
Mg. The feature extractor aims to map support and query samples into a latent space where im- ages with similar characteristies and semantic meaning assigned similar representations. In
5

---

Published as a conference paper at ICLR 2025
150
Class
3b
Cass
Class
Clsss
100
Class Class 5
Clss
14
0
-50
45
100
-100
-50
100
150
3.0
35
4.0
4.5
5.0
6.0
6.5
Feature extractor
(h) Transformer encoder
Figure 2: Visualization of clustered embeddings obtained with CAMcLU after the feature extractor (left) and the transformer encoder (right) on 5-way 5-shot task sampled from the CUB dataset. Crosses indicate the centroids of each class, and the numbers denote the Euclidean distances between the query (triangle) and each class centroid. The plots are obtained using t-SNE (Van der Maaten & Hinton 2008) with perplexity equal to 9.
Appendix A.4 we explore various feature extractors for this purpose, including those pre-trained via supervised approach leveraging an SSL technique. The resulting representations are then concatenated with class embedding. The class embeddings for the support representa- tions are generated by encoding the corresponding classes using the class encoder Jø- However, as the classes of the queries are unknown, a randomly initialized learnable vector is appended to each query representation. The so-combined embeddings are then organized into sequences (f )
1...., Q, resembling the one in Eq. 2 These sequences are fed into the transformer encoder, and only the transformer output corresponding to the query sample is selected and passed through a projection layer predict the query label. This process iterates for all queries in the task, and the aggregated loss is employed for model training. In particular, the training process can be formulated an optimization process where the objective is as follows:
S = (ff((z(()) (,o(v(")))))((("))))))
fo(a//r))
min Es.
((Mo
0,6
with S; S, = (S denoting the set of sequences associated to each task T, generated from D Dtrain and f is the cross-entropy loss function.
During evaluation, when a new task is presented, the available examples in D(sp) are utilized as contextual information to guide the classification of the query samples without requiring any fine- tuning or adaptation steps.
Analysis. To gain a better understanding of how the in-context learner functions during inference, we show the embedding space of an exemplary test task after the feature extractor and the trans- former encoder Fig. 2 The embedding space after the feature extractor appears sparse, with large Euclidean distance between the query sample and the centroid of each class, which indicates limited class separability and less informative representations. significantly improves the representation, producing more compact and well-separated clusters. No- tably, the query representation aligns closely with the support examples the same class, show- casing the effectiveness of the transformer in utilizing the task context to refine predictions. This demonstrates the in-context learner's 's ability to adapt representations dynamically based on the task context and also provides evidence supporting CAMeLU's superior performance, particularly in cross-domain and few-shot scenarios where generalization is more challenging. Experiments for the other datasets can be found in Appendix A.2
contrast, the transformer encoder

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of CAMeLU across different datasets, and we com- pare the results with several baseline methods. In particular, we provide a quantitative comparison with UML baselines in Sect. 4.3 and we highlight the ability of CAMeLU to leverage generalization
over memorization in Sect. 4.4 We then present the results with small-scale training dataset in Sect. 4.5 and a comparison with SSL methods in Sect. 4.6
4.1 DATASETS AND BASELINES
For the evaluation, use two generic object recognition datasets, i.c., minilmageNeet (Ravi & Larochelle 2016) and CIFAR-fs (Bertinetto et al.) 2019), and three fine-grained image classification datasets, T.e., CUB (Wah et al. 2011), Aireraft (Maji al., (013), and Meta-iNat (Wertheimer & Hariharan 2019). While minilmageNet and CIFAR-Is share some classes with ImageNet- CUB,Aireree and Meta-iNat focus on more specialized domains, ensuring rigorous cross-domain evaluation. Each dataset is split into training. validation, and test sets following the splits in Ravi & LLrocchlle| (2016) and Bertinetto et al. (2019) for minilmageNet and CIFAR-fs, respectively, and in Triantafillou et al. (2199) and Poulakakis Poulakakis-Daktylide & Jamali-Ra Rad (2024) for the remaining datasets. All labels are removed from the datasets during the training phase.
We compare CAMeLU with standard UML approaches such as CACTUs (Hsu et al. 2018), UM- TRA (Khodadadeh et al., 2019), Meta-GMVAE (Lee et al., 2020), and PsCo (Jang et al.) 2022). These methods are evaluated in-domain as recommended in the original papers, with training and testing performed on the same dataset. While this setup is relatively simpler than the cross-domain evaluation employed for CAMeLU, applying these methods in cross-domain scenario may not be fair, as they were not explicitly designed for such a challenging scenario. Only PsCo (Jang et al. 2022) is further evaluated in cross-domate setting, as the authors demonstrate its adaptability to this scenario through an additional adaptation phase to the test domain. We also compare CAMelUU with BECLR (Poulakakis- Konlakaaki-Dakkyliidh Daktylidis & Jada 2024) in Sect. 4.5 contrastive framework
for unsupervised few-shot learning, and CAML (Fifty et al. 2024)) a supervised meta-learning ap- proach that assumes tasks are available both during the training and testing phases and leverages the in-context ability of transformer architectures to generalize to new tasks. Due to their similarities, we refer to CAML as the supervised counterpart of CAMeLU. Furthermore, Sect. 4.6 provides comparative analysis with two fine-tuned state-of of-the-art self supervisdd trained networks, namely SimCLR (Chen al. 2020) and SwAV (Caron et al. 2020).
4.2 TRAINING DETAILS
We report the results following the N-way K-shot classification task typical of meta-learning algo- rithms, where N = and K or K 5. All models are trained for 100 epochs with 500 episodes per epoch. Fine-tuning at test time (100 steps) is applied only if required. For CAMeLUU, we do not apply any fine-tuning step to demonstrate the strength of its training stage, which does not require additional parameter updates during inference. Furthermore, of ImageNet-1& k (Deng et al. 2009) where classes from the validation and test splits of minilmageNet removed to prevent data leakage
introduce ImageNet-964, a variant
problem that is not taken into consideration by previous stud- ies (Fifty et al. 2024 Jang et al. 2022). To provide a fair comparison, all cross-domain methods are trained on ImageNer-664, For CAMeLU, we use a ResNet-50 (He et al. (016)) feature extractor pre- trained on TeageNet-964 and a class encoder that maps one-hot label vectors to a 256-dimensional space. In Appendix A.4 we also report the results with different feature extractors. The transformer encoder consists of 8 layers, each with an eight-head self-attention block, an MLP, and a single pro- jection layer that maps the transformer output to the predicted category. The model is trained with the Adam optimizer with a learning rate of 10-5 and a warmup cosine scheduler (Vaswani et al.) 2017). To account for statistical variations, each algorithm is run three times in full, and the com- plete results reporting the standard deviations are presented in Appendix A. The experiments are executed using Python and the Py Torch library on an Nvidia GeForce RTX 3070 Ti Laptop GPU with 8GB of VRAM, while ablation studies and competitors are executed on an Nvidia A100-SXME GPU with 40GB of VRAM. More details about the training settings can be found in Appendix A.I and the code is available at https: /ttps:///ttthtt httgggttuuu..ooo wtt.:/ooo/bbaaa.e 95/ C5..AMM....g....

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison on minllmageNet, CIFAR-fs, CUB, Aireraft, and Meta-iNat datasets for 5-way 1-shot and 5-way 5-shot scenarios. Cross-domain approaches are trained using ImageNet-964 and ResNet-50 feature extractor. The symbol f indicates results that are affected by data leakage. The bold font highlights the best performing UML approach for each setting. Re- sults show the average across three complete runs of the algorithms. Complete results with standard deviations are reported in Tab. 12 in Appendix A.10
mindImageNet
CIFAR-Is
CUB
Aircraft
Meta-iNat
Method
5w1s
5w5s
5wls
5w5s
5w1s
5w5s
5w1s
5w5s
5w1s
5w5s
In-Domain
CACTU--MAMI
43.30
54.21
42.00
56.64
31.19
36.81
24.06
27.26
20.13
21 ..84
BACTUS-ProtoNNC
48.85
62.52
50.90
64.52
33.93
44.41
26.27
30.88
27.30
29.08
UMTRA
39.93
50.73
32.93
46.13
27.06
36.6
22.40
31.73
28.96
37.12
Meta-GMVAE
55.381
65.101
52.02
64.18
33.59
39.09
24.83
27.
34.22
40.23
PsCo
47.29
64.85
42.21
62.92
33.09
51.02
26.19
38.80
36.97
55.88
Cross-Domain
PsCo
67.89
90.17
53.34
76.22
43.35
70.19
29.87
38.20
46.21
70.05
CAMeLU
76.51
92.14
61.79
80.43
65.52
80.35
33.17
39.3
57.27
75.45
CAML (supervised)
81.75
92.31
59.44
75.27
54.63
66.81
28.92
32.06
50.86
67.07
4.3
COMPARATIVE RESULTS
Table D provides an overview of the experimental results for both the 5-way 1-shot and the 5-way 5-shot scenarios. The results demonstrate that CAMeLU outperforms the existing UML methods, regardless of the difference the evaluation setting. As highlighted Sect. 4.1 CACTUs, UM- TRA, and Meta-GMMAE are evaluated only in-domain, requiring knowledge about the test domain prior to training. This is not necessary for CAMeLU as demonstrates high performance in the challenging cross-domain scenario. Even compared PsCo, the only UMI. method designed for cross-domain applications, CAMeLU exhibits performance improvement across all datasets. Fur- thermore, PsCo requires a fine-tuning phase adapt the test domain, whereas CAMeLU achieves good performance with a single forward pass, enhancing its applicability to real-time applications. also worth noting that CAMeLU achieves comparable performance its supervised counter- part, CAML, when evaluated on minilmageNet and it even outperforms CAML when evaluated on more dissimilar domains, such CUB, Aireraft, and Meta-iNat. This finding highlights the efficacy of the task construction strategy used in CAMeLU, which acts as sort of task augmentation and enhances the generalization capability of the model.
4.4 MEMORIZATION TO GENERALIZATION PHASE SHIFT
During the training of CAMeLUU, we observed a distinct trend in the validation accuracy, similar to the findings in Kirsch et al. (002)). Fig. 3 illlus trates this pattern, showing the validation accu- racy relative to its initial value, or, in other words, how much the model learns from datasets differ- ent from the one we are training on. Specifically, the curves in Fig. 3 resemble logistic curve,
Memorizaion
Leam
Generalization
Ralanto val danian accunnyy
0.4
minimagehut
which can be divided into three phases that we denote memorization, learning, and general- ization. In the memorization phase, the model memorizes the tasks seen during training and ex- tends this knowledge to unseen tasks, resulting in a slight improvement for datasets with high sim- ilarity with ImageNet-964 (e.g., minilmageNet and CIFAR-fs). For the other datasets, instead, transferring this knowledge can result in performance decrease due to the intrinsic domain distance of the dataset (see CUB and Aireraft,
0.0
Figure 3: Analysis of learning behavior when transferring knowledge from different prior dataset. The relative validation accuracy shows the difference between the current and first epoch accuracy on the validation set of miniImageNet, CIFAR-fs, CUB, and Aircraft. CAMeLU is trained with ImageNet-964.
8

---

Published as a conference paper at ICLR 2025
Table 2: Accuracy results obtained training PsCo, BECLR, and CAM with a small-scale dataset, namely minilmageNet, denoted as (mini) in the table. Results show both in-domain performance (on the test set of minilmageNet) and cross-domain performance on CIFAR-fs, CUB, Aircraft, and Meta-iNat. The average results across three complete runs of the algorithms are reported. Complete results with standard deviations are presented in Tab. 13 in Appendix A.10
minilmageNet
CIFAR-Fs
CUB
Aircraft
Meta-iNat
Sw1s
5w5s
Swis
Sw5s
5wls
5w5s
5w1s
5w5s
5w1s
5w5s
PsCo (mini)
47.29
64.85
42.21
62.92
33.09
51.02
26.19
38.80
36.97
55.88
BECLR (mini)
81.04
87.88
57.05
72.82
42.47
58.03
27.48
38.
19.87
65.05
CAMclUU (mini)
75.99
90.38
61.25
78.79
60.60
74.77
31.39
36.52
55.60
72.12
which are fine-grained datasets). As training progresses and the model observes more tasks, the learning phase occurs. This phase is characterized by a transition to the learning-to-lean state where the model learns to identify the tasks and extract the features that are more useful for solving them. The duration of this phase varies, with datasets like minilmageNet and CIFAR-fs exhibiting rapid learning within approximately 10 epochs, while datasets such as CUB and Aireraft may necessitate up to 40 epochs. This timespan depends on several factors, including the similarity between the training and evaluation datasets, the size of the test dataset, and the model's 's learning ability (Kirsch et al. 2022 Power et al.) (022). For instance, CUB, with its fine- fine-grained nature and small test set size (around 1770 images), necessitates a longer learning phase compared to the minilmageNet dataset (which has test set with 12 000 images). Subsequently, in the generalization phase, the model can generalize to tasks significantly different from those observed during training using a single forward pass. Further analyses about the generalization capabilities of CAMeLU and the number of epochs required for reaching the generalization phase are presented in Sect. 4.3 and Appendix A.8
4.5 GENERALIZATION ON SMALL-SCALE DATASETS
While most studies on training transformer architectures focus on large-scale training datasets, we investigate the generalization capabilities of CAMeLU using a small-scale training dataset. Specifi- cally, we train CAMeLU on minilmageNet and evaluate its performance both in in-domain (i.e.. on the test set of minilmageNet) and cross-domain on CIFAR-fs, CUB, Aireraft, and Meta-iNat. CAMeLU demonstrates effective generalization in this scenario, showing impressive performance in both in- domain and cross-domain settings, as shown in Tab. 2 surpassing PsCo and BECLR by a large margin.
Additionally, a comparison of CAMeLU's performance when trained on small-scale dataset like minilmageNet (Tab. on ImageNet-964 (Tab. M. and on a large-scale dataset (Tab. Appendix
A.3) show that our method is only slightly af-
fected by the size of the training dataset. This
robustness enhances CAMeLU's applicability scenarios where only a small unlabeled training dataset is available, which is common in real- world applications.
vattee acorary
0.5
0.3
CAMNUU
Finally, Fig. 4 shows the relative validation accu- racy of CAMeLU and CAML while trained and evaluated on minilmageNet. While the curve ob- tained with CAMeLU reflects the three phases of memorization, learning, and generalization dis- cussed in Sect. 4.4 the relative validation ac- curacy of CAML remains flat. This difference may be attributed to the task creation mechanism of CAMeLU, which acts as a task augmentation strategy, increasing the variability of tasks pre- sented to the model during training and thereby enhancing its generalization capabilities.
8.1
0.0
10
20
40
60
80
300
Epacts
Figure 4: Relative validation accuracy of CAMeLU (orange) and CAML (blue) when evaluated in in-domain puted as in Sect. 4.4] The curve obtained with CAMeLU reflects the three phases of mem- orization, learning, and generalization even when using small-scale dataset.
minilmagenet and com-

---

Published as a conference paper at ICLR 2025
Table 3: Comparison between CAMcLU and SSL approaches for the 5-way 1-shot and 5-way 5- shot scenario on minilmageNet, CIFAR-fs, CUB, Aircraft, and Meta-iNat. The symbol i indicates results that are affected by data leakage. Results show the average across three complete runs of the algorithms. Complete results with standard deviations are reported in Tab. 14 in Appendix A.10
minilmageNet
CIFAR-I'
CUB
Aircraft
Meta-iNat
Method
5w1s
5w5s
5w1s
Sw5s
5w1s
5w5s
5wls
5w5s
5wls
5w5s
SimCLR
83.32
94.86'
64.52
84.36
47.35
66.87
29.36
39.99
52.44
73.19
SWAV
74.83'
94.96*
66.97
87.14
47.84
69.31
30.33
47.43
53.57
74.53
CAMeLU
76.51
92.14
61.79
80.43
65.52
80.35
33.17
39.11
57.27
75.45
4.6
COMPARISON WITH SSL METHODS
In this section, we compare CAMeLU with SimCLR (Chen et al. 2020) and SwAV (Caron et al, 2020). For training SSL methods in experiments, employed a backbone network with ResNet-55 architecture pre-trained on ImageNet- and obtained from Py PyTorch Lightningg Bolts (Borovec al.] (022). While this setup leads to data leakage when evaluated on minilmageNet. due to overlap between the test and training classes, pre-training these SSL approaches from scratch using a different training dataset was beyond our available computational resources. To facilitate model adaptation to the test domain, we fine-tuned a classification layer on top of the pre-trained backbone using SGD with an initial learning rate of 0.01, momentum of 0.9, weight decay of 10-4, and 100 fine-tuning steps per task, following/Jang et al. 20.2). This setup differs from the evaluation setting used in CAMeLU, where predictions are obtained with a single forward pass, leveraging the in-context learning ability of transformer architectures. However, SSL approaches must adapt to the test domain before label predictions, resulting in a less challenging evaluation setting than CAMeLU. Results for SSL approaches are averaged over 500 test tasks and presented in Tab. 3 While SSL approaches outperform CAMeLU on minilmageNet and CIFAR-fs, their performance decreases when evaluated on the other datasets. CUB, Aircraft, and Meta-iNat are fine-grained datasets significantly different from ImageNet- challenging the transferability of features learned by SSL methods these datasets. Moreover, the high performance on minilmageNet and CIFAR-fs may be attributed to the presence of data leakage with ImageNet-IK Ik and the high similarity with CIFAR-fs, as discussed in Sect. 44.2] CAMeLU, in contrast, demonstrates effective generalization to tasks sampled from these datasets, once again highlighting its generalization ability over mere memorization.
5 CONCLUSION
In this paper, we introduce CAMeLU, novel approach for UML that leverages the in-context learn- ing capabilities of transformer architectures to extract context from the support samples and make effective predictions ing problem, where support images provide task context for predicting query images. At the core of CAMeLU is a novel task creation mechanism that generates diverse tasks from an unlabeled dataset, promoting effective generalization to unseen tasks. Our experimental results showcase the superior- ity of CAMeLU over existing UML methods, highlighting the applicability of the proposed method to domains different from the training one. Notably, CAMeLU can generalize to new domains with a single forward pass (real-time predictions). and it even outperforms its supervised counterpart thanks to its task creation mechanism. Furthermore, the proposed model can be stored and trained with single GPU with only 8GB of VRAM, underscoring its efficiency rather than using meta-training phase typical of previous meta-learning approaches.
the query data. CAMeLU reframes meta-learning as sequence model-
warning-to-lean in-context,
Future research directions may explore extensions of CAMeLU to more complex domains, as well as investigations into further improving the task creation mechanism for enhanced generalization. It would be interesting to incorporate SSL techniques to obtain more robust feature representations and enhance generalization capabilities. Additionally, conducting further investigation into CAMeLU's ability to encourage generalization over memorization would provide valuable insights into its learn- ing dynamics and potential areas for improvement.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT AND REPRODUCIBILITY GUIDELINES
In this work, we used well-established, publicly available datasets to train and evaluate our archi- tecture. While these datasets and pre-trained models provide valuable foundation for research, acknowledge the potential for inherent biases that may not fully represent diverse real-world scenar- ios. We have taken every precaution to ensure that our experiments are conducted responsibly, with no intention of causing harm or perpetuating any biases present in the data. Furthermore, we declare no conflicts of interest in the execution or reporting of this research. Our objective is to present the findings in transparent manner and contribute positively to the broader research community.
To ensure the reproducibility of our experiments, we have provided the code and detailed instructions on how to run the experiments. The general configuration of our model is described in Sect. 4 with additional technical details outlined Sect. A.1 the Appendix. Moreover, we have employed random seed initialization to ensure consistency across runs. The complete codebase, models, and pre-trained weights are available on GitHub to facilitate further research and replication.
ACKNOWLEDGMENTS
This work was supported by the "Knowledge Foundation" (KK-stiftelsen).
htteee///gthhbe.coorrrrcs//////f 11

---

Published as a conference paper at ICLR 2025
REFERENCES
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005-25017, 2022.
Bertinetto, J Henriques, P Torr, and A Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICLR), 2019. International Conference on Learning Representations, 2019.
Jirka Borovec, William Falcon, Akihiro Nitta, Ananya Harsh Jha, otaj, Annika Brundyn, Donal Byrne Nathan Raw, Shion Matsumoto, Teddy Koker, Brian Ko, Aditya Oke, Sidhant Sundrani, Baruch, Christoph Clement, Clément Poiret, Rohit Gupta, Haswanth Aekula, Adrian Wälchli, Atharva Phatak, Ido Kessler, Jason Wang, JongMok Lec, Shivam Mehta, Zhengyu Yang. and / http://ightttnnggoooee bolts readth
Garry O' O'Donnell. Pytorch lightning bolts. https::
fo/en// latest/ 2022. Online; accessed 25 Apr 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Universeg: Universal medical image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21438-21 -4451,, 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang. Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- ing in transformers. Advances in Neural Information Processing Systems, 15:18878-18891 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. survey on in-context learning. arXiv preprint arXiv:2301. 1002344 2022.
Christopher Fifty, Jure Leskovee, and Sebastian Thrun. In-context learning for few-shot molecular property prediction. arXiv preprint arXiv:2310.0886. 2023.
Christopher Fifty, Dennis Duan, Ronald Guenther Junkins, Ehsan Amid, Jure Leskovee, Christopher Re, and Sebastian Thrun. Context-aware meta-learning In The Twelfth International Conference on Learning Representutions, 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1 1126- 1 135. PMLR, 2017.
Shivam Garg, Dimitris Tsipras. Percy S Liang, and Gregory Valiant. What transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
12

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-97388 2020.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. Interna- tional Conference on Learning Representations, 2018.
Huiwon Jang, Hankook Lee, and Jinwoo Shin. Unsupervised meta-learning via few-shot pseudo- supervised contrastive learning. In The Eleventh International Conference Learning Repre- sentations, 2022.
Siavash Khodadadeh, Ladislau Boloni, and Mubarak Shah. Unsupervised meta-learning for few- shot image classification. Advances in neural information processing systems, 32, 2019.
Siavash Khodadadeh, Sharare Zehtabian, Saeed Vahidian, Weijia Wang, Bill Lin, and Ladislau Boloni. Unsupervised meta-learning through latent-space interpolation in generative models. In International Conference on Learning Representations, 2020.
Louis Kirsch, James Harrison, Jascha Sohl-Dickstein, and Luke Metz. General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:22/2.044458 2022.
Deqian Kong, Bo Pang, and Ying Nian Wu. Unsupervised meta-learning via latent space energy- based model of symbol vector coupling. In Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2021.
Dong Bok Lee, Dongchan Min, Seanie Lee, and Sung Ju Hwang. Meta-gmvae: Mixture of gaussian vae for unsupervised meta-leaminng In International Conference on Learning Representations, 2020.
Dong Bok Lee, Seanie Lee, Kenji Kawaguchi, Yunji Kim, Jihwan Bang, Jung- Woo Ha, and Sung Ju Hwang. Self-supervised set representation learning for unsupervised meta-learning. In The Eleventh International Conference Learning Representations, 2022.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference Machine Learning, pp. 19565-19594. PMLR, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Chen Liu, Yanwei Fu, Chengming Xu, Siqian Yang, Jilin Li, Chengjie Wang, and Li Zhang. Learn- ing few-shot embedding model with contrastive learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 8635-8643, 2021.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv: 1306.553. 2013.
13

---

Published as a conference paper at ICLR 2025
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to leam in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, 2022.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Represenlations, 2018.
Stylianos Poulakakis-
and Hadi Jamali-Rad. Beclr: Batch enhanced contrastive few-shot learning. In The Tweifth International Conference on Learning Representations, 2024.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. Meta-learning requires meta-auggmentttio. Advances in Neural Information Processing Systems, 33:5705-5715, 2020.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International conference on learning representations, 2016
Brigit Schroeder and Yin Cui. Fgvex fungi classification challenge 2018. 2018. Online; accessed 25 Apr 2024.
com/vi www.sipeeiaaaaee
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances neural information processing systems, 30, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amn Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Joaquin Vanschoren. Meta-leaming. Automated machine learning: methods, systems, challenges, pp. 35-61, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
14

---

Published as a conference paper at ICLR 2025
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference machine learning, pp. 6438- 6448. PMLR, 2019.
Anna Vettoruzzo, Mohamed-Rate Bouguelia, Joaquin Vanschoren, Thorsteinn Rognvaldsson, and KC Santosh. Advances and challenges in meta-learning: A technical review. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd binds-200-201/ dataset. California Institute of Technology, 2011.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4);600- 612, 2004.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization in realistic settings. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp- 2019.
Patrick H Winston. Learning and reasoning by analogy. Communications of the ACM, 23(12): 689-703, 1980.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: State-of-the the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Huaxiu Yao, Long- Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, al. Improving generalization in meta-learning via task augmentation. In International conference on machine learning, pp. 11887-11899 PMLR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTAL DETAILS
Datasets. For training CAMeLU, we use ImageNet-964, which is variant of the original ImageNet-Ik Ik dataset (Deng et al. 2009) where classes belonging to the validation and test splits of mindImageNett (Ravi& & Larochelle 2016) are removed. This results in a total of 1 234 487 images for training the model compared to the I 281 167 in the original ImageNet- Ik dataset. When a multi- dataset approach is utilized for training CAMeLU (see Appendix (A.3). MSCOCO (Lin et al. 2014) and Fungi (Schroeder & Cui 2018) are loaded into the program and used together with ImageNet- 964 for creating the whole training dataset. MSCOCO is dataset originally proposed for object detection, where each image is assigned to G classes corresponding to the G objects present in it. To use it for image classification, we replicate each image G times and we assign to each of them one of the G classes. In this way, we obtain dataset with 117 266 images for training, and we rely on the fact that the transformer is capable of applying self-attention to the object of the class in question. Fungi is fine-grained dataset with a size of only 64 307, which is two orders of mag- nitude smaller than ImageNet-964 and MSCOCO. For evaluation and for in-domain training of the baselines, we use minilmageNet, CIFAR-fs, CUB, Aircraft, and Meta-iNar. minilmageNet is split into train/validation/test using the splits proposed in Ravi & Larochelle (2016). resulting in 38 400 images for training, 9600 for validation, and 12 000 for testing. The same number of images are also present in CIFAR-fs, and the splits follow the work in Bertinetto et al. (019). CUB and Aircraft, instead, are two graingr datasets with smaller size compared to the others. CUB (Wah al.) 2011] consists of 8239 in the training set, 1 1779 in the validation set, and 1770 in the test set, while Aircraft has respectively 7000/1500/1500 images in the train/validation/test sets (Triantafillou et al.) 2019). Finally, Meta-iNat (Wertheimer & Hariharan 2019) consists of 243 986 images split into 1135 classes, with 227 reserved for testing. All images are resized to 224 x 224 and normalized with zero mean and unit variance before input into the model. For the UML baselines, due to the smaller model size utilized in the experiments, images are resized to 84 x 84, as suggested in the
original papers Khodadaadeh et al. 2019 Hsu et al. 2018 Lee al. 2020 Jang et al. 2022.
CAMeLU. The architecture used for CAMeLU consists of a fixed pre-trained feature extractor, class encoder, and a transformer encoder. The feature extractor is pre-trained using ResNet-50 on ImageNet-966 964 following the same architecture and hyperparameters in He et al. (2016). The class encoder is single learnable layer with a dimensionality of 256 and initialized with Kaiming initialization (He et al. 2015). Image embeddings are concatenated with class embeddings before being fed into the transformer encoder. This results in a vector with total length of 2304, composed of 2048 features from the image embeddings and 256 from the label embeddings. When ablating the feature extractor with CLIP (Radford et al. [022]) in Appendix A.4 a ViT-B/16 encoder architecture is utilized and downloaded from the Hugging Face website (Wolf et al. 2019). The pre-training is performed using a large dataset with 400 million (image, text) pairs (Radford et al., 202T), and it is fixed during the training phase of CAMeLU. The output size of the image embedding is reduced to 1024, with 768 features from the image embedding, which results in a reduced memory complexity compared to ResNet-50. The transformer encoder comprises 8 encoder layers. Each layer consists of 8 attention heads and an MLP with a reversed bottleneck of 3072 (with GeLU activation function). A projection layer completes the model architecture to map the transformer output class prediction. This architecture enables us to store the entire model in an Nvidia GeForce RTX 3070 Ti Laptop GPU with 8GB of VRAM, while a further reduction of memory can be achieved by utilizing CLIP on ViT-B/16 as feature extractor, which requires only 4 GB of VRAM to store the entire model.
For the task creation mechanism. 3 augmentation functions are selected from a list comprising crop- ping, rotation, horizontal flip, grayscale, color jittering, gaussian blur, and random affine transfor- mation. The exact parameters used in our experiments for each augmentation function are detailed in Tab. 4 For the query set, additional pixel-level mixing strategy with À ~ Beta(a, 33 with and X e (0, 0.5) is utilized. More details about this selection choice can be found in
a = 1, 3
Appendix A.6
The training of CAMeLU is performed for 100 epochs, with 500 episodes each, using the Adam optimizer with an initial learning rate of 10-5 and a warmup cosine scheduler with 1500 warmup steps and a final learning rate of 10 6 For the evaluation, instead, a single forward pass is performed and the accuracy between the output and the true label is calculated on the query set of each given
16

---

Published as a conference paper at ICLR 2025
Table 4: Complete list of transformations used for generating the support set of each task in CAMeLU. The names of the augmentations are taken from the torchvision library in Python.
Augmentation
Parameters
RandomResizedCrpp
image size 224, scale (0.2, 0.8), ratio= (0.75, 1.33) degrees = 60. probability = 1.0
RandomRRtationn
RandomiHorizontalFiig
probability 1.0
Grayscale
output channels =
ColorJitter
brightmess 0.2, contrast 0.2, saturation = 0.2, hue 0.2
GaussianBlur
kernel size = 3, sigma = (0.1,2.0)
RandomAffine
degrees shear 45, 45, -55 45)
task. Results are then averaged across 500 tasks, and the mean and standard deviation across three complete runs (consisting of training and evaluation) of the algorithm are used our experiments.
Baselines. We compare our results with UML methods, an unsupervised few-shot learning method, a supervised meta-learning method, and two SSL methods. For the UML baselines, we consider CACTUs-MAML (Hsu et al. 2018). CACTUs-Protoke
(Hsu et al. 2018), UMTRA (Kho-) dadadeh et al. 2019), Meta-GMVAB Lee et al. 2020), and PsCo (Jang et al. 2022). All these meth- ods evaluated in-domain, i.e., using the same dataset for training and evaluation, to adhere the setting proposed in the original papers. Only PsCo is also extended to the cross-domain scenario that we discuss in this paper. All methods are trained for 100 epochs, using the parameters reported in the original papers (Khodadadeh et al. 2019 Hsu et al. 2018 Lee et al. 2020 Jang et al. 2022). and evaluated with 100 adaptation steps on each task when required by the model. When evaluated in-domain, all approaches use a Conv5 architecture consisting of 5 convolutional layers with 64 filters and a kernel size of 3, followed by batch normalization, ReLU non-linearity, max pooling, and a classifier head. The only exception is Meta-GMVAE. For this method, the authors trained Conv5 feature extractor with SimCLR and input the learned features into a variational autoencoder (VAE) (Lee et al. 2020). Due to time limitations and the computational resources required to train a model with SimCLR, in our experiments, we used a feature extractor consisting of a pre-trained version of SimCLR on ResNet-50 (Borovec et al. 2022) using ImageNet-1k, followed by pro- jection layer fine-tuned for 100 steps to the training dataset, as done for the SSL baselines. This approach results in a better performance than the one reported in the original paper (Lee al. 2020) (see Tab. [T. likely due to the improved ability of the feature extractor to extract meaningful fea- tures. For PsCo, when evaluated on cross-domain setting, we utilized the ResNet-50 architecture trained on ImageNet-964 to avoid data leakage, and we then applied the model to the test domain using 100 adaptation steps to it. We also included BECLR (Poulakakis- baaakiii-aatlliiiii 2024 in our comparison utilizing the same hyperparameters and model architectures proposed in the original paper, given the importance of hyperparameter choice for final performance. Specifically, the ResNet-50 feature extractor was trained only on minilmageNet and evaluated cross-domain scenarios in Tab.
Jamali- Rad
To compare our results with CAML (Fifty et al. 2024), the same architecture and hyperparameter of our approach are applied to this method. This results in a lower performance for CAML compared to the original paper (Fifty et al. 2024), as the reduced size of the model and the different feature extractor (ResNet-50 instead of VIT-CLIP), but it guarantees fair comparisons and lets us train the model with the available computational resources.
We also provide a comparison with two SSL approaches SimCLR (Chen et al. 2020) and SWAV (Caron al. 2020). The details for training and evaluation are provided in Sect. 4.6
A.2 IN-CONTEXT LEARNING ANALYSIS
To verify the contribution of the in-context learner in CAMeLU, we examine the embedding space learned during inference, both after the feature extractor and the transformer encoder. Fig. pressents t-SNE visualization of single test task, where clusters represent the embeddings of the support classes. For simplicity, we illustrate a 5-way 5-shot task with one query sample for each dataset, and report the Fuclidean distance between the query and the centroid of each class. As the dataset
17

---

Published as a conference paper at ICLR 2025
0.5
Class
Claso
0.0
10.0
Class
-055
8.5
Class
-1.0
Class
7.5
-0.5
6.0
0.5
1.0
1.0
(a) Feature extractor (minilmageNet)
(b) Transformer encoder (minilmageNett)
150
Class
33
Class
100
Class
3,0
Clsss
0.13
Class
Class
aP
Class 5
2.5
Clasz 5
2.0
0
66
325
100
-1000
-50
50
100
130
3.0
35
4.0
4.5
5.0
5.5
6.0
8.5
(c) Feature extractor (CUB)
(d) Transformer encoder (CUB)
30
Cless
Cless
-10.0
-10.5
Cleos
1:44
-11.0
-10
-115
-20
11.00
Clss3
Clazz
Dass
-10
-20
(e) Feature extractor (Aireraft)
(f) Transformer encoder (Aircraft)
Figure 5: Visualization of clustered embeddings obtained with CAMeLU after the fixed feature extractor (left) and the transformer encoder (right) across different datasets. The plots represent 5- way 5-shot tasks during inference. Crosses indicate the centroids for each class, triangles represent the query sample embeddings, and the numbers denote the Euclidean distances between the query and each class centroid. The plots are obtained using (-SNE (Van der Maaten & Hinton 2008) with a perplexity equals to 9.
18

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of CAMeLU and CAML when trained on the single ImageNet-964 dataset (IN-964) and on multi-dataset (mds) consisting of ImageNet-964 + MSCOCO + Fungi. Results show the mean and standard deviations for the 5-way 1-shot and the 5-way 5-shot settings across three complete runs of the algorithms.
CIFAR-
Ainczal
MetsoNia
Swis
Swhe
CAMELU (NN-N4
76.51
0:70
61.77 0:30
S0.03
05,52 A 0:37
80:35
066
3511 0014
10:13
1,47
57:07
75.45 0:40
CAMELU (ada)
78,56 0,36
91,82 d: 0.21
62.41 2 0.70
50.36 à 0.32
60.36 à 0.70
79.78
0.85
02.04 a 0.41
07.87 = 0.00
07.96 a 0.53
15.40 0.22
CAME (IN-964)
81.75
16.27 4.77
54.88
66.1
8.18
0.87
02.05
0.43
60,86
07.07 0.550
CAML (nxfs)
61.10/ 0.54
W/.03 0.33
0:45
70.75
36.85
60:43
1.85
18.36
56
54.77
complexity increases, we observe greater variation between the embeddings learned from the fixed feature extractor and those refined by the transformer encoder. For instance, with the manelm dataset, the feature extractor alone is able to recognize the class of the query, clustering the query sample with the support samples belonging to the same class. However, in more challenging datasets such as CUB and Aircraft, the embedding space after the feature extractor appears more sparse, reflected by the large Euclidean distance between the query sample and the centroid of each class. In contrast, the transformer encoder significantly improves the representation, producing more compact and well-separated clusters, underscoring its crucial role in CAMeLU, For instance, in the Aircraft dataset, the query would be misclassified as class 5 (in grey) based on the feature extractor alone, but correctly classified after passing through the transformer. This highlights the role of the transformer encoder in updating the support and query representations based on the context of the task, not only the image context, improving classification accuracy.
A.3 MULTI-DATASET TRAINING
We conduct additional experiments to evaluate the performance of CAMeLU when trained on large-scale dataset. As our focus is on cross-domain classification through in-context learning, we hypothesize that training formance, as suggested in Min et al. (2022) and FFfft et al. (2024). To test this hypothesis, we com- bine three training datasets with varying levels of granulaniy: ImageNet-964 (Deng et al.) 2009), MSCOCO (Lin et al. 2014), and Fungi (Schroeder & Cuil 2018). During each training episode, a dataset is uniformly sampled, and N data points are extracted. These samples are then utilized in method as described in Sect. 3 Table presents the results for CAMeLU and the supervised CAML method. Notably, training with combination of multiple datasets (denoted as mds in Tab. 5) yields improved performance for CAML (supervised) compared to training solely on ImageNet 964, likely due to the increased variability in the sampled tasks. However, CAMeLU does not exhibit a similar performance boost, its task creation mechanism already introduces substantial variability, reduc- ing the benefit of additional dataset diversity. Moreover, the large size of ImageNet-966 (around 80% of the overall dataset), leads to its more frequent selection compared to the other datasets and limits the potential for performance gains from the other datasets. Consequently, to optimize computational resources and time, we conducted all the other experiments by training solely on ImageNet-964.
dataset spanning various concepts could enhance classification per-
A.4 ABLATION STUDIES FEATURE EXTRACTOR
To assess the impact of the pre-trained feature extractor, we evaluate CAMeLU with various extrac- tors pre-trained using different strategies. In particular, we compare the effectiveness of a ResNet-50 encoder pre-trained in supervised manner both on ImageNet- Ik and on ImageNet-964,, a ResNet- 50 pre- pre-trained on ImageNet- Ik using two SSL strategies. i.c., SimCLR (Chen et al. (2220) and SWAV (Caron et al. 2020). and a ViT-B/16 architecture pre-trained with CLIP (Radford et al. 2021 on a large dataset with 400 million (image, text) pairs. The extractors were downloaded from the Hug- ging Face (Wolf et al., 2019) and PyTorch Lightning Bolts (Borovec et al. 2022) websites, except for the Re architecture pre-trained on ImageNet-964. Results in Tab. 6a demonstrate that the models pre-trained on ImageNet- Ik exhibit significantly higher performance on minilmageNet, compared to other datasets, primarily due to the data leakage issue described in Seet. 4.2 This issue is mitigated by pre-training the ResNet-50 architecture on ImageNet-964, which results in a drop in the performance minilmageNet and CIFAR-fs due to the removal of data leakage, making it
19

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of feature extractors utilizzd in CAMeLU. The feature extractors include ResNet-50 pre-trained on ImageNet-964 (ResNet50 (IN-964)), ResNet-50 pre-trained on ImageNet- 1k (ResNet50 (IN-1k)), ResNet-50 pre-trained on ImageNet-1k with SimCLR and SwAV, as well as a ViT-B/16 architecture pre-trained with CLIP. All models are trained using CAMeLU on (a) ImageNet-964 or (b) using a combination of ImageNet-964 MSCOCO + Fungi (multi-dataset). The symbol i indicates results that are affected by data leakage. Results show the mean and standard deviations across three complete runs of the algorithms.
vinlingaaa
CEAR66
CUB
Anxral
Recher 50 GIN 964)
76.51 0.7
10:30
60.75 1150
W1-03 0:01
65.52 8.37
ND
1.0.3
11.17 1.0
28-31 1 1.55
57:47 1433
RoiNa 50
18.17 1.00
05. 0.40
66.02 0.78
84.00
00.00
75.08 = 0.75
33.58 m.3
40.6 0
55.01 -.41
74:85-32-1
StvCLR
79:47=33
1.54
01:00 877
50:3- 8.58
27:30=2911
-3777-377
VIE- -
76.41 031
01356:..:11
0.00:00.0
66.20 0.92
01:00:1100
87
9.14
48.00 10.00:110 1.65
61:32
17:00:0:1
(a)
ImageNet-9644 training
WisaltageNet
Estradox
Sw
Sw5s
Swis
Swds
Swls
Sw5
Swis
Sw5s
Swis
Sw55
50.NN-90
RodNni-t 4D- Sinci
53.31-41100 1:88
76-10-41:91
05 1005
03:90
0.13
67.68
1.70
51.05
1:20
15.31 0:40
08.87
41.55 6-15
18.57-030
RecNn--DD- SD----D--AAA SeiAV
03-15-4 6:31
3733--00 0:57
78-99-4
1:01
19
0.0
00:11
0.83
52.30
-
5130-085
47.15 0.13
15.05-1110
VIT-0066 (TPPP
77.02:411.0
78.01:0000
91
0:45
74.08:1
80.66 2:08
48:11
58.97 1:71
07
02.59
(b) Multi-dataset training
comparable with the results on CLIP-ViT-B/IE For CLIP-VIT-B/16 however, thorough verifica- tion of potential data leakage was not possible due to the undisclosed nature of its training dataset. As such, these results should be interpreted with caution. ELIP-ViT-B//I stands out as the best- performing method due to its dataset-agnostic nature and its ability to learn representations that generalize across a broad range of tasks. Furthermore, when training CAMeLU with a multi-datase approach (Tab. (6b) deseribed in Appendix AA.3] results for CLIP- HIP--VT-BL B/16 improve further, high- lighting its applicability also to datasets significantly different from those used for training (Radford) al.) 2021). These findings demonstrate that CAMeLU's performance scales with the strength of the feature extractor, indicating potential for further investigation as more robust feature extractors become available. However, in this work, we decided use the ResNet-50 architecture to guarantee a fair comparison with previous baselines.
A.5 EVALUATION THE TASK CREATION MECHANISM
To assess the effectiveness of the task cre- ation strategy employed in our proposed ap- proach, we conduct comparative analysis of CAMeLU's performance under two different task creation mechanisms. Specifically, we evaluate CAMeLU when tasks are generated
0.9
Validation accuney
0.7
0.6
0.5
solely using data augmentations for both the support and query sets, following strategy similar to UMTRA (Khodadadeh et al.) 2019),
0.4
Proposed startegy (mini:
0.3
Data augmentation (mit))
Proposed startegy (CUB)
0.2
augmentation (CUB)
versus employing our proposed approach out- lined in Sect. 3.1 By applying our task cre- ation strategy, we generate more complex tasks, making the generalization problem harder and the in-context learner more robust (Chan et al. 2022 Singh et al., (024)). The results presented in Tab. 7a confrrm our claim. Across all the datasets, our proposed strategy enhances the generalization on cross-domain datasets such as CUB, Aircraft, and Meta-iNat. This conclu- sion further supported by Fig. which shows the validation accuracy
80
Epochs
Figure 6: Validation accuracy on minilmageNet (mini) and CUB while training CAMeL.U with two different task creation mechanisms. Red and purple curves are obtained with our proposed strategy in Sect. 3.1 while orange and pink curves are obtained by applying only data augmentations based on image manipulations to generate the sup- port and query samples. The training is performed using ImageNet-964.
minilmageNet and CUB using the two mechanisms discussed be- fore, along with the CLIP--Vi ViT feature extractor described in Appendix A.4 These results confirm
20

---

Published as a conference paper at ICLR 2025
Table 7: Ablation experiments of the proposed task creation mechanism by (a) generating tasks only using data augmentations for the support and query set on ImageNet-964 and (b) applying k-means clustering on the ResNet-50 embeddings on minimmageNet. Results show the mean and standard deviations across three complete runs of the algorithms in the 5-way 1-shot and 5-way 5- shot scenarios.
CIFAR&
CLU
Airend
MoueeNNa
Merboal
Swdc
Suls
Swis
64 0.31 01
81
75.53
1.50
1.09
37.
1.71
56.05 1.51
74:00
Proposol
26.01 2 0.79
92. 0.30
01.79 a 0.30
50
021
66 52 a 0:77
80.36
0.63
38
0.94
59.1
67.27 0.99
78.45 a 0.42
Data augmentation vs. the proposed strategy
CHAR-S
Alron
Mass-
Method
Swls
5u.5
S0ds
Suls
Swis
50.5
k-maaas
TG.86 + 1.27
89.57 = 0.60
1 = 1.00
2
81 a 3.58 60.00
mmm LIN
24.8 aaa
25 4:88
36.22 = 1.08
47 27 1,50
Proposed
70.99
90.38
78.79
0.50
1477 1.70
31.99 1.17
96.02
00.00 0.30
72:13:0:30
(b) k-means clustering vs. the proposed strategy
Table 8: Evaluation of the proposed task creation strategy when applied to build pseudo-tasks on top of MAML. The results are compared with other MAML-based baselines for UML., such UMTRA and CACTU--MAML.
minilmageNett
CIFAR-fs
CUB
Aircraft
Meta-iNat
Method
5w1s
5w5s
5w1s
5w5s
5w
5w5s
5w
5w5s
5w1s
5w5s
CACTUS-MAML
43.30
54.21
42.00
56.64
31.19
36.81
24.06
27.26
20.13
21.84
UMTRA
39.93
50.73
32.93
46.13
27.06
36.60
22.40
31.73
28.96
37.12
Proposed MAMI.
34.04
46.13
37.02
52.00
31.22
41.54
26.
34.34
30.94
42.43
that our task creation strategy is more robust, particularly in cross-domain evaluations, even when stronger feature extractors are utilized.
Additionally, we experimented with k-means clustering as an alternative to the proposed task cre- ation strategy. Inspired by CACTUs (Hsu et al. [0188), we applied clustering on the embeddings generated by the feature extractor to generate pseudo-labels. The results are presented in Tab. 7b when training on minilmageNeet due to the high computational cost of k-means and indicate that our proposed mechanism generalizes better across domains. Furthermore, k-means clustering requires an insight into the number of classes in the training dataset, as choosing high number of clus- ters would lead to a lack of samples per class, whereas a low number may hinder generalization. In contrast, CAMeLU does not rely on such assumptions, enhancing its robustness compared to clustering- based approaches.
Finally, we show that the benefits of our task creation strategy extend beyond CAMeLU. In Tab, 8 we apply our proposed mechanism to generate pseudo-taske on top of MAML (Finn et al. 2017 This allows for direct comparison with MAML-based approaches, such as UMTRA (Khodadadeh al.) 2019) and CACTUS-MAML (Hsu et al. 2018), by replacing their original task creation mechanisms. The increased performance of our strategy applied to the baselines demonstrates its superiority over previous task creation methods. It may be objected that CACTUS-MAMI achieves higher perfor- mance on minilmageNet and CIFAR-fs. However, this caused by the use of feature extractor pre-trained on ImageNet- 1k, which introduces an unfair advantage by leaking information about the test data into the training phase. This performance gap narrows when the test distribution deviates from the training data (e.g., CIFAR-fs) and disappears for datasets with low correlation to ImageNet- 1k, supporting our hypothesis. Indeed, on datasets that share low similarity with ImageNet- 1k, our method consistently outperforms both UMTRA and CACTUS-MAML. While these results high- light the strength of our task creation strategy, the performance still remains significantly lower than CAMeLU, especially in cross-domale scenarios. This emphasizes the critical role of combining our robust task creation mechanism with the in-context learning capabilities of transformer-bass architectures to achieve superior performance.
21

---

Published as a conference paper at ICLR 2025
Table 9: mSSIM values computed as the average be- tween SSIM Engj my ( and SSTM(2), 27 ..(r)) when x (a)
is
obtained using a pixel level or a patch level mixing strategy with X = 0.25 and À = 0.49.
(a) Image &n.j
Random
Query im- (d) Query im- age 2/tr) with age ( with pixel level mix patch level mix
image 2)
Figure 7: Visualization of a query image žn.j generated by mixing (a) the support image and (b) a randomly sampled image (c) the pixel level or (d) the patch level using 0.49.
À = 0.25
À == 0.49
Pixel level Patch level
0.60
0.61
0.56
0.57
A.6 QUERY SAMPLES GENERATION STRATEGY
We also conducted additional experiments to validate the choice of utilizing Eq. for generating query samples. In particular, we compare the results obtained as a linear combination mented image .._j with the randomly sampled image 2j (pixel level), as in Eq. [ and at the patch level, as in Yun et al. (2019). Specifically, for the latter, we randomly select a patch from 2, with an area ratio proportional to X, and we paste it into - Fig. 7 illustrates an example of these two techniques by mixing two images sampled from ImageNet-964. As shown in Fig. 77 merging the images at the pixel level results in mixed image where some information from In.j and zj is retained in every part of the image. Contrarily, in Fig. 7d there no information about žn.5 in the lower left corner, forcing the network to attend only to the upper right part of the image to classify it with the same class as Tn.j- Therefore, we hypothesize that the pixel level strategy is more suitable for our approach as the goal is attend to the whole image extract robust features that allow the model to classify the query image with the same class as the support one while ensuring diversity between the two. To validate this, we utilized the Structural Similarity Index (SSI M) (Wang al., 2004). SSIM is used as a metric to measure the similarity between two given images based on three image features: luminance, contrast, and structure. Formally, considering and y two given images, SSIM is calculated as follows:
the aug-
9 +
SSIM v)
2p + C1
20,0, + c
Ozy - C3
(5)
12 + 12 c1
+2 +
0-
T2 + c3
where μ represents the mean of an image, a denotes the standard deviation, C1, C22 C3 are constant values, and a, B, y denote the relative importance of each metrics. By assuming a = B = and C2 = 02/ c2/2. we get
(2pz2y + (2) + SSTM(a,y) (n2 + p3 + (o) + oo c2)
Instead of applying the above formula all over the image at once, (Wang al. 2004) proposed a local variant that consists of computing the SSIM index locally and averaging these values to obtain the global SSIM value. For our purpose, we computed this metric between each of the two images used for the generation, i.e., En.j and zj. and the resulting mixed image, i.e., x; (/7r) We then average the results, obtaining indicator, denoted mSST M, of how similar the query image is with the images used for the generation, or in other words, how much local information is retained from ănا and 2, into 2,77) Results are shown in Tab. 9 for À 0.25 and X 0.49, confirming the hypothesis that, even for high À values, mixing at the pixel level guarantees more information retained across the whole image compared to using a patch level strategy. This is also confirmed by the results in Tab. 10 which shows decrease in the performance when CAMeLU is trained with the patch level strategy for query generation.
We also ablate the values of the and 3 parameters used in the Beta distribution from which X sampled. Tab. 10 presents the results for different values of a and 3 when X Beta(a, B) and (0, 0.5). The results indicate that the optimal choice for CAMeLU is to select 1,8 1,
22

---

Published as a conference paper at ICLR 2025
Table 10: Accuracy results of CAMcLU when trained with different strategies for generating the query samples. Pixel level mix refers to the scenario where query samples are generated with Eq. while patch level mix refers strategy similar the one proposed in Yun et al. (019). Results are reported in the 5-way 5-shot scenario with À ~ Bela(a, 3) and different a and 3 values. Results show the mean and standard deviations across three complete runs of the algorithms.
minlmaaeNeet
CIFAR-fs
CUB
Aireraft
Pixel level mix
0.1, 3 0.1
90.68 = 0.67
75.88 = 2.08
76.31 + 2.44
35.31 + 2.57
a = 0.5, 3 = 0.5
91.34 0.24
77.70 0.73
79.52 0.73
37.56 + 1.92
=
92.14 0.30
80.43 + 0.21
80.35 + 0.63
39.11 + 1.97
a 3 5
90.68 1.02
77.00 + 1.40
79.62 2.50
38.02 + 1.85
a 3
90.63 0.30
78.12 = 0.15
79.71 0.42
37.57 + 0.11
Patch level mix a = 1 =
91.25 + 0.50
77.80 +: 0.45
76.12 + 1.06
33.60 + 1.27
Table 11: Computational and time complexity of CAMeLU in comparison with PsCo. The compar- ison is performed considering the time required for the task creation, the training time (expressed in time per epoch), the inference time single task, and the GPU and CPU memory usage during training and inference.
Time task construction (ms)
Training time Inference time (ms/epoch)
(ms/task)
PsCO CAMeLU
20772
4613656
605
1376
153000
57
GPU training (MiB)
CPU training (MiB)
GPU inference (MiB)
CPU inference
(MiB)
PsCO
43904
20904
1630
2061
CAMeLU
6250
2588
3224
1667
which appears to be uniform distribution. Additionally, 2, 5 also yields comparable results, highlighting the importance of selecting a sufficiently small to ensure the incorporation of sufficient information from fn.j into 2//07) facilitating the model's ability to classify the latter with
the same class as Fn,5
(sp)
A.7 COMPUTATIONAL COMPLEXITY AND RESOURCES USAGE
In this Sect., we analyze the computational and time complexity of CAMeLU and we compare with PsCoJang al. (2022). Tab. presents the time required for task generation, model training. and inference, along with GPU and CPU memory usage. The results demonstrate that CAMeLU is only faster than PsCo but also significantly more memory efficient. Notably, CAMeLU requires only 57ms for task inference, making it particularly suitable for real-time applications.
The computational complexity of CAMeLU is primarily attributed to the transformer architecture, which is known for its computational demands due to the self-attention mechanism. The transformer model has a computational complexity of O(n2. d) per layer (Vaswani et al.) 2017). where the sequence length and dl is the hidden dimension. In our context, n includes both the support samples and the query sample. Consequently, the total computational complexity for evaluating Q queries is O(Q (N K + 1)2 d), where N is the number of classes, K the number of shots, N K + indicates one query per input sequence, and Q is the total number of queries. This results in quadratic complexity in the number of support samples which can be computationally demanding. However, we have demonstrated Tab. that CAMeLU achieves good performance even with only support sample per class. Additionally, further experiments with only query sample per episode
23

---

Published as a conference paper at ICLR 2025
05
044
04
02
82
0.1
www.iaggeeet logaz00 a0prooir330
(a) CAMcLU minilmageNett
(b) CAMcLU on CUB
07
0.4
2.5
033
9.4
Teahh 0.1
naeeragehel Ingitiic apprositatton
CLDB 500 aporooliaatoon
(c) CAMI. on minilmageNNet
(d) CAMI on CUB
Figure 8: Comparison of logistic function approximations and phase boundaries for learning and generalization phases in CAMeL.U and CAML for minilmageNet and CUB datasets.
and one support sample per class (i.e., N 5, K = =, 1) yield results of 80.74 + 0.65 for minilmageNet, 63.07 + 1.14 for CIFAR-fs, 54.84 + 1.41 for CUB, 30.32 + 0.76 for Aircraft, and 55.76 + 0.08 for Meta-iNat. These results are comparable to those reported in Tab. [or the 5-ways 1-shot scenario using 25 queries, highlighting that a single query is sufficient for good performance, as also demonstrated in CAML.
A.8 QUANTITATIVE ANALYSIS OF LEARNING PHASES
To quantitatively assess the number of epochs required to enter the generalization phase, propose to approximate the validation accuracy curves with the generalized logistic function
d a + --ee-bz-a0 = a + ce-bai
d
f(x) = a
where parameters a, b, c, d are responsible for particular features of the logistic function. Parameters a and d indicate the lower and, respectively, the upper asymptote. Parameter b the logistic growth e is related to the inflection point zo at which the maximum
rate, and finally, parameter growth of the function occurs. To find the best fitting logistic curve we use a standard regression function. The logistic function is strictly increasing, thus the derivative, which is given by f'(x)) be(d-a)e-hz (I+ce-napr, is always positive. The derivative firstly increases (from values close to zero), and after reaching its maximum value at the inflection point, it decreases. To determine the bounds for reaching the learning and generalization phases, we find the values for which the derivative equal given fraction of the maximum possible growth rate. After testing several cases, the results show that the choice of this threshold does not affect the relation between the phases' boundaries. Therefore, we decided conduct our analysis for 20% of maximum growth rate.
Figures Sa and 8b show the results for CAMeLU on the minilmageNett and CUB datasets, respec- 0.51
tively. For minilmageNet, we obtain the approximation function be f(x) 0.04 1+9636e 0.437. Moreover, the number of epochs where the learning phase begins is 15 and the number of epochs where the generalization phase begins is 29. On the other hand, for CUB dataset, the approximation 0.48
function is (xx 0.01 begin is 16 and 28.
155530 0...., and the number of epochs where the following phases
24

---

Published as a conference paper at ICLR 2025
Figures 8c and 8d show the results for CAML on the minilmageNet and the CUB datasets, spectively. Similarly, we obtain the approximation function for the mintImageNet to be f(x) = 0.06 I+1 +1326e 0:20z and the number of epochs where the learning phase and the general- ization phase begin is 18 and 42. Finally, for the CUB dataset, the approximation function is f(x) = -0.07 + 14648e
0.56
0.50 and the number of epochs where the following phases begin is
29 and 74.
Remarkably, CAML requires more training time to reach the generalization phase than CAMeLU. This difference likely arises from CAMeLUUss task creation mechanism, which generates tasks with high cross-task variance. This strategy acts as form of task augmentation, facilitating quicker generalization to unseen tasks.
A.9 LIMITATIONS
Despite the promising results demonstrated by our novel UML approach on several datasets, some limitations remain. Its applicability and robustness in real-world scenarios with diverse and noisy data remain to be thoroughly evaluated. In real-world applications, data can be incomplete, mis- labeled, or drawn from significantly different distributions, leading to potential degradation in the model's 's performance in the presence of noisy or corrupted data. Additionally, the feature extractor used in CAMeLU is pre-trained in a supervised manner. While the pre-training dataset is indepen- dent of the data seen at inference, replacing with an extractor pre-trained using an SSL strategy could make the pipeline fully unsupervised, albeit the price of performance degradation. Lastly, the proposed approach designed to handle a fixed number of classes (ways) per task during train- ing and testing, requiring knowing the value of N in advance. Modifications to the task creation and training process would be necessary to extend our approach to handle an arbitrary number of ways.
A.10 COMPLETE RESULTS WITH STANDARD DEVIATIONS
Performance comparison on minilmageNNt, CIFAR-Is, CUB, Aircraft, and Meta-iNat datasets for 5-way 1-shot and 5-way 5-shot scenarios. Cross- domain approaches are trained using Im2 and a ResNet-50 feature extractor. The symbol f indicates results that are affected by data leakage. The bold font highlights the best performing UMI approach for each setting. Results show the mean and standard deviations across three complete runs of the algorithms. This table refers to Tab. D in Sect. 4.3
Table 12:
iNua
Mahed
Suie
Sta54
SwSt
Swix
St
Sta5t
Sills
In-Dimaai
CACTU-MAME
14.21
8L.9 3.57
24.00
0.14
Proteee
21.71
04:02:0:344
44.43
UMTRA
15.5 L15
3078 0.67
12.90 2 185
2.51
27.06
2 1.41
2:442245
25:00
$1.77 2.25
26.96 2: 0.35
87.12 0.21
McS-OMVAF
13.25-10500 0.50
03.10
5210 0.84
6118 0.52
31.39 545
10:00 1
21.55 0151
27:01 0.53
56.22 0.35
01.33 0.54
PuCo
2..20
64.55 0.35
12:11 0..45
02:02 0:44
33.05
0.44
51.00 = 0.42
26.10 -0 0.30
38. 0.38
36.97 0.33
03.85 + 0.41
PuCo
S1.17 2
T0 13.4
43.35 2: 9.47
70:99:
20.87 = 0:3)
58.21 0.21
01.21 2:4.4
70415 2: 2.46
CAMcLU
T6.61 0.779
02114:030
01.70 0.50
80.43 0.21
65.62 5 0.57
05.17 0.94
30:11:2:191
07.27 5
78.45 0.42
CAML Depervood)
41.76-2..1 0L18
$2.51 #..1
05.31
25.50 =2 0.37
$2.01 0.41
0.350
07.07 4: 0.30
Table 13: Accuracy results obtained training PsCo, BECLR, and CAMeLU with a scall-scale dataset, namely minilmageNet, denoted as (mini) in the table. Results show both in-domain per- formance (on the test set of minilmageNet) and cross-domain performance on CIFAR-fs, CUB, Air- craft, and Meta-iNat. The mean and standard deviation across three complete runs of the algorithms. This table refers to Tab. 2 in Sect. 4.5
Swis
Sw1s
Sw/s
Sw1s
41 0.41
62.92
6.45
26.
38.00 0.38
55.68
BBCLI:
CAMe
70.99-4 0.50
90
78
74.77 1.70
36.52
55.60
72:12 0.35
25

---

Published as a conference paper at ICLR 2025
Table 14: Comparison between CAMeLU and SSL approaches for the 5-way 1-shot and 5-way 5- shot scenario on minilmageNet, CIFAR-fs, CUB, Aireraft, and Meta-iNat. The symbol indicates results that are affected by data leakage. Results show the mean and standard deviations across three complete runs of the algorithms. This table refers to Tab. in Sect. 4.6
CIRARS
CUB
Aircoft
Met--Nai
Methad
6652
14.8
94.96 0.91"
0031.
50.3 031
17.43 0.11
58.57
76.53
CAMALU
81.79
67.27
26


Reviews:
[{"review_id": "0bXlxp4XCw", "reviewer": "Reviewer_Lqp1", "summary": "This work aims to tackle few-shot learning (FSL) by learning from unsupervised dataset. It shares a similar idea with CAML[1] in model design by taking FSL as a sequence modeling problem. The instance embedding $f_{\\psi}$ and the class encoding $g_{\\phi}$ of  \n a support image are contacted to form a \"token\" in the context. of the sequence, and a transformer encoder learns from the context to generated output feature for the query. The output feature finally is projected by a top linear layer predict the class of the query. \n\nTo learn from the given unsupervised dataset, this work proposes a novel task creation mechanism to generate pseudo-tasks. In each run of the task creation mechanism, $N$ samples are randomly selected from the unsupervised dataset and data augmentation strategies are applied to each sample to generate $K$ instances (which are utilized as the support examples), simulating the $N$-way $K$-shot setting. Query samples in the pseudo-task are generated by mixup the selected sample (and its augmented version) with randomly selected samples in the \"background\" dataset. \n\nThe pseudo-task creation provides a way to perform meta-learning in the pre-training stage when no labeled dataset is available, which is interesting and instructive.\n\n\n[1] Fifty, C., Duan, D., Junkins, R. G., Amid, E., Leskovec, J., Re, C., & Thrun, S. Context-Aware Meta-Learning. In The Twelfth International Conference on Learning Representations.", "questions": "One more question:\nAs the class of instances in FSL are pseudo-labels randomly picked in range $0-N$, how to explain the top linear lay for classification that project the output embedding of the query into the final category distribution? An instance could be of class 0 in task $T_i$ but of class 1 in task $T_{i+1}$, how can the in-context classifier (i.e. the top linear lay) predict its class without finetuning upon the support set?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The work proposes a novel pseudo-task creation mechanism to generate FSL tasks out of unsupervised dataset, providing a new (maybe efficient) way to learn from unlabeled data. \n2) Experiments with CAML show that CAMeLU outperforms other unsupervised methods to achieve promising results.", "weaknesses": "However, I have the following concerns:\n1) The model in this work is the same as CAML, which shows limited novelty. \n2) Though the task formulation is interesting, only CAML and its successor (e.g. CAMeLU) are fairly trained in the setting. Other unsupervised method cannot leverage the supervised pseudo-tasks as supervised methods do. Essentially, CAMeLU is a supervised method. The pseudo-task generation provides a way to bridge the unsupervised scenario and the supervised scenario.\n3) Instead of providing a specific instance (e.g. CAML) to illustrate how the proposed task generation facilitates FSL from unsupervised dataset. I prefer to see more evidences that showcase the pseudo-task creation is a valid and efficient mechanism that makes supervised method like ProtoNet, MAML, etc, feasible in unsupervised scenario. This, I think, is the core contribution of this work.\n4) In experiment, building a new subset of Imagenet-1k (i.e. ImageNet-964) is not necessary since the method is so-called unsupervised. Note that, CLIP and its variants would cause information leakage if not carefully considered, because the texts may explicitly work as anchors to categorize instances belong to the same class together.\n5) I donot think it is proper to put experiments on CIFAR-FS into the cross-domain setting, because CIFAR-FS contains natural scene color images as in the miniImagenet. Conventionally, cross-domain setting refers to either transferring from coarse-grained setting to fine-grained setting as in FSL, or transferring from natural scene color image setting to painting image setting where instances from two domains show quite different appearances.\n6) Typos: e.g. In A.1 Line 793, the iamge embedding's output size decreases to 768, resulting ....", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AvxUX7JS7n", "reviewer": "Reviewer_HhvW", "summary": "The paper develops CAMeLU for unsupervised meta-learning setting, where the goal is to leverage the in-context learning capabilities of transformers to create transferable feature representations from unlabeled data for new tasks with small numbers of labeled data. By modeling meta-learning as a sequence prediction problem, each observed task is modeled as a sequence of support instances and an unknown query image. The transformer encoder subnetwork then identifies task-specific patterns from support images for that task which enhances accuracy of predictions on query images by making the model specialized for the input task. Data augmentation and task mixing are also used to generate diverse training tasks that challenge the model to improve generalization on new tasks. Experimental results on a benchmark formed from five datasets are offered to demonstrate that CAMeLU is effective for unsupervised meta-learning and leads to a performance close to supervised and self-supervised meta-earning methods.", "questions": "1. I was wondering if mixup is used during model training, would we still get a reasonable performance in a meta learning setting? \n\n\n2. What is the specific justification for the design that was used for the transformer model that is used for experiments? What is the computational overhead that it adds?\n\n3. Metalearning is overcrowded field with so many existing works. What was the reasoning to select methods that were used for comparison? It looks some of these methods are not the most recent methods which likely will have better performance. It is necessary to include 8-10 methods for comparison with majority of them to be most recent works that achieve the highest performances to demonstrate that the proposed method is competitive.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is relatively straightforward to read and follow.\n\n2. The performance boost compared to the baselines is significant on all benchmarks.\n\n3. The code is provided and although I did not run it but it looks well documented and easy to run which will make reproducing the results straightforward.", "weaknesses": "1. The model may become very large due to the added transformer model which computationally may become expensive and challenging to handle compared to methods are using only CNN architectures.\n\n2. Comparison is limited to only a few methods, despite the fact meta learning have a rich literature.\n\n3. Analytic experiments are limited and do provide much insight about the proposed method. For example, what are the circumstance and criteria under which the proposed method is more effective? What are the limitations? What is the reason behind the improved performance compared to the baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xgufj0L9Dp", "reviewer": "Reviewer_j5he", "summary": "This paper proposes CAMeLU, a novel approach to unsupervised meta-learning (UML) that utilizes transformer architectures' in-context learning capabilities to extract contextual information from support samples and perform predictions on query data as a sequence modeling task.", "questions": "1.\tDuring the construction of the query set using mixup, does the correlation between the sampled image and the combined augmented image affect the model's performance and stability? For instance, do cases where they belong to the same class or are completely unrelated impact the outcome?\n\n2.\tUnder time constraints, does the computational cost of this construction method lead to a significant performance improvement in the model?\n\n3.\tHow sensitive is the model's performance to the parameters used in the query set construction method?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tOvercomes the limitation in the UML field of relying solely on simple data augmentation for constructing training tasks, by proposing a novel sequence modeling-based task construction approach.\n\n2.\tEffectively leverages the advantages of in-context learning in LLM.\n\n3.\tProvides extensive experiments and a thorough hyperparameter tuning process, which comprehensively demonstrate the advantages of the proposed algorithm.", "weaknesses": "1.\tLacks an explanation of the benefits of combining transformer architecture with the proposed data construction method, and does not clarify whether other architectures could also be adapted. In short, there is insufficient discussion of the model's generalizability.\n\n2.\tLacks detailed descriptions of the fixed feature extractor f and the learned class encoder g.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOo7FI8JOj", "reviewer": "Reviewer_FYCV", "summary": "This paper reframes meta-learning as a sequence modeling problem, allowing the transformer encoder to learn task context from support images and use it to predict query images. The core is a mechanism for generating diverse tasks using a combination of data augmentations and a mixing strategy. Experimental results demonstrate the superiority over existing unsupervised meta-learning baselines, highlighting the efficacy of the model in leveraging generalization over memorization.", "questions": "* What are the training costs for CAMeLU? When constructing tasks from unlabeled datasets, especially when using a mixup strategy, do the hyperparameters need to be carefully tuned?\n\n* The comparison with CAML in Figure 3. Why doesn't CAML show an improvement in relative validation accuracy during training? What could be the reason for this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors propose a novel task creation mechanism that generates diverse few-shot tasks from unlabeled datasets through a combination of data augmentations and a mixing strategy. This seems reasonable.\n\n* They utilize the transformer’s in-context learning capability, eliminating the need for fine-tuning to specific test domains, which is commendable.\n\n* The experiments are thorough, and the exploration shows that the model achieved generalization.", "weaknesses": "* The comparisons in the experiments may not be entirely fair. For instance, differences in model architecture between methods, and whether additional feature extractors or class encoders were used, should be noted in a table to enable clearer comparisons. To my knowledge, this seems somewhat unfair to the in-domain baselines. It would be appreciated if a table showing the parameter count and computational overhead for each method were included.\n\n* In comparison to CAML, another transformer-based in-context learning method, or SSL methods, CAMeLU doesn’t perform as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)