Unsupervised Meta-Learning via In-Context Learning

Authors: Anna Vettoruzzo, Lorenzo Braccaioli, Joaquin Vanschoren, Marlena Nowaczyk
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: meta-learning, unsupervised learning, in-context learning

Abstract:
Unsupervised meta-learning aims to learn feature representations from unsupervised datasets that can transfer to downstream tasks with limited labeled data.
In this paper, we propose a novel approach to unsupervised meta-learning that leverages the generalization abilities of in-context learning observed in transformer architectures. Our method reframes meta-learning as a sequence modeling problem, enabling the transformer encoder to learn task context from support images and utilize it to predict query images. 
At the core of our approach lies the creation of diverse tasks generated using a combination of data augmentations and a mixing strategy that challenges the model during training while fostering generalization to unseen tasks at test time. 
Experimental results on benchmark datasets showcase the superiority of our approach over existing unsupervised meta-learning baselines, establishing it as the new state-of-the-art. Remarkably, our method achieves competitive results with supervised and self-supervised approaches, underscoring its efficacy in leveraging generalization over memorization.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729061785536

thread_type:
official_review

<thread_object>
creator: reviewer_fycv

<content>
summary:
This paper reframes meta-learning as a sequence modeling problem, allowing the transformer encoder to learn task context from support images and use it to predict query images. The core is a mechanism for generating diverse tasks using a combination of data augmentations and a mixing strategy. Experimental results demonstrate the superiority over existing unsupervised meta-learning baselines, highlighting the efficacy of the model in leveraging generalization over memorization.

strengths:
* The authors propose a novel task creation mechanism that generates diverse few-shot tasks from unlabeled datasets through a combination of data augmentations and a mixing strategy. This seems reasonable.

* They utilize the transformer’s in-context learning capability, eliminating the need for fine-tuning to specific test domains, which is commendable.

* The experiments are thorough, and the exploration shows that the model achieved generalization.

weaknesses:
* The comparisons in the experiments may not be entirely fair. For instance, differences in model architecture between methods, and whether additional feature extractors or class encoders were used, should be noted in a table to enable clearer comparisons. To my knowledge, this seems somewhat unfair to the in-domain baselines. It would be appreciated if a table showing the parameter count and computational overhead for each method were included.

* In comparison to CAML, another transformer-based in-context learning method, or SSL methods, CAMeLU doesn’t perform as well.

questions:
* What are the training costs for CAMeLU? When constructing tasks from unlabeled datasets, especially when using a mixup strategy, do the hyperparameters need to be carefully tuned?

* The comparison with CAML in Figure 3. Why doesn't CAML show an improvement in relative validation accuracy during training? What could be the reason for this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590101827

thread_type:
official_review

<thread_object>
creator: reviewer_j5he

<content>
summary:
This paper proposes CAMeLU, a novel approach to unsupervised meta-learning (UML) that utilizes transformer architectures' in-context learning capabilities to extract contextual information from support samples and perform predictions on query data as a sequence modeling task.

strengths:
1.	Overcomes the limitation in the UML field of relying solely on simple data augmentation for constructing training tasks, by proposing a novel sequence modeling-based task construction approach.

2.	Effectively leverages the advantages of in-context learning in LLM.

3.	Provides extensive experiments and a thorough hyperparameter tuning process, which comprehensively demonstrate the advantages of the proposed algorithm.

weaknesses:
1.	Lacks an explanation of the benefits of combining transformer architecture with the proposed data construction method, and does not clarify whether other architectures could also be adapted. In short, there is insufficient discussion of the model's generalizability.

2.	Lacks detailed descriptions of the fixed feature extractor f and the learned class encoder g.

questions:
1.	During the construction of the query set using mixup, does the correlation between the sampled image and the combined augmented image affect the model's performance and stability? For instance, do cases where they belong to the same class or are completely unrelated impact the outcome?

2.	Under time constraints, does the computational cost of this construction method lead to a significant performance improvement in the model?

3.	How sensitive is the model's performance to the parameters used in the query set construction method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613385992

thread_type:
official_review

<thread_object>
creator: reviewer_hhvw

<content>
summary:
The paper develops CAMeLU for unsupervised meta-learning setting, where the goal is to leverage the in-context learning capabilities of transformers to create transferable feature representations from unlabeled data for new tasks with small numbers of labeled data. By modeling meta-learning as a sequence prediction problem, each observed task is modeled as a sequence of support instances and an unknown query image. The transformer encoder subnetwork then identifies task-specific patterns from support images for that task which enhances accuracy of predictions on query images by making the model specialized for the input task. Data augmentation and task mixing are also used to generate diverse training tasks that challenge the model to improve generalization on new tasks. Experimental results on a benchmark formed from five datasets are offered to demonstrate that CAMeLU is effective for unsupervised meta-learning and leads to a performance close to supervised and self-supervised meta-earning methods.

strengths:
1. The paper is relatively straightforward to read and follow.

2. The performance boost compared to the baselines is significant on all benchmarks.

3. The code is provided and although I did not run it but it looks well documented and easy to run which will make reproducing the results straightforward.

weaknesses:
1. The model may become very large due to the added transformer model which computationally may become expensive and challenging to handle compared to methods are using only CNN architectures.

2. Comparison is limited to only a few methods, despite the fact meta learning have a rich literature.

3. Analytic experiments are limited and do provide much insight about the proposed method. For example, what are the circumstance and criteria under which the proposed method is more effective? What are the limitations? What is the reason behind the improved performance compared to the baselines?

questions:
1. I was wondering if mixup is used during model training, would we still get a reasonable performance in a meta learning setting? 


2. What is the specific justification for the design that was used for the transformer model that is used for experiments? What is the computational overhead that it adds?

3. Metalearning is overcrowded field with so many existing works. What was the reasoning to select methods that were used for comparison? It looks some of these methods are not the most recent methods which likely will have better performance. It is necessary to include 8-10 methods for comparison with majority of them to be most recent works that achieve the highest performances to demonstrate that the proposed method is competitive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614653508

thread_type:
official_review

<thread_object>
creator: reviewer_lqp1

<content>
summary:
This work aims to tackle few-shot learning (FSL) by learning from unsupervised dataset. It shares a similar idea with CAML[1] in model design by taking FSL as a sequence modeling problem. The instance embedding $f_{\psi}$ and the class encoding $g_{\phi}$ of  
 a support image are contacted to form a "token" in the context. of the sequence, and a transformer encoder learns from the context to generated output feature for the query. The output feature finally is projected by a top linear layer predict the class of the query. 

To learn from the given unsupervised dataset, this work proposes a novel task creation mechanism to generate pseudo-tasks. In each run of the task creation mechanism, $N$ samples are randomly selected from the unsupervised dataset and data augmentation strategies are applied to each sample to generate $K$ instances (which are utilized as the support examples), simulating the $N$-way $K$-shot setting. Query samples in the pseudo-task are generated by mixup the selected sample (and its augmented version) with randomly selected samples in the "background" dataset. 

The pseudo-task creation provides a way to perform meta-learning in the pre-training stage when no labeled dataset is available, which is interesting and instructive.


[1] Fifty, C., Duan, D., Junkins, R. G., Amid, E., Leskovec, J., Re, C., & Thrun, S. Context-Aware Meta-Learning. In The Twelfth International Conference on Learning Representations.

strengths:
1) The work proposes a novel pseudo-task creation mechanism to generate FSL tasks out of unsupervised dataset, providing a new (maybe efficient) way to learn from unlabeled data. 
2) Experiments with CAML show that CAMeLU outperforms other unsupervised methods to achieve promising results.

weaknesses:
However, I have the following concerns:
1) The model in this work is the same as CAML, which shows limited novelty. 
2) Though the task formulation is interesting, only CAML and its successor (e.g. CAMeLU) are fairly trained in the setting. Other unsupervised method cannot leverage the supervised pseudo-tasks as supervised methods do. Essentially, CAMeLU is a supervised method. The pseudo-task generation provides a way to bridge the unsupervised scenario and the supervised scenario.
3) Instead of providing a specific instance (e.g. CAML) to illustrate how the proposed task generation facilitates FSL from unsupervised dataset. I prefer to see more evidences that showcase the pseudo-task creation is a valid and efficient mechanism that makes supervised method like ProtoNet, MAML, etc, feasible in unsupervised scenario. This, I think, is the core contribution of this work.
4) In experiment, building a new subset of Imagenet-1k (i.e. ImageNet-964) is not necessary since the method is so-called unsupervised. Note that, CLIP and its variants would cause information leakage if not carefully considered, because the texts may explicitly work as anchors to categorize instances belong to the same class together.
5) I donot think it is proper to put experiments on CIFAR-FS into the cross-domain setting, because CIFAR-FS contains natural scene color images as in the miniImagenet. Conventionally, cross-domain setting refers to either transferring from coarse-grained setting to fine-grained setting as in FSL, or transferring from natural scene color image setting to painting image setting where instances from two domains show quite different appearances.
6) Typos: e.g. In A.1 Line 793, the iamge embedding's output size decreases to 768, resulting ....

questions:
One more question:
As the class of instances in FSL are pseudo-labels randomly picked in range $0-N$, how to explain the top linear lay for classification that project the output embedding of the query into the final category distribution? An instance could be of class 0 in task $T_i$ but of class 1 in task $T_{i+1}$, how can the in-context classifier (i.e. the top linear lay) predict its class without finetuning upon the support set?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111764431

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We would like to thank each reviewer for the valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, we propose a novel task creation mechanism to generate tasks from unsupervised data (reviewers Lqp1, j5he, FYCV) and we reframe meta-learning in the context of in-context learning for leveraging transformer’s capabilities (reviewers j5he, FYCV), thus eliminating the need for fine-tuning to test domains. We conducted extensive experiments to demonstrate the superiority of our approach compared to previous baselines (reviewers Lqp1, HhvW, j5he, FYCV) and we made the code available for easy reproducibility.

We have carefully considered and addressed all reviewers' comments. We have updated the manuscript (changes in blue) with the following additional discussions and experiments:
  - We have further clarified the contributions of existing in-context learning approaches relevant to our study and we have highlighted the main differences with our proposed method in Section 2.
  - We have highlighted in Section 3 the importance of considering the combination of our task creation strategy together with an in-context learner to achieve superior cross-domain performance.
  - We have conducted additional experiments (Table 8 in Appendix A.5 and the relative discussion in lines 1066-1080), to demonstrate the versatility of the proposed task creation mechanism which can be easily used in combination with other meta-learning approaches to enhance the unsupervised meta-learning performance. It is worth noting that the performance of CACTUs-MAML is affected by data leakage from its feature extractor (with *mini*ImageNet), but the performance gap decreases as the data distribution deviates from the training data. Nonetheless, the performance remains significantly lower than CAMeLU, emphasizing the unique advantage of CAMeLU in leveraging the in-context learning capabilities of transformer architectures.

>**Table 8 in Appendix A.5:**
| Method             | miniImageNet 5w1s | miniImageNet 5w5s | CIFAR-fs 5w1s | CIFAR-fs 5w5s | CUB 5w1s | CUB 5w5s | Aircraft 5w1s | Aircraft 5w5s | Meta-iNat 5w1s | Meta-iNat 5w5s |
|--------------------|-------------------|-------------------|---------------|---------------|----------|----------|----------------|----------------|----------------|----------------|
| **CACTUs-MAML**    | **43.30**         | **54.21**         | **42.00**     | **56.64**     | 31.19    | 36.81    | 24.06          | 27.26          | 20.13          | 21.84          |
| **UMTRA**          | 39.93            | 50.73            | 32.93        | 46.13        | 27.06    | 36.60    | 22.40          | 31.73          | 28.96          | 37.12          |
| **Proposed + MAML**| 34.04            | 46.13            | 37.02        | 52.00        | **31.22**| **41.54**| **26.12**      | **34.34**      | **30.94**      | **42.43**      |


  - We have added more details about the computational and time complexity of CAMeLU in Appendix A.7 (Table 11), by considering the time required for the task construction mechanism, the training time, the inference time and the GPU and CPU memory consumption during training and inference of the model. These results are compared with PsCo, which is the current state-of-the-art in the unsupervised meta-learning field and demonstrate that CAMeLU is significantly faster and memory efficient compared to it.

>**Table 11 in Appendix A.7:**\
> Time complexity
| Method   | Time Task Construction (ms) | Training Time (ms/epoch) | Inference Time (ms/task) |
|----------|-----------------------------|--------------------------|--------------------------|
| **PsCo** | 20772                       | 4613656                  | 605                      |
| **CAMeLU**| 1376                       | 153000                   | 57                       |

> Memory usage
| Method   | GPU Training (MiB) | CPU Training (MiB) | GPU Inference (MiB) | CPU Inference (MiB) |
|----------|--------------------|--------------------|---------------------|---------------------|
| **PsCo** | 43904              | 20904              | 1630                | 2061                |
| **CAMeLU**| 6250              | 2588               | 3224                | 1667                |

We hope our response and revision sincerely address all the reviewers’ concerns and we are open for further discussions during the rebuttal period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732172026325

thread_type:
public_comment

<thread_object>
creator: ~chris_fifty1

<content>
title:
Public Discussion Phase Remark

comment:
Was doing some late-night reading of meta-learning ICLR submissions and stumbled across this submission. Very cool work & interesting. Really like A.2; wish you could have brought it up into the main text---this shows why CAMeLU works & what's happening under the hood in a clear, simple way. Just a suggestion; I think it's understandings like this that'll help steer future method development. 

Will be sharing this with other meta-learning researchers.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work addresses the challenge of few-shot learning (FSL) by leveraging an unsupervised dataset. It adopts a similar approach to CAML [1] in its model design, framing FSL as a sequence modeling problem. This allows the transformer encoder to learn task context from support images and utilize it to predict query images. To effectively learn from an unsupervised dataset, the authors propose a novel task creation mechanism that combines data augmentation with a mixing strategy to generate pseudo-tasks. Experimental results demonstrate significant improvements over existing unsupervised meta-learning baselines, underscoring the model's ability to generalize effectively rather than rely on memorization.

All reviewers agreed that the pseudo-task creation mechanism is both novel and promising, with the empirical results showing extensive and significant improvements that validate its effectiveness.

Given the potential impact of this work on the meta-learning and few-shot learning communities, I recommend its acceptance.