Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new approach to meta-learning that addresses vulnerability to pseudo-labeling errors in state of the art meta-learning methods by improving pseudo-labeling using a momentum network and a queue of previous batches. The main insight in this work comes from the recognition that pseudo-labels are usually fixed in state of the art methods and not allowed to change as new information comes in. The proposed technique enables changing of the pseudo-labels using cross batch training. The proposed technique advances over the state the state of the art significantly when applied to a variety of few-shot classification benchmark tests. The proposed technique also advances over the state of the art in its scalability.  


strength_and_weaknesses: Strengths
1. Sound motivation of the work through thorough literature survey.
2. Innovative progressive refinement of pseudo-labels through the combination of the momentum network and a batch queue that ensures that information from past batches is used throughout.
3. Sound experimental results that show a clear advance over the state of the art. 

Weaknesses

1. No significant weaknesses.
clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is well written and easy to understand.

Reproducibility
The technique is certainly described in enough detail to be reproducible.

Quality and Novelty

The paper advances significantly over the state of the art in both the algorithm and the experimental results. The proposed technique is novel. 
summary_of_the_review: The paper motivates the proposed technique well. The proposed technique is sound and innovative, with a strong main insight of adapting pseudo-labels progressively. That represents a clear advance over the state of the art. The experimental results also show advancement over the state of the art. Good paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach to unsupervised meta-learning by drawing inspiration from recent self-supervised learning methods. Specifically, they employ an architecture similar to MoCo, where one branch is updated with momentum updates of the other branch, which is updated ‘online’. To form each task for meta-learning, they treat each example in the current batch as a ‘query’ and then propose a way of selecting relevant ‘shots’ for each query from a queue, where (the representation of) examples seen in previous batches are added, to serve as the support set for the episode. They utilize the Sinkhorn-Knopp algorithm to do this matching and pick the top K examples from the queue, for each query (where K is the shot of each class). Constructing tasks dynamically in this way can be advantageous compared to the approach followed by previous works where the set of tasks for meta-learning is created at the start of the meta-training phase and fixed, as this can lead to more variety as well as ‘better’ tasks (e.g. the support set of a class is a good match for its query set), since the representations become more informative throughout training. The use of a queue not only is useful for training with the MoCo objective in a self-supervised manner, but also can increase diversity within the episode, since the support examples can be selected from previous mini-batches too, leading to a larger pool of candidates. They use a supervised contrastive algorithm to solve each meta-training task. Their overall loss for training is this per-task contrastive loss, as well as the usual MoCo loss. At meta-test time, they use a nearest-centroid algorithm for single-source few-shot learning, while for cross-domain scenarios they further adapt the projection layers (small number of parameters) to handle the distribution shift. Specifically, they adapt on the support set via supervised contrastive learning where each support example plays the role of a ‘query’ and the pool for ‘searching’ is the rest of the support set; positive examples are those with the same label, since labels are known for the support examples. Empirically, they show improved results on some benchmarks over some previous methods.
strength_and_weaknesses: Strengths
========
[+] This is a really neat idea for leveraging progress in self-supervised learning for the problem of unsupervised meta-learning

[+] The paper is for the most part clearly written and well-organized

[+] Empirically, the proposed method seems to outperform some previous baselines

[+] A thorough ablation analysis is presented along with a set of additional experiments to further understand the performance of the proposed method

Weaknesses
===========
[-] The problem of unsupervised meta-learning is poorly motivated. More generally, whether meta-learning is the most fruitful approach for few-shot classification is questionable. Recent works [1-3] (see references below) show that simpler methods often work better. So it’s not clear why this framework is useful.

[-] The empirical investigation is limited to simple benchmarks for the most part (though I appreciated the cross-domain experiments), limited to very small convnet architectures (the few-shot learning community has long ago started using resnet-12 [1], resnet-18 [4], and larger models [3]), and the baselines compared against for supervised meta-learning are very basic and very far from the state-of-the-art.

References
==========
[1] A closer look at few-shot classification. Chen et al. ICLR 2019.

[2] Revisiting few-shot image classification: A good embedding is all you need? Tian et al. 2020.

[3] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin et al. NeurIPS 2021.

[4] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou et al. ICLR 2020.

clarity,_quality,_novelty_and_reproducibility: Clarity
=====
The paper is clear for the most part, with a few exceptions below.

In Equation 1, state what M represents exactly.

In Equation 1, for negative pairs where A_{i,j} is 0, the term inside the log will be 0, so the contrastive loss in this case will not be a function of the similarity between q_i and k_j. Is this an error? E.g. should A_{i,j} be -1 instead of 0 for negative pairs? 

Notation: k_i (e.g. in Equation 2) and z_i (in Section 3.1) seem to refer to the same quantity. Am I missing something? If not, change the notation to be consistent.

In Table 4, it’s not clear what removing Sinkhorn entails exactly. What is done instead? I’m assuming some different matching algorithm is used?

Novelty
======
While the proposed method heavily relies on MoCo, the particular adaptation to the problem of unsupervised meta-learning is novel to the best of my knowledge and interesting.

Reproducibility
============
The authors discuss implementation details in the Appendix. It’s hard to say how easy it would be to reproduce the presented results. I recommend the authors to release code upon publication.

Quality
======
The technical quality of the paper is high for the most part. However, I have some concerns about experimental comparisons, ablations.

Were SimCLR / MoCo trained on the meta-training set of Omniglot / mini-ImageNet for the experiments? This is kind of strange as the strength of these approaches exhibits itself when training on large datasets, with architectures much larger than 4 or 5 convolutional layers. So I think to truly compare against those approaches, it would be more interesting to use larger-scale pre-training.

The supervised meta-learning baselines used here are old and really far from the state-of-the-art, especially for cross-domain few-shot learning.

In Table 3, why are comparisons made only against MoCo v2? For instance, in addition to the self-supervised and supervised meta-learning baselines, it would be interesting to compare to a supervised off-the-shelf ImageNet trained model (that was not meta-learned, was simply trained for supervised classification), by finetuning it on each transfer task.

More generally, add transfer learning baselines to the paper. In the scenario of small architectures (4 or 5 conv layers), transfer learning is likely to not work very well, but for larger architectures and more training data it seems to dominate.

Missing ablation: train with only L_{PsCo} instead of the sum of that loss with MoCo’s loss; that is, train only for the contrastive loss that matches each query (example in the current batch) with the corresponding selected ‘support examples’ from the queue (and pushes it away from other support examples that were assigned different pseudolabels).

summary_of_the_review: Overall, I found the idea presented in this paper interesting, the paper well-written and well-organized. I’m generally unconvinced of the value of unsupervised meta-learning, or meta-learning in general as a means for representation learning for the problem of few-shot classification, especially lacking transfer learning baselines. I also found the experimental setup to be not very convincing, as it focuses on very small architectures and the comparisons are against models that are far from state-of-the-art (for the supervised meta-learning baselines). These are the reasons that I recommend rejection.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a novel algorithm for unsupervised few-shot meta-learning that combines contrastive learning with unsupervised task generation to train a few-shot model. They achieve good improvements on state-of-the-art benchmarks as well as cross-domain meta-learning benchmarks.
strength_and_weaknesses: Strength: 
Interesting and novel algorithm.
Good and structured writing and presentation.
Comparing with cross-domain meta-learning.

clarity,_quality,_novelty_and_reproducibility: The authors mention that they will release their code upon acceptance. The pseudo code of algorithm is provided and the details are explained to a good degree.
summary_of_the_review: Interesting and novel paper. Given the strength and reproducibility comments I made, I tend to vote for acceptance of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose Pseudo-supervised Contrast (PsCo), an unsupervised meta-learning framework that solves a few-shot learning problem from the constructed few-shot pseudo-tasks. The key idea of the approach is to improve pseudo-labelling progressively during meta-learning by using current and previous mini-batches which is achieved using momentum network and momentum queue. In particular, few-shot tasks are constructed using the current mini-batch to select queries and the queue of previous mini-batches to select K shots. The label assignment matrix is found using Sinkhorn-Knopp algorithm. The proposed approach is evaluated on in-domain and cross-domain few-shot classification benchmarks and the method is shown to outperform baselines.

strength_and_weaknesses: Strengths:
- The paper is well organized and well written.
- The idea of progressively improving pseudo-labelling strategy  during meta-learning is novel. Inspired by self-supervised learning, authors achieve this using momentum queue and momentum network. Since the shots are selected from the entire dataset, the tasks constructed in the proposed approach are diverse.
- The experiments are comprehensive and the method outperforms other unsupervised meta-learning methods.
- The method scales to large datasets better than existing baselines.

Weaknesses:
- When constructing the pseudo-tasks, the method first randomly selects N samples and treats them as N queries of different N labels. What if selected samples share the same label, but we assume that they have different labels and try to separate them in the feature space during training? This needs to be clarified.
- The method constructs the pseudo-tasks using the queue Q of previous mini-batch samples. Where does the Q come from for the first few batches? It would be more clear if there is more explanation about pseudo-tasks construction.
- The authors compare only to MAML and ProtoNet as supervised meta-learning benchmarks which are not SOA methods so the statements about outperforming supervised meta-learning benchmarks need to be toned-down. Furthermore, for cross-domain learning how would MAML and ProtoNet perform if the representations were pretrained using self-supervised learning before? It is contra-intuitive that unsupervised meta-learning outperform supervised meta-learning and the authors need to be careful about such statements.
- Why do authors compare only to MoCO on Image-Net and not to other self-supervised learning baselines such as SimCLR and RotationNet?
- The authors claim that their method is more efficient than other unsupervised meta-learning baselines but they do not compare running time of their approach and the baselines. Additional information that would be needed is the comparison of running time of their approach and self-supervised learning baselines on the ImageNet.


clarity,_quality,_novelty_and_reproducibility: The paper is of high quality but some aspects need to be improved in the response (see weaknesses).  The idea is original and novel and the extensive experiments demonstrate its effectiveness. The authors need to release the code to make their work reproducible.
summary_of_the_review: Overall, this is a good paper with an interesting approach and well designed experiments. There are some things that should be improved and some details remain unclear or are not convincing. The authors should clarify these points in their response.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present Pseudo-supervised Contrast (PsCo), an unsupervised meta-learning framework that solves a few-shot learning problem from the constructed few-shot pseudo-tasks. The key idea of the approach is to improve pseudo-labelling progressively during meta-learning by using current and previous mini-batches which is achieved using momentum network and momentum queue. 

During the discussion phase, the additional ablation and comparisons against stronger transfer learning baselines, as well as the clarifications to the paper, strengthen the submission significantly.

I agree with the consensus from the reviewers that this work could be published at ICLR.
justification_for_why_not_higher_score: The presented method's scope is still limited, with a limited and fairly "artificial kinda" of setting. I do not think it is at an oral level, which is supposed to interest a larger group of researchers. 
justification_for_why_not_lower_score: The well-presented and clear/neat idea with extensive theoretical and empirical studies, make the work worth a spotlight.
Decision: Accept: notable-top-25%