Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper empirically finds that popular meta-learning algorithms are very sensitive to the selections of the support images. The author shows that six meta-learning algorithms can achieve testing accuracy within a wide accuracy (worst case accuracy is extremely poor) given different support samples. They also find that this problem cannot be trivially solved by adversarial training.

main_review: The paper is easy to follow and the topic and findings are interesting to me. The robustness to the support data seems important to areas such as medical data and autonomous driving. It’s interesting to see the robustness of large support samples (10-shot) is not robust enough to testing samples. However, the authors fail to get a solution to this problem and thus their analysis part is not enough to address how important the problem is.

limitations_and_societal_impact: 1) I wonder whether this sensitivity to support data is only to meta-learning or to the whole few-shot learning. For example, will the pre-trained feature embedding from regular supervised training or self-supervised learning suffer from the same problem?

2) I think we need more analysis for the worst cases. For example, are the worst cases the same across different meta-learners?  How do the feature representations perform for the worst case images, are they on the boundary of classification? 

3) I am not sure how important the question is, especially for 1-shot cases. As long as your model is not 100% accurate for meta-test tasks, you can always find the misclassification cases, and set them to be support data in which case I don’t think it’s possible to get a good accuracy.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper studies the sensitivity of meta-learning to the support data. The paper presents an algorithm for finding the best and worst support examples relative to the test accuracy, and demonstrates empirically the sensitivity of several meta-learning methods to the support data. The paper explored using  adversarial robustness strategies, however, while these methods seem to work on the training tasks, they fail to generalize on new test tasks. The paper explores the sensitivity of meta-learning algorithm to the support data from the margin perspective, arguing that higher margins and interclass tightness leads to more robust meta-learning methods.
main_review: ## Originality

- The paper presents a greedy search algorithm (Algorithm 1) to study the sensitivity of meta-learning algorithms to the support data. The empirical analysis presented in the paper to quantify this sensitivity is novel and useful for the meta-learning practitioners to be informed about.However, it’s limited only to few-shot classification tasks, and doesn’t include other meta-learning applications  for instance in language and speech. 

- Related work was discussed in the introduction and background section, some missing references include: “ Setlur, Amrith, Oscar Li, and Virginia Smith. "Is Support Set Diversity Necessary for Meta-Learning?." arXiv preprint arXiv:2011.14048 (2020).” and the references within.

## Quality

- The submission is technically sound, and the claims are well supported empirically. The paper provides a simple theoretical analysis for the effect of increasing the classification margin on robustness.

- The proposed greedy search method is appropriate and succeeds in finding the best and worst support training point, despite lacking the guarantees for global convergence. 

- Evaluation was performed only on computer vision tasks.

## Clarity

- The submission is clearly written, well organized, and easy to follow. 
- Algorithm 1 could be illustrated better with a figure or a running example.

## Significance

- The analysis is useful for the meta-learning community to be informed about, however, no direct solution for the problem was presented in the paper. The paper presents the insight that maximizing the classification margin and minimizing inter-class variations would improve robustness. Adversarial training didn’t improve the results significantly on new test tasks.
limitations_and_societal_impact: - Empirical analysis was only conducted in a few-shot text classification setup.
- Actionable solutions for mitigating the sensitivity of the meta-learning methods to support data wasn’t presented.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is studying the performance of several meta-learning approaches depending on the support images used for meta-test tasks. 
First, the authors propose a simple greedy algorithm that select for each class in a given task, the image giving the worst/ the best performance. With this algorithm, they show that popular meta-learning approaches have a very high variance in their performance and that the worst case scenario is often worse than a random guess. The authors argue that the worst tasks found by their algorithm are not artifacts in the distribution, and can realistically appear. 
Then, they apply their algorithm to train the meta-learning methods in an adversarial way, by learning on the worst cases found by their algorithm. Even though the adversarial training improves the worst-case performance on the meta-training tasks, there are no improvements on the worst-case accuracy on meta-testing tasks.
Finally, the authors argue that the worst-case performance are better when the task embeddings are sufficiently separated. 
main_review: __**Originality:**__  The paper presents an interesting approach to find worst/best case performance and a novel evaluation of performance for meta-learning methods. The evaluations are similar to some experiments in semi-supervised setting [1], but it was never investigated in the case of meta-learning.  
 This paper shows actual evidence that there is a high variance when evaluating meta-learners. It is the first time I see an actual computation of worst and best performance, even though the problem was already raised [2].

__**Quality:**__  The paper propose a simple but effective greedy algorithm for computing worst/best case performance. If I understand correctly, the algorithm searches from *all* possible support images for a given class. I think it would be interesting to have the computing time of this algorithm, since it seems quite heavy to compute the performance for all possible images.   
Using an adversarial training scheme with the proposed algorithm is interesting, even though it did not help to improve the worst case performance. It shows that reducing the variance in performance is complicated. 

__**Clarity:**__  The main message and the writing of the paper are clear. The paper is also well-structured.  
However, I think the discussions about the experiments could be improved, particularly section 3.2:
- Each point in the section could benefit from their own conclusion to link it to the main point that worst support examples are not artifacts. Currently, the reader must draw the conclusion from each point themselves. 
- Figures 2 and 3 are hard to understand because they appear before their introductions and the explanations of the experiments in the text. This makes it difficult to understand the conclusions that we have to draw from them. 
- The meaning of an *iteration* of the algorithm should be clarified. If I understand correctly, an iteration means going through all support images for each class. I had to look at the code to really understand the algorithm.

__**Significance:**__ I think the paper is valuable for practitioners. It brings concrete evidence that there is high variance in performance of meta-learning methods. This is currently a significant problem in few-shot learning. The adversarial training experiment is a naive idea to try to alleviate this problem. Even though the authors show that it doesn't work, it could help future work.

__**Minor Remarks:**__

- What projection method was used ? I assumed it was t-SNE.
- Experimental details for ProtoNets and R2D2 are missing in the appendix. From the performance, I assumed the authors used a ResNet-12 backbone, similarly to MetaOptNet.
- Why is there so much variance for MAML and MC on average accuracy ? The variance is a lot higher than for other methods, and I never saw that much variance for these algorithms. I assumed it was a typo when reporting results.

[1]: Sohn et al., *FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.* NeurIPS 2020  
[2]: Dhillon et al., *A baseline for few-shot image classification.* ICLR 2020
limitations_and_societal_impact: The authors did not discuss potential negative impacts of their work. Even though the work presented here is not fully theoretical, it is high upstream of the applications. To study the possible negative impacts of this kind of work is to study the negative impacts of the whole field, since it is discussing the way we are evaluating meta-learning models. As any other field in machine learning, meta-learning could also be used for harmful applications, such as quick racial profiling of a user.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work observes that meta-learners are incredibly sensitive to support data for fine-tuning and explains this phenomenon in terms of margins.
main_review: 1)  Better understanding exactly how meta-learning works and understanding its properties is an interesting direction and is important given the popularity and success of meta-learning.

2)  [1] examines sensitivity to support data (during training) from an adversarial view in which the worst-case support data is not natural images but adversarially perturbed ones.  It would be good to relate your work to the findings from this paper.

3)  [2] studies the sensitivity of meta-learning to the support set during training rather than testing in Section 3.  This work should also be considered in your paper.

[1] Attacking Few-Shot Classifiers with Adversarial Support Sets
[2] Data Augmentation for Meta-Learning

If the authors relate their work to the above papers and justify the novelty and significance of their conclusions in the context of these works, I am open to raising my score.

[Update: increased score to reflect that my comments were addressed]
limitations_and_societal_impact: The authors do adequately address the limitations and impact of their work.
review_title: null

Meta-review: The submission investigates the extent to which few-shot classification algorithms are sensitive to the selection of support images. It introduces a greedy algorithm for finding the worst/best support sets for a learning episode and reports that the performance of all six evaluated approaches drastically varies as a function of the support set composition. When adversarial training is employed to increase robustness, the authors report success for training episodes, but crucially these benefits do not translate to test episodes. Finally, the submission presents theory that suggests that robustness in metric-based few-shot learners could be obtained by encouraging inter-class separation and tighter intra-class clusters.

Reviewers appreciated the paper's writing clarity and found the problem setting to be interesting and of practical significance to the community. The authors provided the clarifications requested by the reviewers. Some reviewers pointed out that the authors failed to propose an effective solution, to which the authors rightfully replied that "problem-reporting" submissions can still make a substantial contribution to the field, citing Szegedy et al.'s paper on adversarial examples as evidence. Some reviewers noted that the paper's impact could be increased by considering domains beyond image classification, but ultimately found that it meets the bar for acceptance as it stands.

I therefore recommend acceptance.
Decision: Accept (Poster)