Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors analyze the hardness of different episodes for an episodic training regime in the context of a meta-learning training (for few-shot classification tasks). The authors show that even though different meta-learners perform relatively similar on average on these tasks, their performances vary significantly when it comes to harder tasks. Authors also showed that the performance on hard tasks can be somewhat improved using an adversarial training strategy. 
main_review: First of all, I would like to appreciate the authors for discussing and comparing against another contemporary work on the same topic (NeurIPS 21) rather than ignoring or hiding the fact that the work is out there. 

Strengths
-------------
In this paper, the authors tackle a problem in meta-learning setup that has not received enough recognition -- performance of the meta-learner on easy vs hard tasks in a collection of few-shot tasks (used to train the meta-learner). The authors use a simple quantifiable metric to measure hardness (performance on query samples) and analyze the performance and catastrophic forgetting of various meta-learners on those tasks both during training and at the end. They also discuss two potential improvements which can improve performance on hard episodes - i) adversarial training ii) curriculum learning and show that the first one helps while the second one does not. I found the paper quite enjoyable to read and easy to follow and overall the presentation is nice. The experimental protocol and the ablation studies also complement the narrative well.

Weaknesses
------------------
Although I enjoyed reading the manuscript, I have to say that I do not think the paper at this point, has sufficient novelty either in terms of new technical insights or new experiments. Via some qualitative examples, the authors try to argue on what type if data samples the meta-learner underperform (which eventually lead to hard episodes). However, I would like to see if there is any formal way to identify if an episode will be hard for a meta-learner or not based on the data-points present in support vs query. The qualitative explanations may not hold good for all few-shot tasks or otherwise may require post-hoc deduction as to why it is a hard episode. I would have also liked to see a comparison of hardness for the same task across different learners e.g. if a task is hard for ProtoNet, is it also hard for MetaOptNet? If not, then what is the reason it is not hard for MetaOptNet but for ProtoNet and these type of experiments would be beneficial. Finally, the authors mention that ACT (adversarial + curriculum) performs better than adversarial only on hard episodes - but from table 2, the way I see it, adversarial works better than adversarial + curriculum more often than not. 





summary_of_the_review: The paper is nicely written, easy to follow, tackles and under-explored problem and provides a good analysis, but lacks sufficient novelty in methods or experiments and also does not have a strong justification with respect to some of the findings.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to deviate from studying the average performance of meta-learners on different tasks, and also report separately their performance on ‘hard’ episodes, measuring ‘worst case’ performance which may be critical for deployment in real-world applications. They provide a definition for episode hardness, and investigate the performance on easy and hard episodes. They find that hard episodes are more often forgotten during meta-training compared to easy episodes. They then propose two strategies to mitigate this, based on adversarial and curriculum training. Their experiments on 3 datasets using 3 meta-learners show that the former sometimes leads to improved few-shot learning performance.
main_review: It is an interesting point that we should not evaluate meta-learners only on average accuracy at test time. I agree with the authors about the need to also understand worse-case behavior before real-world deployment. The findings that there is a large discrepancy between the ‘end’ and ‘max’ accuracy for some episodes is also interesting, and so is the connection to forgetting that is established in this paper.

However, I have a number of concerns about the paper, relating to the motivation of the work, clarity, relationship to related work and experiments. Detailed comments below:

- There is a disconnect between how the work is motivated, and the proposed approach(es). The first paragraph of section 3 criticizes the practice of optimizing the average loss (across episodes) at *training* time (and the proposed approaches address this) but then discusses that this doesn’t give enough insights into the performance when deploying ‘in the wild’ to diverse *test tasks*. There is a disconnect here: If the aim is to gain insights on how meta-learners perform on various *test episodes*, it would be sufficient to simply modify the *evaluation* procedure (e.g. look at the distribution of test performance instead of collapsing to the average). So there seems to be a hidden hypothesis that modifying the *training procedure* will lead to more robust and generally-applicable meta-learners that perform well on diverse *test* tasks? It would be good to state this hypothesis and provide some intuition or evidence for why it’s the case.

- ‘Existing meta-learners … primarily focus on improving prediction performance on average across multiple episodes’. Again, It is not clear whether the criticism relates to the *training objective* (since all training episodes are treated equally without weighted loss usually) or to the *evaluation protocol* (since the evaluation metric is the average over test episodes’ query accuracy).

- ‘We first order the test episodes in decreasing order of hardness’ - how is this done? The provided definition of hardness is specific to a model. Which meta-learner was used for this? Further, ‘we evaluate different meta-learners on the easiest and hardest test episode’: is there a common set of easiest/hardest ones, or does each meta-learner here create its own easy/hard sets? Answering these questions is important for interpreting the results, and unfortunately this information is missing from the paper.

- Fig 1: it would be more informative to show the entire distribution instead of only the min and max test accuracy, as those two points might be extreme outliers.

- Fig 2: which meta-learner was used to obtain the hard and easy episodes visualized here? Are the semantics of hard episodes consistent across meta-learners?

- In Section 5, it’s not clear whether the episodes whose performance is tracked throughout meta-training are training episodes or test episodes. In the first paragraph of Section 5.1: ‘we first randomly select a set of k episodes … and track their accuracy, throughout the course of meta-training’. This sentence does not clarify this. My assumption is that these are *training* episodes in this section (unlike in Section 3), due to the very high accuracy (almost 100%) for easy episodes shown in Figure 3. This should be clarified.

- In section 5.2, the chosen episodes are somehow divided into the hardest and easiest ones. How is this done? Are these the ones that are hardest according to the model *at the last step of training*? If so, it is not too surprising that those don’t perform well at the end of training, by definition of how they were chosen.The large gap between the ‘max’ and ‘end’ for these episodes is still interesting though.

- Which meta-learner was used to produce Figures 3 and 4? Unless I missed it, this isn’t stated anywhere. Further, it would be good to run this analysis for different ones. Do all meta-learners suffer from forgetting hard episodes, or are some more prone to this than others?

- In Fig 5, I was expecting the curves to be monotonically decreasing, since for two thresholds \alpha_1 < \alpha_2, all of the local forgetting events that occurred at threshold \alpha_2 also occur at threshold \alpha_1. Can you help me understand how come this isn’t the case?

- In explaining general adversarial training: ‘Then, for each element of the batch, we additionally sample a number of similar episodes’. What does ‘similar’ mean here? How are similar episodes selected? e.g. same classes but different support/query examples? This needs to be clarified.

- In Table 2, how are hard episodes defined? I assume in this case they’re hard *test* episodes? Is it the same set of episodes that all models are evaluated on? Which meta-learner was used to determine which episodes are hard?

- It would be useful to also experimentally compare with the re-weighted loss method of (Arnold et al. (2021)) that is mentioned in the paper.

- Another closely-related approach is the ‘Hard task (HT) meta-batch’ approach from [1] - a curriculum learning method that schedules hard tasks in meta-training batches by resampling failure cases. They show in that paper that this consistently improves upon random task sampling, in contrast to the curriculum approach explored here.

- Related to the above point, the curriculum learning approach used here is a very simple one, so perhaps it’s premature to claim its failure as a negative result for curriculum learning. For instance, using only easy episodes in the first half and only hard episodes in the second half might not be the right configuration. How were these hyperparameters chosen? Were other design choices explored?

Minor
- In the line below Equation 1, ‘\theta = … is the fine-tuning step …’. I would say that \theta is the parameters resulting from the fine-tuning step, not the fine-tuning step itself. Same comment applies to the sentence below Equation 3.

- ‘we find a strong correlation between the episodic loss and the accuracy (...)’ - some numbers are provided here without explaining what these are. I’m assuming they’re some correlation co-efficients, but what exactly are these?


References
- [1] Meta-Transfer Learning for Few-Shot Learning. Sun et al. CVPR 2019.

summary_of_the_review: Defining and studying hard episodes in meta-learning is an interesting direction that may lead to more robust models that can be more safely deployed in various scenarios in the real world. So this type of study has the potential to be impactful. Certain findings in this paper are also quite interesting, like the large discrepancy between ‘end’ and ‘max’ for certain episodes, and connections to forgetting. However, the paper lacks clarity in certain important areas that makes it difficult to correctly interpret the results. I also found that the paper can be improved in terms of comparisons with related work and more thorough experiments. I believe that another round of review would be required to address these issues, so I do not recommend acceptance at this stage.


#######
Edit: after rebuttal, and reading the other reviews:

I maintain my opinion that the paper, in its current form, falls below the bar for publication. To summarize, albeit some clarity concerns resolved during the rebuttal, I still find the narrative of the paper unclear and the proposed approach not sufficiently well motivated (please refer to my latest response to the authors for details). I also agree with reviewer uYEr that OOD tasks are a natural candidate for ‘hard’ tasks. In fact, an important advantage of defining hardness in this way is that it no longer is dependent on each particular meta-learner. Based on my understanding of the author’s response to my review, the results reported in the tables are computed on different episodes for each meta-learning model, which is problematic for making direct comparisons between them. Finally, I feel that the experimental section is weak and lacks comparisons with relevant previous work. For instance, ‘hard-task meta-batch’ employs a different curriculum learning approach which they find beneficial, whereas the simple curriculum learning proposed here doesn’t work very well, and I felt that they prematurely declare this to be a negative result for curriculum learning. As discussed, running additional experiments to compare against different design choices and related work would strengthen the paper for future revisions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper the authors present an investigation of the relations between meta-learning methods and hard episodes. They found that (i) there is a large gap in performance between easy and hard episodes, (ii) hard episodes are forgotten more easily than easy episodes, (iii) a comparison between adversarial training and curriculum learning strategies show that the former are more effective in improving the performance gap between hard and easy episodes.
main_review: 
Strengths
---------

- Interesting analysis on the dynamics of hard vs easy tasks during training.
- Evaluation on adversarial and curriculum strategies is a plus.
- The authors recommend practitioners to report the prediction accuracy on easy and hard episodes. This is a sensible suggestion, since this metric is easy to report and can provide a glimpse on the robustness of the method.
- The paper is well written and easy to follow.

Weaknesses
----------

- No discussion about out-of-distribution tasks. It seems to me that there is a strong connection between hardness and out-of-distribution samples. Hard episodes could either be out-of-distribution samples or outliers that are on the margin of the decision boundaries. The authors do not dive into this link at all, neither in the related work section nor in the experiments. Note that, there has been substantial work on evaluating meta-learners on out-of-distribution episodes (Jeong et al. 2020, Lee et al. 2019, Wang et al. 2019). Unfortunately this is a strong weakness of the paper, since it completely disregard a line of work which is very relevant for the problem at hand. It is not easy to provide an actionable feedback on this point, since it will require substantial changes to the structure of the paper. I would like to see (i) a contextualization of hard tasks in the framework of out-of-distribution learning, (ii) experiments to evaluate the distance of hard tasks from the dataset distribution and (iii) evaluation of methods specifically designed to address out-of-distribution, e.g. OOD-MAML (Jeong et al. 2020).

- Limited number of conditions. For an experimental paper like this, I would expect to see an evaluation on more conditions. (i) In terms of methods, evaluation on methods developed for out-of-distribution learning (Jeong et al. 2020, Lee et al. 2019, Wang et al. 2019). Some Bayesian methods have also showed robustness against outliers and domain shift (Patacchiola et al. 2020). (ii) In terms of datasets, the Meta-Dataset (Triantafillou et al. 2019) has been recently proposed to evaluate meta-learner on more realistic conditions. The Meta-Dataset could be the perfect benchmark to explore the link between hardness and out-of-distribution samples. (iii) The authors compare adversarial training and curriculum learning strategies, but other strategies could also be tested. For instance, if the hardness of the episodes is known in advance, the loss could be balanced accordingly by using a scalar weight. This strategy would be similar in spirit to the Focal Loss (Lin et al. 2017). 

- Nature of hard tasks. The authors qualitatively compare a few hard samples in Section 4 and Appendix E. However, it is difficult to draw some conclusions from this limited number of samples. It would be helpful to see a quantitative comparison based on clear metrics. For instance, in Section 4 the authors argue that wrong predictions are often due to multiple objects of different categories surrounding the primary object of interest. This hypothesis could be verified empirically by using few-shot datasets like ORBIT (Massiceti et al. 2021) that include a set of cluttered and clean images of the same objects. By managing the proportion of clean VS cluttered images in the task it should be possible to see if episodes get more or less hard.

References
----------

Arnold, S. M., Dhillon, G. S., Ravichandran, A., & Soatto, S. (2021). Uniform Sampling over Episode Difficulty. arXiv preprint arXiv:2108.01662.

Jeong, T., & Kim, H. (2020). OOD-MAML: Meta-learning for few-shot out-of-distribution detection and classification. Advances in Neural Information Processing Systems, 33.

Lee, H. B., Lee, H., Na, D., Kim, S., Park, M., Yang, E., & Hwang, S. J. (2019). Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. arXiv preprint arXiv:1905.12917.

Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).

Massiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L., Harris, M. T., Cutrell, E., ... & Stumpf, S. (2021). ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition. arXiv preprint arXiv:2104.03841.

Patacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. (2020). Bayesian meta-learning for the few-shot setting via deep kernels.

Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.

Wang, K. C., Vicol, P., Triantafillou, E., Liu, C. C., & Zemel, R. (2019). Out-of-distribution Detection in Few-shot Classification.

summary_of_the_review: In its current form the paper is not ready for acceptance. The main issues are the lack of discussion of out-of-distribution literature and shallow empirical comparisons.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission investigates and characterizes episode difficulty (as defined in terms of its query loss) in few-shot classification and reports empirical observations on CIFAR-FS, mini-ImageNet, and tiered-ImageNet.

Across all benchmarks, a wide accuracy gap is observed between the hardest and easiest test episodes for multiple combinations of meta-learner (Prototypical Networks, R2D2) and network architecture (four-layer ConvNet, ResNet). The paper presents support and query images for easy and hard episodes and draws the conclusion that a mismatch in semantic or shape characteristics or in the number of objects between support and query images often cause misclassification.

The submission then examines how the accuracy of easy and hard episodes evolves over training. It uncovers global and local forgetting events and reports that the latter occur more frequently for hard episodes.

Finally, the paper examines two strategies for taking easy and hard episodes into account: adversarial training, and adversarial curriculum training. Adversarial training is shown to yield modest improvements over (and adversarial curriculum training does not perform significantly better than) regular training.
main_review: The investigated topic is interesting and relevant to the few-shot classification community, and the writing quality is good. I think going beyond the average-case in episodic evaluation has the potential to reveal useful insights, and I am happy to see work in that direction.

Some experimental details were not clear from the paper's description:

- **[major]** In Section 5, were easy and hard episodes sampled from training or test classes? This is a crucial distinction, and it determines whether I agree or not with the use of the term "forgetting". If episodes are sampled from test classes, then I have a hard time seeing how the model could forget what it has never seen throughout training. In that case, is "forgetting" simply another word for "meta-overfitting"?
- **[major]** What is the decision process for bolding entries in Tables 1–2? Was a statistical test for the significance of the difference in means performed? What are the 95% confidence intervals on the average query accuracies? Is a difference of 0.4% in query accuracy (e.g. Conv-ProtoNet+AT on mini-ImageNet) statistically significant? This is important, because quite a few entries in both tables show modest improvements over the baselines, and the claimed efficacy of adversarial training hinges on those improvements.
- I had to scroll down to Appendix B.1 to determine what "ways" value was used for Figures 1–5 (5-way), and I could not find what "shot" was used.
- Section 3.2 describes sorting the test episodes in decreasing hardness order, then evaluating various meta-learners on the easiest and hardest test episode. Which meta-learner was used to measure episode hardness? Were different easy and hard episodes chosen for different meta-learners?
- Which benchmark and meta-learner was used for Figure 2?
- Which meta-learner was used for Figures 3–5?
- How are "similar episodes" selected for adversarial training?

Additional questions/comments:

- **[major]** "For example, in the case of mini-ImageNet, prototypical networks with a stronger architecture such as ResNet performs better than the 4-layered convolutional architecture on an average, but not on the hard episodes." Can the authors clarify what numbers are used in drawing this conclusion? Is this based on Figure 1 and Table 1? Table 2 appears to contradict this statement.
- **[major]** In the strictest sense of the term, MetaOptNet and R2D2 can be characterized as optimization-based meta-learners, but Prototypical Networks could also be implemented with an optimization inner-loop rather than with the analytical solution for Gaussian classifiers, so is the distinction relevant? In all cases, the feature extractor remains frozen during adaptation. Would meta-learners that adapt the feature extractor (e.g. MAML, CNAPs) behave differently with respect to easy and hard episodes? In my opinion, this is an experimental blind spot for the submission.
- Going beyond the fixed-ways setting (like is done in Meta-Dataset, for instance), are there dynamic range issues arising when comparing query losses for episodes with various numbers of ways? Do the authors prescribe some procedure for normalizing the losses?
- The term "forgetting event" is used before being defined, which I found confusing.
- In Figure 5, what happens with threshold value 7.5? Why are the numbers of forgetting events collapsing towards zero for that specific threshold value?
summary_of_the_review: The submission examines a very interesting topic, but falls short on execution in its current state, especially in terms of experimental clarity and generalizability of the reported observations.

---

**Post-rebuttal**:  I appreciate the clarifications and think the paper has made good progress towards better clarity.

At this point, however, I'm not ready to reconsider my score, mainly because my concern with how results are reported remains unaddressed. In the absence of confidence intervals it's hard for me to determine whether the observed improvements are significant or not.
review_title: null

Meta-review: This paper explores the contrast in performance between easy and hard tasks (episodes) in few-shot image classification and propose mitigating strategies to avoid large performance gaps.

None of the reviewers support the acceptance of this work, despite the authors' detailed rebuttals, with all reviewers confirming their preference for rejection following the author response. Issues raised included lack of clarity of writing and lack of sufficiently convincing experimental results.

I unfortunately could not find a good reason to dissent from the reviewers majority opinion, and therefore also recommend rejection at this time.
Decision: Reject