On Hard Episodes in Meta-Learning
Authors: Samyadeep Basu
Amr Sharaf
Nicolo Fusi
Soheil Feizi
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
few-shot learning

Abstract: Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes. Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner's performance across episodes. Understanding this issue is particularly critical in industrial few-shot settings, where there is limited control over test episodes as they are typically uploaded by end-users. In this paper, we empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets: CIFAR-FS, mini-ImageNet, and tiered-ImageNet. Surprisingly, we observe a wide gap in accuracy of around $50\%$ between the hardest and easiest episodes across all the standard benchmarks and meta-learners. We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forgetting during meta-training.  To address the issue of sub-par performance on hard episodes, we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning.  We find that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.

Under review as a conference paper at ICLR 2022
ON HARD EPISODES IN META-LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes, Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner's performance across episodes. Understanding this issue is particularly critical in industrial few-shot settings, where there is limited control over test episodes as they are typically uploaded by end-users. In this paper, empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets: CIFAR- FS, mini-ImageNNtt, and liered-ImageNet. Surprisingly, we observe a wide gap in accuracy of around 50% between the hardest and easiest episodes across all the standard benchmarks and meta-learners. We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forget- ting during meta-training. To address the issue of sub-par performance on hard episodes, we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning. We find that adversarial train- ing strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.
INTRODUCTION
Humans have a remarkable ability to learn new concepts from very few examples and generalize effectively to unseen tasks. However, standard deep learning approaches still lag behind human capabilities learning from few examples. For large werrparaaeterizzd deep models, learning with general supervision from only a few examples leads to over-fitting and thus poor generaliza- tion. To circumvent this, the paradigm of few-shot learning (Wang al. 2020 Fei-fei et al. 2006
Vinyals et al.) (017)) aims to effectively learn new concepts from very few labeled examples. These learned concepts can generalize well future unseen learning tasks. Several frameworks have been proposed for tackling the few-shot learning scenario: transfer-learning (Dhillon et al.) 2019). self- training (Phoo Hariharan] 2020) and meta-learning (Hospedales et al., 20200 Finn et al. 2017] Snell et al. (017)). Meta-learming in particular aims learn the process of learning from few exam- plcs and has shown remarkable performance across various few-shot benchmarks (Hospedales et al. 2020). In meta-learning, several few-shot tasks (episodes) are sampled from set of base classes and the underlying model is trained to perform well on these tasks leading to improved generalization in learning from only few examples belonging to novel and unseen classes.
Existing meta-learners such as prototypical networks (Snell et al. 2017). MAML (Finn et al. 2017) MetaOptNet (Lee et al. 2019), and R2D2 (Bertinetto al. 2018)) primarily focus on improving prediction performance on average across multiple episodes. However, different episodes have distinct characteristics and hardness which might lead wide variance prediction accuracy across episodes. This problem is much more prevalent in few-shot models deployed in the industry. For example, meta-trained models are often deployed in the cloud for the end-users to use for various tasks such as object recognition, detection, semantic segmentation in computer vision and natural language understanding in NLP. In such settings, the end-users upload their own few-shot dataset to perform predictions new and unseen examples belonging to novel classes. In practice, different users may upload few-shot datasets of varying quality and hardness, leading to a wide disparity in performance across different users. To draw a parallel to the widely accepted experimental protocols in meta-learning, each of the uploaded fhw-shot dataset and the corresponding unseen examples is equivalent to test episode.

---

Under review as a conference paper at ICLR 2022
In this paper, study this issue and investigate how existing state-of-the-arr meta-learners (Snell jet al. 2017] Bertinetto et al. 2018 Lee et al., 2019) perform on episodes of varying hardness. Across three benchmark datasets: CIFAR-FS, mini ImageNet and magredI there is a gap of R 50% in prediction accuracy between the easiest and hardest episodes. To this end, we identify several intriguing properties of hard episodes in meta-learning. For instance, we find that hard episodes are forgotten more easily than casy episodes during meta-training. Episode forgetting occurs when the underlying meta-learner forgets acquired knowledge by the end of the meta-traie training. To improve prediction performance on hard episodes, we investigate and benchmark various adversarial training and curriculum learning strategies that can be used jointly with any ex- isting meta-learner. Empirically, we find that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes. The aim of our paper is not to chase another state-of-the-ar in meta-learning, but to perform a fine-grained inspection of hard episodes across various meta-lcarning methods.
observe that
In summary, we make the following contributions:
We present detailed analysis of episode hardness in meta-learning across few-shot bench- marks and state-of-the-aan meta-leamers. In particular, we study various properties (e.g., semantic characteristics, forgetting) of episode hardness across different meta-learners and architectures.
We find strong connections between episode hardness and catastrophic forgetting in meta- learning. While catastrophic forgetting can occur when meta-training with multiple datasets in sequence (Yap et al. 2020), we observe that forgetting events can occur even when the tasks during meta-traiming are drawn from a single dataset. In particular, we find that hard episodes are easy forget, while easy episodes difficult to forget.
Based on our analysis, we investigate and benchmark different adversarial training and cur- riculum training strategies to augment general purpose meta-training for improving predic- tion performance on hard episodes. Empirically, we find that although there is no one-size- fits-all all solution, adversarial meta-training strategies more powerful when compared to curriculum learning strategies.
2 BACKGROUND AND RELATED WORK
Meta-learning aims to learn an underlying model that can generalize and adapt well to examples from unseen classes by the process of learning to learn. This is primarily achieved by mimicking the
evaluation and adaptation procedure during meta- meta-training. In general, there are three types of meta-
learners: (a) Memory- based methods (Ravi & Larochelle 2017; Munkhdalai et al.) 2018 Santoro
et al. 2016) adapt to novel classes with a memory attached to the meta-learner; (b) Metric-learning
based methods (Snell et al. 2017: (Sung et al. 2017) aim to learn transferable deep representations
which can adapt to unseen classes without any additional fine-tuning; (c) Optimization based meth-
ods (Finn al. 2017 Lee al. 2019 Bertinetto et al., (0018) learn good pre-training initialization for effective transfer to unseen tasks with only few optimization steps. Although the primary fo- cus of our work is meta-learning. we note that other few-shot learning paradigms such as transfer learning (Chen et al. [021]] Sun et al. 2019 Dhillon et al.] 2020) have also shown competitive performance with meta- meaa-nn.
While there has been significant progress in improving the state-of-the-art in meta-learning, very few work investigates the effectiveness of existing meta-learning approaches on episodes of varying hardness. A recent and concurrent work by Arnold et al. (2021) discusses episode difficulty and the impact of random episodic sampling during meta-training. Based on their analysis, Arnold et al. (2021) propose re- reightt optimization framework for meta-training based importance sampling. Although our paper and Arnold et al. (2021) tackle similar problems of episodic hardness, there are several points which distinguishes work:
We provide a much more fine fine-graing grained analysis of episode hardness than Arnold et al. (2021). Arnold et al.] (202I) primarily discuss ferent meta-learners, while we find and investigate a strong connection between episode hardness and catastrophic forgetting.
the transferability of episodes across dif-
2

---

Under review as a conference paper at ICLR 2022
Minimum Accuracy
Maximum Accuracy
300
Accuracy
60-
ConsProtoWWi
Com-R202
ComsPratohwl
Com-RZZZ
Com-ProisNNe
Con--on2
Rés/vi R2D2
Protohitt
R20Z
Protoliet
(g) CITARIFS
(5) miri--magalel
(c] Saesdimagelel
Figure I: Accuracy (y-axis) of existing meta-learners on the hardest and the easiest episode across standard few-shot datasets and meta-learners (x-axis), Note that there is wide gap of 50% between the prediction performance on the easiest and hardest episode.
Arnold et al. 2021 propose loss re-weighting framework for improving the average curacy across episodes. contrary, we investigate the effectiveness of adversarial training (Gong al. 2020) and general curriculum learning techniques in improving the average as well as worst-case prediction performance in meta-learning.
Adversarial meta-learning techniques have previously been used in conjunction with data- augmentation (Ni et al. [202I) to select the augmentation type resulting in the worst-case loss among different augmentation techniques. In this paper, focus how such strategies can be useful in improving the prediction performance of the hard episodes in addition to the average accuracy.
RETHINKING EPISODIC ACCURACY
Existing state-of the-trr meta-learnrss (Finn al. 2017 Lee al. 2019 Snell et al. 2017 Bertinetto et al. 2018) primarily focus on optimizing for the average loss across multiple train- ing episodes or tasks. However, solely the average performance in isolation does give enough insights into how meta-learners perform episodes of varying quality and hardness. Such insights can be particularly crucial to investigate and debug meta-learning models deployed in the wild, where the model can encounter diverse test episodes. In this section, we go beyond the average accuracy across different test episodes and evaluate meta-learners on episodes of varying hardness. First, we discuss how to quantify the hardness of an episode and then discuss the performance of meta-learners on hard episodes.
3.1
WHAT IS A GOOD MEASURE OF EPISODE HARDNESS?
Episodic sampling (i.e. sampling various few-shot tasks from a base dataset) in meta-learning takes place in two steps: (i) First the episode classes are sampled from the class distribution of the base classes e p(Crase); (ii) Next, an episode т is sampled from data distribution conditioned on p(D]c), where ' is the base dataset. An episode consists of set of support examples т, and query examples Ty- In few-shot learning, a n-way, k-shot episode is sampled which results in sampling classes and & support examples per class. Based this, the meta-learning optimization objective can generalized as the following:
the set of sampled classes c:
0 = arg min E,, ((((FFFTTTT)
(1)
where F is the base architecture with 0 as the model parameters and 0' A(0, T.) is the fine- tuning step with the support examples. Different meta-learners have different types of fine-tuning procedures and direct the readers to (Finn et al. 20177; Snell et al. 2017 Bertinetto et al. 2018) for more information on the characteristics A. Based on this definition, we define the hardness of an episode H((T) in terms of the loss incurred on the query examples in episode:
H(T) ((For,
3

---

Under review as a conference paper at ICLR 2022
We choose query loss as metric for hardness because of its inherent simplicity in computation as well as interpretation. In addition, we find a strong negative correlation between the episodic loss and the accuracy (== -0.92 for mini-ImageNet and R -0.89 for tieredImageNet with prototypical networks). This is true for other meta-learners such as R2D2 too (See Appendix A for more de- tails). Alternatielyy, hardness of an episode can also be defined the average log-odds of the query example (Dhillon et al., 2020).
3.2 PERFORMANCE OF MFTA-LEAR ON HARD EPISODES
To understand the effectiveness of meta-learners on episodes of varying hardness, first order the test episodes in decreasing order of their hardness. Then, we evaluate different meta-learners on the easiest and the hardest test episode. Across all the few-shot benchmark datasets such as mini- ImageNet, CIFAR-FS and tieredImageNet, we find in Fig. ( that there is a gap in accuracy of 50% between the episodes with the highest and the lowest loss. Furthermore, we find that the meta-learner and architecture which performs well on average, does not necessarily perform well on the hardest episode. For example, in the case of mini-ImageNet, prototypical networks with a stronger architecture such as ResNet performs better than the 4-layered convolutional architecture on an average, but not on the hard episodes. Morcover in Fig. [ we notice that for ticredImageNet, prototypical networks with ResNet performs the best on easy episodes, while R2D2 with ResNet performs slightly better on the hard episodes. The wide gap in accuracy between hard and easy episodes can be magnified for meta-learners deployed in the wild, where the model might encounter episodes which are significantly hard and diverse in nature. Going forward, we believe that meta- learning studies should not only report average episodic accuracy, but also the prediction accuracy on easy and hard episodes, to present a complete picture of the effectiveness of the meta-learner.
4 VISUAL SEMANTICS OF HARD EPISODES
Based on the observed disparity in performance between hard and easy episodes, a natural question arises: what causes certain episodes to incur high loss? While quantitatively, the hardness of an episode can be defined in terms of the loss incurred on the query examples, it does not provide salient insights into the qualitative nature of hard episodes. In general, we find that episodes incur high loss when there is mismatch in the semantic characteristics between the support and query examples. For example, when a majority of the support examples have objects only one category in the frame and the query examples have multiple objects of different categories surrounding the primary object of interest, the underlying meta-learner often leads to a wrong prediction. Furthermore, when the shape of the objects in the query examples is slightly different from the objects the support examples, the prediction is often erroneous.
(a)
(b)
Support
Query
Figure 2: Semantic Properties of Hard and Easy Episodes: (a) Hard episode, Class: electric-guita -uuitar (b) Easy episode, Class: mixing-bowl; bowl; The images marked in red borders are misclassified query examples.
In Fig. (a), we notice that the query examples that have different objects (e.g., humans) along with the primary object (i.e., guitar) are often misclassified. In Fig. (2)(b), where most of the query examples are classified correctly, we find that the misclassified examples are of two types: (i) query images in which the primary object is occluded with a secondary object; (ii) the shape of the object in the query example is different from the object shapes in the support examples. We provide additional examples of hard episodes in Appendix E.
4

---

Under review as a conference paper at ICLR 2022
100
Erd
90-
63-
Accuracy
Exd
M Mmnna
90-
00
30-
End
Easy Sploade
30-
20
Hard Epiodd
Hord toii
Epochs
Epochs
30
34
Epochs
(o) CIFAR-FS
[0) mini-lmageNet
(c) dieredimageNett
Figure 3: Accuracy of easy and hard episodes (y-axis) during the course of meta-training across different epochs (X-axis); Hard episodes often have a final accuracy less than the maximum aceu- racy reached during meta-training. The meta-learner used is prototypical networks with a Conv-4 backbone trained using episodes with 5-shot, 5-way.
5 HARD EPISODES SUFFER FROM FORGETTING
In supervised learning, catastrophic forgetting is prevalent when tasks from different distributions are learned sequentially (Kirkpatrick et al.) 2016). In such cases, old tasks are forgotten as the model encounters and learns from new tasks. Toneva et al. (2019) has shown that certain examples can be forgotten with high frequency during the course of supervised training even when the samples are drawn from a single dataset. meta-learning, Yap et al. (2020) has shown that in meta-training with tasks from multiple task distributions sequentially. tasks from the old distribution in the sequence can be forgotten as the meta-learner encounters new tasks. However, we observe that even in the case of meta-traing meta-training with tasks drawn from single task distribution, certain types of tasks (episodes) can be forgotten during the course of training. In particular, we analyze the connection between the hardness of episodes and catastrophic forgetting in meta-learning. We track the behaviour of easy and hard training episodes during meta-training and summary find that:
(i) For hard episodes, we notice that the final accuracy at the end of the training drops significantly (x 15% in some cases) from the maximum accuracy obtained during the course of meta- eeta-training.
(i) Hard episodes have more number of forgetting events in comparison to the easy episodes during the course of meta-train trainin.. This behaviour more pronounced in the later stages of meta-training where the accuracies of the easy episodes have already stabilized.
5.1 DEFINING FORGETTING EVENTS IN META LEARNING
During the course of meta-training, the set of sampled training tasks are different in each epoch. In order track forgetting events during meta-training. we first randomly select set of episodes (E = {T] [[] and track their accuracy, throughout the course of meta-training, In our experiments, we set 160 We primarily define two types of forgetting events: (i) Local forgetting event; (ii) Global forgetting event.
Global forgetting events. For given episode, a global forgetting event encountered if the accu- racy of the episode at the end of meta-training is less than the maximum accuracy reached during the course of training by a particular threshold. Formally, given an episode 7 with the maximum accu- racy aCCmex (T) max, (T)) global forgetting event occurs if accmax (r) 2 acCena(T) + a, where a is a threshold and acCena (T) is the accuracy at the end of meta- meta-training. Note that for each episode, global forgetting event can occur only once.
Local forgetting events. For an episode r in the jth epoch of meta vretattriiinng a local forgetting event encountered if the accuracy of the episode the jth epoch (acc; is less than the accuracy the (j 1hth epoch (acc)-1(T)) by particular threshold, denoted by a. Formally, local forgotten event is encountered if acc,(T) + a S aCC)-1(T). Empirically, we study local forgetting events for 0.03 v < 0.15.
5

---

Under review as a conference paper at ICLR 2022
5.2 FORGETTING EVENTS AND HARD EPISODES
Global forgetting events. In Fig. (3). we track the accuracy of the hardest and easiest training episode from each of the few-shot datasets during the entire course of meta-training across differ- ent epochs. Visually, we observe that for the hard episode, the accuracy decreases after a certain point during the course of meta-training. However for the easy episode, this not the case and the accuracy increases till the end of meta-training. To draw more insights, we compute the global forgetting behaviour of different episodes. We first choose the 15 hardest and easiest episodes re- spectively from CIFAR-FS, mini-ImageNet, tieredImageNet and compute their final episodic accu- racy and the maximum episodic accuracy reached during meta-training. In Fig. 4.. we find that for the hard episodes, the gap between the final accuracy and the maximum accuracy reached dur- ing meta- cest-training is significantly larger than the easy episodes. The gap in particular is large for mini-ImageNNet and tieredlmageNet, while for CIFAR-FS the gap is relatively narrow. For exam- ple, in case of the mini-ImageNer dataset, the gap can be R 15%, whereas for the mageee dataset, this gap can be ~ 10%. For CIFAR-FS, this gap hard episodes are globally forgotten in comparison to the casier episodes during the entire course of meta-trant meta-training. We provide further results on global forgetting events Appendix C.
6%. This characteristic shows that
100-
Easp falaades Herd Epiades
100-
Epiodts
Hard Eptodds
Accumnry at and training
20-
60-
60-
43
40-
68
08
300
Maaiiium eccurecy during training
Meximum eccuracy during tnsining
Maximum accuracy during trannnng
(a) CIFARIFS
(b) mini-ImageNatt
(c) lierediiageNee
Figure 4:
Hard episodes have a wider gap between the final accuracy (y-axis) and the maximum accuracy reached during meta-training (x-axis), in comparison to easy episodes. This behaviour is more pronounced for mini- Amaielmaeeeet and tieredImageNet.
Local forgetting events. In order to compute the frequency of local forgetting events for easy and hard episodes across the three few-shot datasets, we first choose 15 easy and hard episodes from each dataset. Across this set of easy and hard episodes, we then compute the number of local forgetting events across various thresholds. In general, across the entire duration of meta-training, we find that the hard episodes have more local forgetting events than the easy episodes. For instance, in Fig. (a), we observe that there is a substantial gap in the number of forgetting events encountered for easy and hard episodes during the entire course of meta-training. Furthermore, to gain more insights about this gap the number of encountered forgetting events, we understand how this gap behaves in the first 20 epochs of meta-training (Fig. (5)-(b) and the last 20 epochs of meta-training (Fig. (c)). In particular, we find that the gap is narrow during the initial stages of meta-training, whereas the gap widens substantially during the later stages.
To summarize, we find that forgetting occurs in meta-learning even when the tasks are drawn from single task distribution. Furthermore, find a strong connection between episode hardness and forgetting events, where we show that hard episodes are more easily forgotten than easy episodes, both in the local and global contexts. In the next section, investigate two meta-training strategies to improve few-shot performance on hard episodes.
6 IMPROVING PERFORMANCE ON HARD EPISODES
In this section, we investigate and benchmark two different meta-training strategies based on adver- sarial training and curriculum learning in order to improve prediction performance on hard episodes. Recent work (Gong et al. 2020 Ni al. 2021) uses adversarial training select episode specific

---

Under review as a conference paper at ICLR 2022
Pagating mant duing entee mete-inehing
Forgating merts durng rited singan of rets-
Forgeting evants bear dagan of meit-mmiinng
CPAR, PS hard
CPAI, het
30
CEAR FC
CFAIL Fi-wayy
sendirogetet hard
Geredinagatte
Number Forgetting
23-
20-
15-
10-
20-
Threshrids
Thresholds
Thrashalds
(9)
29
(c)
Figure 5: (a) Total number of local forgetting events (y-axis) across different thresholds (X-axis)
during the course of meta-training: (b) Total number of local forgetting events during the first 20 epochs of meta-training; (c) Total number of local forgetting events during the last 20 epochs of meta-training. The number of local forgetting events higher for hard episodes in comparison to the easy episodes across different thresholds.
augmentations from a wide pool of diverse data augmentation methods resulting in the highest loss. This loss is then optimized with respect to the model parameters during training. Such data augmen- tation selection strategies have been shown to mitigate over- over-fitting and improve generalization. In work, study how such training strategies can be used select hard episodes and improve pre- diction performance across these episodes. In particular, we investigate two variants of adversarial training as proposed first in Gong et al. (002)) (i) General adversarial training (AT); (ii) Adversarial curriculum training (ACT).
6.1 GENERAL ADVERSARIAL TRAINING
We adopt the adversarial training procedure from (Gong et al. 2020) to first select episodes with a high loss and optimize the meta-learner only with respect to the loss incurred by such hard episodes. Specifically, this involves solving a saddle -oodne optimization problem where the underlying loss is minimized with respect to the parameters of the model and maximized with respect to the input. In particular, during each update of the meta-learner, we first draw batch of episodes each containing support and query examples. Then for each element (episode) of the batch, we additionally sample new episodes with support and query examples belonging to the same classes as the original element of the batch. Then the episode with the highest loss selected amongst the set consisting of the original episode of the batch and the additional sampled episodes. The model parameters are then updated with respect to the gradient of the loss incurred by the selected hard episode. Formally, we solve the following min-max max optimization:
min E, max ((For or, ky) TEg(T)
where 0' = t(0,t,) is the fine-tuning step with the support examples from the selected episodes with the highest loss, g(T) is an operator which samples additional episodes select from for each task 7 in the batch, F is the base learner and & is the loss function. In our experiments, we let g(T) select four additional episodes per sampled episode r. We provide a more detailed description of the parpee-as in Appendix B.I.
6.2 ADVERSARIAL CURRICULUM TRAINING
Curriculum learning (Hacohen & Weinshall 2019 Bengio et al. 2009) aims to mimic the learning process of humans and animals. In particular, humans learn tasks in well defined order; i.e., first learns easy tasks and then gradually moves towards learning more complex tasks. Inspired by this, we modify Eq. (3) and investigate curriculum meta-training strategy. Specifically, during the initial phase of meta-training, we select only a set of easy episodes to learn from, while in the later stages of training, harder episodes are selected. Formally, given the underlying model is meta meta-trained for le| epochs, for the first le/2 epochs, the following loss is optimized:
min E. min ((Fo', tEg(T)
(4)
7

---

Under review as a conference paper at ICLR 2022
Method
C-FAR-ES
mini-ImageNet
mageee
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Conv-ProtoNet
62.6
80.9
52.3
70.4
52.6
71.5
AT
63.3
80.8
52.7
70.8
52.7
71.9
+ ACT
63.1
80.0
52.4
70.3
52.9
71.5
ResNet12-Protooe
71.0
84.0
59.0
74.8
61.8
80.0
AT
71.2
83.9
58.9
74.9
62.9
80.4
ACT
69.3
82.6
56.0
72.8
63.4
79.9
Conv-R2D2
67.6
82.6
55.3
72.4
56.8
75.00
+AT
68.0
82.8
55.7
72.4
57.2
75.00
+ACT
68.3
82.6
55.4
71.3
57.8
74.9
ResNet12-R2D2
70.0
84.8
58.6
75.5
62.8
80.4
+AT
71.0
84.9
58.63
76.0
63.7
81.3
+ACT
69.4
83.2
56.3
74.4
63.5
80.9
Re -MetaOpiNet
70.8
84.0
60.1
77.4
62.9
80.7
+AT
71.2
84.6
60.9
78.2
63.3
80.9
+ACT
70.2
83.9
59.8
77.1
62.1
79.8
Table 1: Average episodic performance of general adversarial training (AT) and adversarial curricu- lum training (ACT) across different meta-learners.
and for the last lel/2 epochs during meta-training, the general min-max adversarial loss described Eq. 33 is used.
EXPERIMENTS
7.1 EXPERIMENTAL SETUP
Datasets. We use three standard few-shot classification datasets for our experiments (i) CIFAR- FS (Bertinetto et al. 2018); (ii) mini-ImageNNt (Vinyals et al., 2016) and (iii) tieredImageNet (Ren sampled from CIFAR-100, whereas mini-ImageNet and tieredImageNet
et al. 2018). CIFAR- are subsets of ImageNet (Deng et al. 2009). We use the class splits from (Lee et al.) 2019) for all the three datasets. Note that mage the splits are constructed from near the root of the ImageNet hierarchy. We provide more details on the datasets in Appendix B.2.
challenging dataset than mini-ImageNer as
Architectures. We primarily use two standard architectures for our experiments: (i) 4-layer convo- lutional network introduced in Vinyals al. (2017); (ii) ResNet-12 (He al., 2015) which is used by Oreshkin et al. (2018) in the few-shot setting. Both of these architectures use batch-normalizatte after every convolutional layer and use ReLU the activation function. Similar architectures for
few-shot learning have been previously used in Lee et al. (2019): Arnold et al. (202T).
Meta-learners. We use prototypical networks (Snell et al. 2017) from the metric learning family of few-shot algorithms. In addition, we use MetaOptNet (Lee et al.
2019) and R2D2 (Bertinetto et al.) 2018 as representative algorithms from the optimization based meta-learning methods.
Model Selection. We use the validation set for each dataset to select the best model. Primarily, we run the validation procedure every iterations on episodes from the validation set to select the best model. Finally, we evaluate on k test episodes from the test set for each dataset.
7.2 DISCUSSION
7.2.1
ADVERSARIAL TRAINING IMPROVES PERFORMANCE ON HARD EPISODES
Across different meta-learners (ProtoNets, R2D2, MetaOptNet) and few-shot datasets (CIFAR-FS. mini-ImageNet, tieredImageNet), we find that the Adversarial Training (AT) strategy works well in general to improve both the average episodic performance well as the episodic performance hard episodes. The comprehensive results for the average performance of meta-learners is presented in Table (T), while the results hard episodes are presented in Table (2. In particular, find
8

---

Under review as a conference paper at ICLR 2022
Method
CIFAR-FS
mini-ImageNe
mageed
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Conv-ProtoNet
28.8
63.0
20.2
53.2
22.2
52.3
AT
29.3
62.9
19.3
53.5
24.6
52.7
+ ACT
28.5
61.3
19.4
52.7
22.1
53.1
ResNet12-Protooe
36.0
66.7
26.2
56.5
29.8
60.4
AT
33.7
66.7
29.2
58.3
31.2
60.9
ACT
29.46
64.9
24.5
54.7
29.6
60.4
Conv-R2D2
32.2
65.3
21.4
55.7
27.0
56.6
+AT
35,3
65.7
24.0
55.6
27.6
56.6
+ACT
32.8
65.5
23.7
54.5
27.8
56.7
ResNet12-R2D2
30.6
68.0
28.0
59.0
34.0
61.5
+AT
35.6
68.0
28.5
60.1
33.8
62.1
+ACT
31.8
65.9
24.4
56.8
30.8
62.0
Re -MetaOpiNet
37.2
69.2
29.5
61.1
30.4
61.5
+AT
37.9
70.0
31.2
62.3
31.5
61.4
+ACT
37.1
69.1
29.2
60.5
30.2
60.9
Performance of general adversarial training (AT) and adversarial curriculum training (ACT) across different meta-learners on hard episodes. We report the mean accuracy over 30 hardest episodes for each meta-learner.
Table 2:
that adversarial meta-training strategies never hurt the average episodic performance and improves over the baseline in majority of our experimental settings. However, we find that the Adversarial Training strategy (AT) leads to large gain over the baseline meta-training strategy for hard episodes. Specifically, we find that the improvements are more significant for the 1-shot case when compared to the 5-shot case. For example, in the 1-shot case, we observe 5% gain for CIFAR-FS with R2-D2 and 3% gain for mini-Imagel ImageNet with prototypical networks. For tieredImage improvement on episodic performance for hard episodes with prototypical networks.
observe = 2%
7.2.2 ADVERSARIAL TRAINING IS BETTER THAN CURRICULUM TRAINING
Although curriculum training leads to better generalization in supervised learning (Hacohen & Wein- shall 2019 |Begioo al. 2009), we find that in meta-learning, the Adversarial Curriculum strategy (ACT) generally performs worse than both the baseline and general Adversarial Training (AT) in majority of our experimental settings. Our observation on curriculum training for meta-learning consistent with the recent work of (Arnold et al.) (021) where they show that curriculum meta- training strategies underperform significantly when compared to the baseline meta-training. We note that although the curriculum formulation (Amold et al. 2021 is different than ours, both methods present easy episodes the meta-learner first followed by hard episodes. While present neg- ative result on curriculum meta-training, we believe that this observation can be used as a guide to develop more advanced and improved curriculum meta-training strategies in the future. In summary, we find that although there is no one-size-fitte -itts-itts-- solution to improve performance on hard episodes, adversarial meta-training strategies perform better than the baseline and curriculum learning.
CONCLUSION
In this paper, we investigated how different meta-learners perform on episodes of varying hardness. We found that there exists wide gap in the performance of the meta-learners between the easiest and the hardest episode across different few-shot datasets. Furthermore, we investigated various facets of the hard episodes and uncovered two major properties: (i) Hard episodes usually have multiple diverse objects in the query examples, whereas the support set primarily consists objects single category, (ii) Hard episodes are forgotten by the underlying meta-learner at higher frequency than the easier episodes. To improve prediction performance over hard episodes, we investigated and benchmarked different training strategies such adversarial training and curriculum learning during meta- training.iiing... We found that adversarial training strategies are beneficial towards meta- training and improve the average episodic performance well performance over hard episodes. Based

---

Under review as a conference paper at ICLR 2022
analysis in this paper, designing more robust meta- meta-learning algorithms that can generalize to hard episodes is an important direction of future work.
REPRODUCIBILITY STATEMENT
Small experimental details are crucial to reproduce results in deep learning, In order to foster the reproducibility of our paper, we provide all the small details of our experiments in the Appendix (Section: Hyperparameters), We would like point out that use the general hyper-parameters in our experimental setup, as used in (Lee et al.) 2019) to ensure reproducibility of the baselines which is particularly important in few-shot learning research. Additionally, considering majority of our paper is on analysis of meta-learners and generating plots, we provide all the necessary details (e.g, number of episodes chosen for generating the plots) available in each corresponding section (See the subsections of Scction
10 ETHICS STATEMENT
In recent times, there has been significant increase in the usage of deep learning models in the in- dustry. Domains of application include NLP, computer vision and healthcare to name a few. Training task and domain specific deep models require large amounts of labeled training data which is not available for variety of use-cases. The paradigm of few-shot learning (or meta-learning) aims to learn generalizable models from only a few training examples. Our paper primarily geared towards understanding the science of such few-shot learning methods. We believe our work can accelerate research on few-shot learning with a specific focus on improving learning from hard episodes. One possible indirect risk is the deployment of our model without following the best practices for col- lecting fair and unbiased datasets for training. Our hope that the positive impacts outweigh the negatives, specifically because standard best-use practices should mitigate most of the risks.
REFERENCES
Sébastien M. R. Arnold, Guneet S. Dhillon, Avinash Ravichandran, and Stefano Soatto. Uniform sampling over episode difficulty, 2021.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, TCML. '09, pp. 41 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN URL
9781605585161. doi: 153344 1553374. 1553380.11338
Luca Bertinetto, João Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. CoRR, abs/1805.08 01888050083666 18136, 2018.
abs/ 20ss/1804 1805.1805.. 08136
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Explor- ing simple meta-learning for few-shot learning, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. WWW.VVVPP..00950664..4
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. CoRR, abs/1990.02729, 2019. URL http:::/axxiv..o////
abs/ 0bs/1900 /9099 02729
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. few-shot image classification, 2020.
baseline for
Li Fei-fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE TRANSAC- TIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 28:2006, 2006.
10

---

Under review as a conference paper at ICLR 2022
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. CoRR, abs/1703.03400, 2017. URL http:://arxxiv.orr////////
03400
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. CoRR, abs/ obss/1804.9945, 2018. URL http::/arxiv.. orw. http:/bb//11100/ httt:p///aaxi..oorr//////1111 09458
ChengYue Gong, Tongzheng Ren, Mao Ye. and Qiang Liu. Maxup: A simple way to im-
prove generalization of neural network training. CoRR, abs/2002.09024, 2020. URL https:
//arxiv.org/abaee oas/ 2.0.2.000220
Guy Hacohen and Daphna Weinshall. On the power of currieulum learning in training deep net- works. CoRR, 0tbs/1904.03622 2019. UK//ctt:://rxxxv.orr//aabs
00..9904.66688 03626
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residulll for image recog-
nition. CoRR, abs/1 0ttp/1512.03330 12.03385, 2015.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. CoRR, abs/1903. 12261, 2019. URL http://arxiv.orgg
12261
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey, Meta-learning in neural networks: A survey. CoRR, abs/2004.05439, 2020. URL http:/tts//exx..orgg////////////
5004.05433
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinskk Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, abs/1612.00776, 2016. URL http::///xxii..crrrtee/e [672600796]
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-lcarning with differentiable convex optimization. CoRR, abs/1994.00758, 2019. URLattp://arxxv.orrg abs/1904.0374
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons, 2018
Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-leaming. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8161-8161. PMLR, 18-24 Jul 2021. URL.
ngs. mgs.mnmmmm... wwwp.ppees//v19/..
n mi21a.2 ht.ml
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task deependentadappiin metric for improved few-shot learning. CoRR, hiss/805.10123 2018. URL http://arxiv.
org/abs/ 1805. 10123
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. CORR, abs/2010.07734, 2020. URL.ittps.//arrxv..orr//a.s/
07734
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. CoRR, habss//70.00366. 2017. URL 1 7.7660 03466
abs/
S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classifica- tion. CoRR,
2018. http////////////avvv..ro//////// wwtp./www..////////////// 00676
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory memory-augmentt augmented neural networks. In Proceedings of the 33rd International Con- ference on International Conference on Machine Learning Volume 48, ICML 16. pp. 1842-1850. JMLR. 2016.
11

---

Under review as a conference paper at ICLR 2022
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. CoRR, abs/1703.05115, 2017. URLhttp: /ttp/aaxivv.../...//eee/eaaa wwt.:oogggga.aoo.. abs/1703.05173
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning, 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang. Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. CoRR, abs/1711.06025, 2017.
URL htt.:ppp./aaxi...o////////111111
06025
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. 2019.
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. CoRR, abs/1606. (0s./000,,4000, 2016. UtL/ptt:://axxi
org/ hos/abs/1600 04080
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning, 2017.
Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. Generalizing from a few exam- ples: A survey few-shot learning. ACM Comput. Sur., 53(3), June 2020. ISSN 0360-0300. doi: 10. 10.1145/3386252. URL https: //t::///doo.oo wttp:..........oooooooo//oo 145/3386251
Pau Ching Yap, Hippolyt Ritter, and David Barber. Bayesian online meta-learning with laplace approximation. CoRR, abs/2005, 00146, 2020. URL https: http//axxvvv./// wwwp:ii..rr/ab//22200
00146
12

---

Under review as a conference paper at ICLR 2022
A LOSS AS A MEASURE OF HARDNESS
minilmageNNe
Conv-ProtoNet
ResNat-ProtoNet
Accuracy
Conv-R222
ResNet-R2D2
30
80
80
70
60
07
0.4
0.6
Losses
Figure 6: Loss (x-axis) vs. Accuracy (y-axis) plot for minilmageNNet.
CIFAR-FS
Corn-PruloNee
ResNet-ProtoNet
100
100
50-
Accuracy
6.
0.2
0.4
8.5
0.8
1.0
Conv-R202
ResNel-R2D2
30
20
68
42
04
066
1.0
Losses
Figure 7: Loss (x-axis) vs. Accuracy (y-axis) plot for CIFAR-FS.
13

---

Under review as a conference paper at ICLR 2022
tiereclmageNet
Conv-ProtoNat
NootNe-rrotoNel
300
60
Accuracy
Canv-R2D2
ResNet-R222
100
90
00
60-
Losses
Figure 8: Loss (x-axis) vs. Accuracy (y-axis) plot for tieredImageNet.
60
R2D2 Conv
ProtoNet Conv
50
ProtoNet ResNet12
R2D2 ResNet12
40
Density
30
10
0.0
0.5
1.0
1.5
2.0
Episode Difficulty (Loss)
Figure 9: Distribution of episode hardness (difficulty) for mini-ImageNNt across different meta- learners. We find that the distribution of hardness approximately follows a gamma distribution. From the density plots, we find that significant fraction of the episodes lie in the range of medium- hard range of difficulty.
14

---

Under review as a conference paper at ICLR 2022
Accuracy of Hard Episodes
Accuracy Episodes
Accuracy Episodes
100
200
AccursOy
66-
40-
20
20
Accuracy of Easy Episodes
Accuracy Easy Episodes
of Easy
130
100
78-
Acuuacy
23
20
Epacis
Eppcche
(a] mini-ImageNet
(b) CIFAR-FS
(c) DeredImageNet
Figure 10: Accuracy of hard and easy episodes (y-axis) during the course of meta-training different epochs (x-axis). Different colors signify different episodes.
B IMPLEMENTATION DETAILS
B.1 HYPERPARAMETERS
Similar to (Le et al. 2019). for the optimizer we use SGD with Nesterov momentum and weight decay. We set the momentum be 0.9 and weight decay to be 0.0005. During meta-training, the underlying meta-learner trained for 60 epochs with each epoch consisting of 1000 episodes. We use 8 episodes per batch during the meta-learner update. The initial learning rate is kept at 0.1 which is then decayed using a learning rate scheduler at epoch 20, 40 and 50 as shown in (Lee et al., 2019). We use 5-way classification for each of our experimental settings. During each iteration of meta-training, we use 6 query examples, while during meta sest-ntssting we use 15 query examples. Following the practice of (Gidaris Komodakis 2018 Qiao al. 2017 Lee al., 2019), we use horizontal flip, random crop, and color jitter data augmentation techniques during meta-training. Across each of the experimental setup, the best model for meta-testing is chosen based on the best validation set performance. In general, few-shot learning be performed in two scenarios: (i) Inductive setting where each test example is evaluated independently; (ii) Transductive setting where the few-shot learner has access to all the test examples. In all experimental settings, follow the inductive setting where each query example is evaluated independently. For general adversarial training method (AT), choose sample 4 additional episodes per sampled episode. For batch of size 8, there would be 32 additional episodes with a total of 40 episodes. 8 episodes with the highest loss selected from this pool for optimizing the meta-learner. For adversarial curriculum learning method (ACT), we choose the set of 8 episodes with the lowest loss for the first 30 epochs of meta-training and choose the set of episodes during the last 30 epochs of meta-traine meta-training. Across all our experimental settings, we use the same number of shots during meta-training and meta-testing match the training and testing conditions.
B.2 FEW-SHOO DATASETS
mini- mini-ImageNett. The mini-ImageNNe dataset is introduced by (Vinyals al.) 2016) and is standard few-shot classification benchmark. This dataset consists of 100 classes which are chosen from 11SV2222012 (Deng et al. 2009). The 100 classes are split into 64, 16, and 20 classes for
15

---

Under review as a conference paper at ICLR 2022
meta-training, meta- validation and meta-testing. Each class has 600 images resulting in total of 60000 images for the entire dataset.
tieredImageNet. The tieredImageeet dataset is larger and more challenging few-shot learning benchmark. It consists 608 classes total which are subclasses of ILSVRC-2012 (Deng et al, 2009). The total number of classes split 351, 97 and 160 classes for meta-training, meta- validation and meta-testing respectively. These classes are selected such that there minimum semantic similarity between the different splits, which makes this dataset challenging.
CIFAR-FS. The CIFAR-FS dataset (Bertinetto et al. 2018) is curated from CIFAR-I00 and com- prises of 100 classes in total with each class consisting of 600 images. The classes are randomly split into 64, 16 and 20 for meta-training, meta-validation and meta-testing respectively.
Amongst the three datasets, the images are of size 84x84 for mini-ImageNet and tieredImageNet, while for CIFAR-FS, the images are of size 32x32.
C GLOBAL FORGETTING EVENTS
To gain further insights into global forgetting events, we select 5 hard and easy episodes from each of the few-shot datasets and track their accuracy during the course of meta-training. We use proto- typical networks the base meta-learner for analysis. We find and validate in Fig. (IO), that our analysis on global forgetting (as shown in the main paper) holds true for more number of hard episodes. Additionally, maximum accuracy reached during meta-training and the accuracy at the end training. We find that for hard episodes on average, the gap between the maximum accuracy reached during train- ing and the final accuracy is large. However, for easy episodes we find this gap to be narrow. This large gap for hard episodes signifies they they undergo catastrophic forgetting during the course of meta-tran mota-training.
also choose 25 hard and easy episodes to compute the mean of their
Global Fargetting Hard Episcdes
Global Forgetting for Easy Episodes
during Neta trani-nnnngg
100
Average. Accuracy
60-
30-
End Mede
mini-imageNel
CIFAR-FS
beredImageNet
mini-Image
CIFAR-FS
tiorodimaggNeet
Datasets
For set of 25 hard and easy episodes, we report the mean of the maximum accuracy reached during the course of meta-training and the accuracy the end of meta-training. For hard episodes, we observe a substantial gap denoting global forgetting.
Figure 11:
D ON TRANSFERABILITY OF EPISODES
Recent and concurrent work such as (Amold et al.) 2021) shows that episode hardness transfer across different architectures and meta-learners. In this section, revisit their analysis and take closer look at the transferability of episodes across various meta-learners and architectures to obtain certain new insights. To this end, compute the Pearson and Spearman correlations of the losses (hardness) incurred by episodes across different architectures and meta-learners for each few-shot dataset.
16

---

Under review as a conference paper at ICLR 2022
0.95
ResNeetMMeaapptteet Re Re
RecNet-Prots, Racheef-222
0.90
Proto, ResluedAeeaaalllii
Spearman Correlation
Conv-R202, Conv-Prot
0.85
0.80
Proto. Ceniv R282
ResNet R2D2, Convv Prago
0.70
0.70
0.75
0.80
0.85
0.90
0.95
Pearson Correlation
Figure 12: Transferability of episode hardness for tieredImage
Restet-MetaOptttet Re NeD-R22
Spearman Correlation
RasNoat-Pato, Rastjrf-
ResNet-Proto, Reslltt- Sastattggttapptiaat
ResNet-Proto, Conv-Preto
Conv-R202 Comv-breta
ResNett-Pooe Comy-12g2
0.70
ResNet-RRD2, Conv-Prato
0.70
0.75
0.80
0.85
0.90
Pearson Correlation
Figure 13: Transferability of episode hardness mini-ImageNNet.
17

---

Under review as a conference paper at ICLR 2022
0.92
ResNet- Metthett-eaa@iiee.. ResNet-R202
0.90
Spearman Correlation
0.88
ResNet- Resleet-Poooyffeseeete2Z222 Restt-RRZ2
0.86
ResNet Aratee--PotttkkeeeeeetMeMeee
0.84
ResNet-Proto
0.82
ResNet-Proco, Conv-R2OZ ResNet A202 Conv-Proto
0.80
0.80
0.82
0.84
0.86
0.88
0.90
0.92
Pearson Correlation
Figure 14:
Transferability of episode hardness for CIFAR-FS.
Across all the three few-shot datasets, we find that the order of transferability across meta-learners and architectures similar. For example, episodes transfer the best consistently between ResNet- MetaOptNet and ResNet-R2D2 across all the datasets, while episodes transfer the worst between ResNet-R2D2 and Conv-Proto. This observation makes intuitive sense as the meta-training strate- gies for both R2D2 and MetaOptNet are more similar for than R2D2 and prototypical networks. Amongst the datasets, we find that transfer of episode hardness between meta-learness and archi- tectures is much more effective for tieredImageNet than mini- Imageee. For example, out of the combinations for transfcrability episodes, 3 of them have a correlation of more than 0.90 in case of mieredImageNet.
18

---

Under review as a conference paper at ICLR 2022
E
MORE ON VISUAL SEMANTICS OF HARD EPISODES
Class: Malamute
Support Examples
Class: Dalmatian
Origina:: Dalmation
Original Dalmstian Prodicted: Malarmto
Origina: Dalmaton
Predicted Malamute
Predicted: Delmstion
Quary Examples
Oricinal: Malamute
Original:
Origina:
Predicted: Daimation
Dalmatan
Figure 15:
Visual semantic properties of hard episodes: We find that if there are closely related fine-grained categories in an episode (c.g, Malamute vs. Dalmatian), the meta-learner (prototypical networks + ResNet-12) often gives a wrong prediction.
In this section, we provide more instances of hard episodes. particular, we find that episodes also become hard when there closely related fine-grained categories in the episode. For exam- ple, in an episode consisting dogs from the classes Malamute and Dalmatian, we notice that the underlying meta-learner often gets confused between the fine-grained categories leading roneous predictions on the query examples. Presence of similar fine-grained categories in episodes therefore one failure mode of existing meta-learning methods.
19

---

Under review as a conference paper at ICLR 2022
F
DISTRIBUTION OF LOSSES FOR CIFAR-FS AND TIEREDIMAGENET
CIFARFF
BeredinegaNaa
Conué Provt--rrtoNets
Convé-ProtoNetts
Conv4 RZD2
35
Conv4-92202
Net-
Density
Density
20
15
10
10
0.2
0.4
0.8
1.0
1.2
3.4
1.6
Loss
Loss
Figure 16: Distribution of query losses for ticredImageNet and CIFAR-FS across Prototypical networks, R2D2 and MetaOptNet (SVM).
G FORGETTING EVENTS FOR R2D2 AND METAOPTNET
mini minimagahNi--Rd..
Ipredir R2D2
M /
I0A2
1 WMV M M
80-
M
80
Hard Epioadte
Easy Epicede
30-
Easy
20
60
Epochs
Epochs
Hard Epiode spisede
Hard Eaisode
50-
Easy Epioade
W
70
50
30
20
40
30
60
20
Epochs
Epochs
Figure 17: Forgetting events for R2D2 and MetaOptNet across mini mini-ImageNe ImageNet and tieredImageNet Fotasets.For R2D2 we use the Conv-4 backbone, while for MetaOptNet we use ResNet- -2.
20

---

Under review as a conference paper at ICLR 2022
H HARDNESS OF OOD EPISODES
Episode hardness and out-of-distribution (OOD) episodes have a strong connection between them. We emphasise that the primary aim of our paper to analyse episode hardness when the episodes during meta-testing are drawn from the same task distribution as meta-training. However, we note that episodes can be hard for a meta-learner when the meta-test episodes are drawn from a different distribution. To analyse the hardness of OOD episodes, we first curate a new meta-testing set where the support examples the new episodes are slightly perturbed. We use the standard corruptions used previously generate the ImageNet-C dataset (Hendryeks & Dietterich, 2019) in order to create new episodes where the support examples are perturbed. Precisely, for every support example, randomly choose from a set of perturbations: (None, Gaussian Noise, Shot Noise, Impulse Noise, Blur, Frost, Fog. Brightness, Contrast, Elastic) This creates a new test episode which is OOD with respect to the episodes encountered during meta-training. From Fig. (T8), find that the query losses incurred for such perturbed or OOD episodes are higher than the in-distribution episodes. Notably, there is overlap in the distribution of losses for the in-distribution and OOD episodes, which might indicate that some of the hard in-distribution episodes are actually outliers or OOD. An end-to-end study of the susceptibility of meta-learners OOD episodes (and also training strategies to improve on OOD episodes) is an interesting research direction and an important direction for future work.
CIFAR-F8
seredimageeet
Conv4 Protolets
Conv4 ProtoNets
episodes ProtoNets)
episodes (Conv4 ProtoNets)
30
30-
25
Density
20
Density
5
5-
0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Loss
Loss
Figure 18: Distribution of query losses for in-distribution episodes (Red) and OOD episodes (Blue) for Prototypical Networks with a Conv-4 backbone.
21


Reviews:
[{"review_id": "kDY4LSJM0QS", "reviewer": "Reviewer_uYEr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n---------\n\n- Interesting analysis on the dynamics of hard vs easy tasks during training.\n- Evaluation on adversarial and curriculum strategies is a plus.\n- The authors recommend practitioners to report the prediction accuracy on easy and hard episodes. This is a sensible suggestion, since this metric is easy to report and can provide a glimpse on the robustness of the method.\n- The paper is well written and easy to follow.\n\nWeaknesses\n----------\n\n- No discussion about out-of-distribution tasks. It seems to me that there is a strong connection between hardness and out-of-distribution samples. Hard episodes could either be out-of-distribution samples or outliers that are on the margin of the decision boundaries. The authors do not dive into this link at all, neither in the related work section nor in the experiments. Note that, there has been substantial work on evaluating meta-learners on out-of-distribution episodes (Jeong et al. 2020, Lee et al. 2019, Wang et al. 2019). Unfortunately this is a strong weakness of the paper, since it completely disregard a line of work which is very relevant for the problem at hand. It is not easy to provide an actionable feedback on this point, since it will require substantial changes to the structure of the paper. I would like to see (i) a contextualization of hard tasks in the framework of out-of-distribution learning, (ii) experiments to evaluate the distance of hard tasks from the dataset distribution and (iii) evaluation of methods specifically designed to address out-of-distribution, e.g. OOD-MAML (Jeong et al. 2020).\n\n- Limited number of conditions. For an experimental paper like this, I would expect to see an evaluation on more conditions. (i) In terms of methods, evaluation on methods developed for out-of-distribution learning (Jeong et al. 2020, Lee et al. 2019, Wang et al. 2019). Some Bayesian methods have also showed robustness against outliers and domain shift (Patacchiola et al. 2020). (ii) In terms of datasets, the Meta-Dataset (Triantafillou et al. 2019) has been recently proposed to evaluate meta-learner on more realistic conditions. The Meta-Dataset could be the perfect benchmark to explore the link between hardness and out-of-distribution samples. (iii) The authors compare adversarial training and curriculum learning strategies, but other strategies could also be tested. For instance, if the hardness of the episodes is known in advance, the loss could be balanced accordingly by using a scalar weight. This strategy would be similar in spirit to the Focal Loss (Lin et al. 2017). \n\n- Nature of hard tasks. The authors qualitatively compare a few hard samples in Section 4 and Appendix E. However, it is difficult to draw some conclusions from this limited number of samples. It would be helpful to see a quantitative comparison based on clear metrics. For instance, in Section 4 the authors argue that wrong predictions are often due to multiple objects of different categories surrounding the primary object of interest. This hypothesis could be verified empirically by using few-shot datasets like ORBIT (Massiceti et al. 2021) that include a set of cluttered and clean images of the same objects. By managing the proportion of clean VS cluttered images in the task it should be possible to see if episodes get more or less hard.\n\nReferences\n----------\n\nArnold, S. M., Dhillon, G. S., Ravichandran, A., & Soatto, S. (2021). Uniform Sampling over Episode Difficulty. arXiv preprint arXiv:2108.01662.\n\nJeong, T., & Kim, H. (2020). OOD-MAML: Meta-learning for few-shot out-of-distribution detection and classification. Advances in Neural Information Processing Systems, 33.\n\nLee, H. B., Lee, H., Na, D., Kim, S., Park, M., Yang, E., & Hwang, S. J. (2019). Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. arXiv preprint arXiv:1905.12917.\n\nLin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).\n\nMassiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L., Harris, M. T., Cutrell, E., ... & Stumpf, S. (2021). ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition. arXiv preprint arXiv:2104.03841.\n\nPatacchiola, M., Turner, J., Crowley, E. J., O'Boyle, M., & Storkey, A. (2020). Bayesian meta-learning for the few-shot setting via deep kernels.\n\nTriantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.\n\nWang, K. C., Vicol, P., Triantafillou, E., Liu, C. C., & Zemel, R. (2019). Out-of-distribution Detection in Few-shot Classification.\n", "summary_of_the_paper": "In this paper the authors present an investigation of the relations between meta-learning methods and hard episodes. They found that (i) there is a large gap in performance between easy and hard episodes, (ii) hard episodes are forgotten more easily than easy episodes, (iii) a comparison between adversarial training and curriculum learning strategies show that the former are more effective in improving the performance gap between hard and easy episodes.", "summary_of_the_review": "In its current form the paper is not ready for acceptance. The main issues are the lack of discussion of out-of-distribution literature and shallow empirical comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iuIBfHPIvUj", "reviewer": "Reviewer_NN1B", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is an interesting point that we should not evaluate meta-learners only on average accuracy at test time. I agree with the authors about the need to also understand worse-case behavior before real-world deployment. The findings that there is a large discrepancy between the ‘end’ and ‘max’ accuracy for some episodes is also interesting, and so is the connection to forgetting that is established in this paper.\n\nHowever, I have a number of concerns about the paper, relating to the motivation of the work, clarity, relationship to related work and experiments. Detailed comments below:\n\n- There is a disconnect between how the work is motivated, and the proposed approach(es). The first paragraph of section 3 criticizes the practice of optimizing the average loss (across episodes) at *training* time (and the proposed approaches address this) but then discusses that this doesn’t give enough insights into the performance when deploying ‘in the wild’ to diverse *test tasks*. There is a disconnect here: If the aim is to gain insights on how meta-learners perform on various *test episodes*, it would be sufficient to simply modify the *evaluation* procedure (e.g. look at the distribution of test performance instead of collapsing to the average). So there seems to be a hidden hypothesis that modifying the *training procedure* will lead to more robust and generally-applicable meta-learners that perform well on diverse *test* tasks? It would be good to state this hypothesis and provide some intuition or evidence for why it’s the case.\n\n- ‘Existing meta-learners … primarily focus on improving prediction performance on average across multiple episodes’. Again, It is not clear whether the criticism relates to the *training objective* (since all training episodes are treated equally without weighted loss usually) or to the *evaluation protocol* (since the evaluation metric is the average over test episodes’ query accuracy).\n\n- ‘We first order the test episodes in decreasing order of hardness’ - how is this done? The provided definition of hardness is specific to a model. Which meta-learner was used for this? Further, ‘we evaluate different meta-learners on the easiest and hardest test episode’: is there a common set of easiest/hardest ones, or does each meta-learner here create its own easy/hard sets? Answering these questions is important for interpreting the results, and unfortunately this information is missing from the paper.\n\n- Fig 1: it would be more informative to show the entire distribution instead of only the min and max test accuracy, as those two points might be extreme outliers.\n\n- Fig 2: which meta-learner was used to obtain the hard and easy episodes visualized here? Are the semantics of hard episodes consistent across meta-learners?\n\n- In Section 5, it’s not clear whether the episodes whose performance is tracked throughout meta-training are training episodes or test episodes. In the first paragraph of Section 5.1: ‘we first randomly select a set of k episodes … and track their accuracy, throughout the course of meta-training’. This sentence does not clarify this. My assumption is that these are *training* episodes in this section (unlike in Section 3), due to the very high accuracy (almost 100%) for easy episodes shown in Figure 3. This should be clarified.\n\n- In section 5.2, the chosen episodes are somehow divided into the hardest and easiest ones. How is this done? Are these the ones that are hardest according to the model *at the last step of training*? If so, it is not too surprising that those don’t perform well at the end of training, by definition of how they were chosen.The large gap between the ‘max’ and ‘end’ for these episodes is still interesting though.\n\n- Which meta-learner was used to produce Figures 3 and 4? Unless I missed it, this isn’t stated anywhere. Further, it would be good to run this analysis for different ones. Do all meta-learners suffer from forgetting hard episodes, or are some more prone to this than others?\n\n- In Fig 5, I was expecting the curves to be monotonically decreasing, since for two thresholds \\alpha_1 < \\alpha_2, all of the local forgetting events that occurred at threshold \\alpha_2 also occur at threshold \\alpha_1. Can you help me understand how come this isn’t the case?\n\n- In explaining general adversarial training: ‘Then, for each element of the batch, we additionally sample a number of similar episodes’. What does ‘similar’ mean here? How are similar episodes selected? e.g. same classes but different support/query examples? This needs to be clarified.\n\n- In Table 2, how are hard episodes defined? I assume in this case they’re hard *test* episodes? Is it the same set of episodes that all models are evaluated on? Which meta-learner was used to determine which episodes are hard?\n\n- It would be useful to also experimentally compare with the re-weighted loss method of (Arnold et al. (2021)) that is mentioned in the paper.\n\n- Another closely-related approach is the ‘Hard task (HT) meta-batch’ approach from [1] - a curriculum learning method that schedules hard tasks in meta-training batches by resampling failure cases. They show in that paper that this consistently improves upon random task sampling, in contrast to the curriculum approach explored here.\n\n- Related to the above point, the curriculum learning approach used here is a very simple one, so perhaps it’s premature to claim its failure as a negative result for curriculum learning. For instance, using only easy episodes in the first half and only hard episodes in the second half might not be the right configuration. How were these hyperparameters chosen? Were other design choices explored?\n\nMinor\n- In the line below Equation 1, ‘\\theta = … is the fine-tuning step …’. I would say that \\theta is the parameters resulting from the fine-tuning step, not the fine-tuning step itself. Same comment applies to the sentence below Equation 3.\n\n- ‘we find a strong correlation between the episodic loss and the accuracy (...)’ - some numbers are provided here without explaining what these are. I’m assuming they’re some correlation co-efficients, but what exactly are these?\n\n\nReferences\n- [1] Meta-Transfer Learning for Few-Shot Learning. Sun et al. CVPR 2019.\n", "summary_of_the_paper": "This paper proposes to deviate from studying the average performance of meta-learners on different tasks, and also report separately their performance on ‘hard’ episodes, measuring ‘worst case’ performance which may be critical for deployment in real-world applications. They provide a definition for episode hardness, and investigate the performance on easy and hard episodes. They find that hard episodes are more often forgotten during meta-training compared to easy episodes. They then propose two strategies to mitigate this, based on adversarial and curriculum training. Their experiments on 3 datasets using 3 meta-learners show that the former sometimes leads to improved few-shot learning performance.", "summary_of_the_review": "Defining and studying hard episodes in meta-learning is an interesting direction that may lead to more robust models that can be more safely deployed in various scenarios in the real world. So this type of study has the potential to be impactful. Certain findings in this paper are also quite interesting, like the large discrepancy between ‘end’ and ‘max’ for certain episodes, and connections to forgetting. However, the paper lacks clarity in certain important areas that makes it difficult to correctly interpret the results. I also found that the paper can be improved in terms of comparisons with related work and more thorough experiments. I believe that another round of review would be required to address these issues, so I do not recommend acceptance at this stage.\n\n\n#######\nEdit: after rebuttal, and reading the other reviews:\n\nI maintain my opinion that the paper, in its current form, falls below the bar for publication. To summarize, albeit some clarity concerns resolved during the rebuttal, I still find the narrative of the paper unclear and the proposed approach not sufficiently well motivated (please refer to my latest response to the authors for details). I also agree with reviewer uYEr that OOD tasks are a natural candidate for ‘hard’ tasks. In fact, an important advantage of defining hardness in this way is that it no longer is dependent on each particular meta-learner. Based on my understanding of the author’s response to my review, the results reported in the tables are computed on different episodes for each meta-learning model, which is problematic for making direct comparisons between them. Finally, I feel that the experimental section is weak and lacks comparisons with relevant previous work. For instance, ‘hard-task meta-batch’ employs a different curriculum learning approach which they find beneficial, whereas the simple curriculum learning proposed here doesn’t work very well, and I felt that they prematurely declare this to be a negative result for curriculum learning. As discussed, running additional experiments to compare against different design choices and related work would strengthen the paper for future revisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SohJHQyOujW", "reviewer": "Reviewer_tHng", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The investigated topic is interesting and relevant to the few-shot classification community, and the writing quality is good. I think going beyond the average-case in episodic evaluation has the potential to reveal useful insights, and I am happy to see work in that direction.\n\nSome experimental details were not clear from the paper's description:\n\n- **[major]** In Section 5, were easy and hard episodes sampled from training or test classes? This is a crucial distinction, and it determines whether I agree or not with the use of the term \"forgetting\". If episodes are sampled from test classes, then I have a hard time seeing how the model could forget what it has never seen throughout training. In that case, is \"forgetting\" simply another word for \"meta-overfitting\"?\n- **[major]** What is the decision process for bolding entries in Tables 1–2? Was a statistical test for the significance of the difference in means performed? What are the 95% confidence intervals on the average query accuracies? Is a difference of 0.4% in query accuracy (e.g. Conv-ProtoNet+AT on mini-ImageNet) statistically significant? This is important, because quite a few entries in both tables show modest improvements over the baselines, and the claimed efficacy of adversarial training hinges on those improvements.\n- I had to scroll down to Appendix B.1 to determine what \"ways\" value was used for Figures 1–5 (5-way), and I could not find what \"shot\" was used.\n- Section 3.2 describes sorting the test episodes in decreasing hardness order, then evaluating various meta-learners on the easiest and hardest test episode. Which meta-learner was used to measure episode hardness? Were different easy and hard episodes chosen for different meta-learners?\n- Which benchmark and meta-learner was used for Figure 2?\n- Which meta-learner was used for Figures 3–5?\n- How are \"similar episodes\" selected for adversarial training?\n\nAdditional questions/comments:\n\n- **[major]** \"For example, in the case of mini-ImageNet, prototypical networks with a stronger architecture such as ResNet performs better than the 4-layered convolutional architecture on an average, but not on the hard episodes.\" Can the authors clarify what numbers are used in drawing this conclusion? Is this based on Figure 1 and Table 1? Table 2 appears to contradict this statement.\n- **[major]** In the strictest sense of the term, MetaOptNet and R2D2 can be characterized as optimization-based meta-learners, but Prototypical Networks could also be implemented with an optimization inner-loop rather than with the analytical solution for Gaussian classifiers, so is the distinction relevant? In all cases, the feature extractor remains frozen during adaptation. Would meta-learners that adapt the feature extractor (e.g. MAML, CNAPs) behave differently with respect to easy and hard episodes? In my opinion, this is an experimental blind spot for the submission.\n- Going beyond the fixed-ways setting (like is done in Meta-Dataset, for instance), are there dynamic range issues arising when comparing query losses for episodes with various numbers of ways? Do the authors prescribe some procedure for normalizing the losses?\n- The term \"forgetting event\" is used before being defined, which I found confusing.\n- In Figure 5, what happens with threshold value 7.5? Why are the numbers of forgetting events collapsing towards zero for that specific threshold value?", "summary_of_the_paper": "The submission investigates and characterizes episode difficulty (as defined in terms of its query loss) in few-shot classification and reports empirical observations on CIFAR-FS, mini-ImageNet, and tiered-ImageNet.\n\nAcross all benchmarks, a wide accuracy gap is observed between the hardest and easiest test episodes for multiple combinations of meta-learner (Prototypical Networks, R2D2) and network architecture (four-layer ConvNet, ResNet). The paper presents support and query images for easy and hard episodes and draws the conclusion that a mismatch in semantic or shape characteristics or in the number of objects between support and query images often cause misclassification.\n\nThe submission then examines how the accuracy of easy and hard episodes evolves over training. It uncovers global and local forgetting events and reports that the latter occur more frequently for hard episodes.\n\nFinally, the paper examines two strategies for taking easy and hard episodes into account: adversarial training, and adversarial curriculum training. Adversarial training is shown to yield modest improvements over (and adversarial curriculum training does not perform significantly better than) regular training.", "summary_of_the_review": "The submission examines a very interesting topic, but falls short on execution in its current state, especially in terms of experimental clarity and generalizability of the reported observations.\n\n---\n\n**Post-rebuttal**:  I appreciate the clarifications and think the paper has made good progress towards better clarity.\n\nAt this point, however, I'm not ready to reconsider my score, mainly because my concern with how results are reported remains unaddressed. In the absence of confidence intervals it's hard for me to determine whether the observed improvements are significant or not.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rt4tYDn6fSH", "reviewer": "Reviewer_ceM1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I would like to appreciate the authors for discussing and comparing against another contemporary work on the same topic (NeurIPS 21) rather than ignoring or hiding the fact that the work is out there. \n\nStrengths\n-------------\nIn this paper, the authors tackle a problem in meta-learning setup that has not received enough recognition -- performance of the meta-learner on easy vs hard tasks in a collection of few-shot tasks (used to train the meta-learner). The authors use a simple quantifiable metric to measure hardness (performance on query samples) and analyze the performance and catastrophic forgetting of various meta-learners on those tasks both during training and at the end. They also discuss two potential improvements which can improve performance on hard episodes - i) adversarial training ii) curriculum learning and show that the first one helps while the second one does not. I found the paper quite enjoyable to read and easy to follow and overall the presentation is nice. The experimental protocol and the ablation studies also complement the narrative well.\n\nWeaknesses\n------------------\nAlthough I enjoyed reading the manuscript, I have to say that I do not think the paper at this point, has sufficient novelty either in terms of new technical insights or new experiments. Via some qualitative examples, the authors try to argue on what type if data samples the meta-learner underperform (which eventually lead to hard episodes). However, I would like to see if there is any formal way to identify if an episode will be hard for a meta-learner or not based on the data-points present in support vs query. The qualitative explanations may not hold good for all few-shot tasks or otherwise may require post-hoc deduction as to why it is a hard episode. I would have also liked to see a comparison of hardness for the same task across different learners e.g. if a task is hard for ProtoNet, is it also hard for MetaOptNet? If not, then what is the reason it is not hard for MetaOptNet but for ProtoNet and these type of experiments would be beneficial. Finally, the authors mention that ACT (adversarial + curriculum) performs better than adversarial only on hard episodes - but from table 2, the way I see it, adversarial works better than adversarial + curriculum more often than not. \n\n\n\n\n", "summary_of_the_paper": "In this paper, the authors analyze the hardness of different episodes for an episodic training regime in the context of a meta-learning training (for few-shot classification tasks). The authors show that even though different meta-learners perform relatively similar on average on these tasks, their performances vary significantly when it comes to harder tasks. Authors also showed that the performance on hard tasks can be somewhat improved using an adversarial training strategy. ", "summary_of_the_review": "The paper is nicely written, easy to follow, tackles and under-explored problem and provides a good analysis, but lacks sufficient novelty in methods or experiments and also does not have a strong justification with respect to some of the findings.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the contrast in performance between easy and hard tasks (episodes) in few-shot image classification and propose mitigating strategies to avoid large performance gaps.

None of the reviewers support the acceptance of this work, despite the authors' detailed rebuttals, with all reviewers confirming their preference for rejection following the author response. Issues raised included lack of clarity of writing and lack of sufficiently convincing experimental results.

I unfortunately could not find a good reason to dissent from the reviewers majority opinion, and therefore also recommend rejection at this time.
Decision: Reject