On Hard Episodes in Meta-Learning

Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes . Different episodes , however , may vary in hardness and quality leading to a wide gap in the meta-learner ’ s performance across episodes . Understanding this issue is particularly critical in industrial few-shot settings , where there is limited control over test episodes as they are typically uploaded by end-users . In this paper , we empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets : CIFARFS , mini-ImageNet , and tiered-ImageNet . Surprisingly , we observe a wide gap in accuracy of around 50 % between the hardest and easiest episodes across all the standard benchmarks and meta-learners . We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forgetting during meta-training . To address the issue of sub-par performance on hard episodes , we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning . We find that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes . 1 INTRODUCTION . Humans have a remarkable ability to learn new concepts from very few examples and generalize effectively to unseen tasks . However , standard deep learning approaches still lag behind human capabilities in learning from few examples . For large over-parameterized deep models , learning with general supervision from only a few examples leads to over-fitting and thus poor generalization . To circumvent this , the paradigm of few-shot learning ( Wang et al. , 2020 ; Fei-fei et al. , 2006 ; Vinyals et al. , 2017 ) aims to effectively learn new concepts from very few labeled examples . These learned concepts can generalize well to future unseen learning tasks . Several frameworks have been proposed for tackling the few-shot learning scenario : transfer-learning ( Dhillon et al. , 2019 ) , selftraining ( Phoo & Hariharan , 2020 ) and meta-learning ( Hospedales et al. , 2020 ; Finn et al. , 2017 ; Snell et al. , 2017 ) . Meta-learning in particular aims to learn the process of learning from few examples and has shown remarkable performance across various few-shot benchmarks ( Hospedales et al. , 2020 ) . In meta-learning , several few-shot tasks ( episodes ) are sampled from a set of base classes and the underlying model is trained to perform well on these tasks leading to improved generalization in learning from only few examples belonging to novel and unseen classes . Existing meta-learners such as prototypical networks ( Snell et al. , 2017 ) , MAML ( Finn et al. , 2017 ) , MetaOptNet ( Lee et al. , 2019 ) , and R2D2 ( Bertinetto et al. , 2018 ) primarily focus on improving prediction performance on average across multiple episodes . However , different episodes have distinct characteristics and hardness which might lead to a wide variance in prediction accuracy across episodes . This problem is much more prevalent in few-shot models deployed in the industry . For example , meta-trained models are often deployed in the cloud for the end-users to use for various tasks such as object recognition , detection , semantic segmentation in computer vision and natural language understanding in NLP . In such settings , the end-users upload their own few-shot dataset to perform predictions on new and unseen examples belonging to novel classes . In practice , different users may upload few-shot datasets of varying quality and hardness , leading to a wide disparity in performance across different users . To draw a parallel to the widely accepted experimental protocols in meta-learning , each of the uploaded few-shot dataset and the corresponding unseen examples is equivalent to a test episode . In this paper , we study this issue and investigate how existing state-of-the-art meta-learners ( Snell et al. , 2017 ; Bertinetto et al. , 2018 ; Lee et al. , 2019 ) perform on episodes of varying hardness . Across three benchmark datasets : CIFAR-FS , mini-ImageNet , and tieredImageNet , we observe that there is a gap of ≈ 50 % in prediction accuracy between the easiest and hardest episodes . To this end , we identify several intriguing properties of hard episodes in meta-learning . For instance , we find that hard episodes are forgotten more easily than easy episodes during meta-training . Episode forgetting occurs when the underlying meta-learner forgets acquired knowledge by the end of the meta-training . To improve prediction performance on hard episodes , we investigate and benchmark various adversarial training and curriculum learning strategies that can be used jointly with any existing meta-learner . Empirically , we find that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes . The aim of our paper is not to chase another state-of-the-art in meta-learning , but to perform a fine-grained inspection of hard episodes across various meta-learning methods . In summary , we make the following contributions : • We present a detailed analysis of episode hardness in meta-learning across few-shot benchmarks and state-of-the-art meta-learners . In particular , we study various properties ( e.g. , semantic characteristics , forgetting ) of episode hardness across different meta-learners and architectures . • We find strong connections between episode hardness and catastrophic forgetting in metalearning . While catastrophic forgetting can occur when meta-training with multiple datasets in sequence ( Yap et al. , 2020 ) , we observe that forgetting events can occur even when the tasks during meta-training are drawn from a single dataset . In particular , we find that hard episodes are easy to forget , while easy episodes are difficult to forget . • Based on our analysis , we investigate and benchmark different adversarial training and curriculum training strategies to augment general purpose meta-training for improving prediction performance on hard episodes . Empirically , we find that although there is no one-sizefits-all solution , adversarial meta-training strategies are more powerful when compared to curriculum learning strategies . 2 BACKGROUND AND RELATED WORK . Meta-learning aims to learn an underlying model that can generalize and adapt well to examples from unseen classes by the process of learning to learn . This is primarily achieved by mimicking the evaluation and adaptation procedure during meta-training . In general , there are three types of metalearners : ( a ) Memory-based methods ( Ravi & Larochelle , 2017 ; Munkhdalai et al. , 2018 ; Santoro et al. , 2016 ) adapt to novel classes with a memory attached to the meta-learner ; ( b ) Metric-learning based methods ( Snell et al. , 2017 ; Sung et al. , 2017 ) aim to learn transferable deep representations which can adapt to unseen classes without any additional fine-tuning ; ( c ) Optimization based methods ( Finn et al. , 2017 ; Lee et al. , 2019 ; Bertinetto et al. , 2018 ) learn a good pre-training initialization for effective transfer to unseen tasks with only a few optimization steps . Although the primary focus of our work is meta-learning , we note that other few-shot learning paradigms such as transfer learning ( Chen et al. , 2021 ; Sun et al. , 2019 ; Dhillon et al. , 2020 ) have also shown competitive performance with meta-learning . While there has been a significant progress in improving the state-of-the-art in meta-learning , very few work investigates the effectiveness of existing meta-learning approaches on episodes of varying hardness . A recent and concurrent work by Arnold et al . ( 2021 ) discusses episode difficulty and the impact of random episodic sampling during meta-training . Based on their analysis , Arnold et al . ( 2021 ) propose a re-weighted optimization framework for meta-training based on importance sampling . Although our paper and Arnold et al . ( 2021 ) tackle similar problems of episodic hardness , there are several points which distinguishes our work : • We provide a much more fine-grained analysis of episode hardness than Arnold et al . ( 2021 ) . Arnold et al . ( 2021 ) primarily discuss the transferability of episodes across different meta-learners , while we find and investigate a strong connection between episode hardness and catastrophic forgetting . • Arnold et al . ( 2021 ) propose a loss re-weighting framework for improving the average accuracy across episodes . In contrary , we investigate the effectiveness of adversarial training ( Gong et al. , 2020 ) and general curriculum learning techniques in improving the average as well as worst-case prediction performance in meta-learning . Adversarial meta-learning techniques have previously been used in conjunction with dataaugmentation ( Ni et al. , 2021 ) to select the augmentation type resulting in the worst-case loss among different augmentation techniques . In this paper , we focus on how such strategies can be useful in improving the prediction performance of the hard episodes in addition to the average accuracy . 3 RETHINKING EPISODIC ACCURACY . Existing state-of-the-art meta-learners ( Finn et al. , 2017 ; Lee et al. , 2019 ; Snell et al. , 2017 ; Bertinetto et al. , 2018 ) primarily focus on optimizing for the average loss across multiple training episodes or tasks . However , solely the average performance in isolation does not give enough insights into how meta-learners perform on episodes of varying quality and hardness . Such insights can be particularly crucial to investigate and debug meta-learning models deployed in the wild , where the model can encounter diverse test episodes . In this section , we go beyond the average accuracy across different test episodes and evaluate meta-learners on episodes of varying hardness . First , we discuss how to quantify the hardness of an episode and then discuss the performance of meta-learners on hard episodes . 3.1 WHAT IS A GOOD MEASURE OF EPISODE HARDNESS ? . Episodic sampling ( i.e . sampling various few-shot tasks from a base dataset ) in meta-learning takes place in two steps : ( i ) First the episode classes are sampled from the class distribution of the base classes : c ∼ p ( Cbase ) ; ( ii ) Next , an episode τ is sampled from the data distribution conditioned on the set of sampled classes c : τ ∼ p ( D|c ) , where D is the base dataset . An episode τ consists of a set of support examples τs and query examples τq . In few-shot learning , a n-way , k-shot episode is sampled which results in sampling n classes and k support examples per class . Based on this , the meta-learning optimization objective can be generalized as the following : θ∗ = argmin θ Eτ [ ` ( Fθ′ , τq ) ] ( 1 ) where F is the base architecture with θ as the model parameters and θ′ = A ( θ , τs ) is the finetuning step with the support examples . Different meta-learners have different types of fine-tuning procedures and we direct the readers to ( Finn et al. , 2017 ; Snell et al. , 2017 ; Bertinetto et al. , 2018 ) for more information on the characteristics ofA . Based on this definition , we define the hardness of an episodeH ( τ ) in terms of the loss incurred on the query examples in an episode : H ( τ ) = ` ( Fθ∗ , τq ) ( 2 ) We choose query loss as a metric for hardness because of its inherent simplicity in computation as well as interpretation . In addition , we find a strong negative correlation between the episodic loss and the accuracy ( ≈ −0.92 for mini-ImageNet and ≈ −0.89 for tieredImageNet with prototypical networks ) . This is true for other meta-learners such as R2D2 too ( See Appendix A for more details ) . Alternatively , hardness of an episode can also be defined as the average log-odds of the query example ( Dhillon et al. , 2020 ) .

Summary: The submission investigates and characterizes episode difficulty (as defined in terms of its query loss) in few-shot classification and reports empirical observations on CIFAR-FS, mini-ImageNet, and tiered-ImageNet.  Across all benchmarks, a wide accuracy gap is observed between the hardest and easiest test episodes for multiple combinations of meta-learner (Prototypical Networks, R2D2) and network architecture (four-layer ConvNet, ResNet). The paper presents support and query images for easy and hard episodes and draws the conclusion that a mismatch in semantic or shape characteristics or in the number of objects between support and query images often cause misclassification.  The submission then examines how the accuracy of easy and hard episodes evolves over training. It uncovers global and local forgetting events and reports that the latter occur more frequently for hard episodes.  Finally, the paper examines two strategies for taking easy and hard episodes into account: adversarial training, and adversarial curriculum training. Adversarial training is shown to yield modest improvements over (and adversarial curriculum training does not perform significantly better than) regular training.