Class Imbalance in Few-Shot Learning

1 INTRODUCTION . Deep learning methods are well known for their state-of-the-art performances on a variety of tasks ( LeCun et al. , 2015 ; Russakovsky et al. , 2015 ; Schmidhuber , 2015 ) . However , they often require to be trained on large labeled datasets to acquire robust and generalizable features . Few-Shot Learning ( FSL ) ( Chen et al. , 2019 ; Wang et al. , 2019b ; Bendre et al. , 2020 ) aims at reducing this burden by defining a distribution over tasks , with each task containing a few labeled data points ( support set ) and a set of target data ( query set ) belonging to the same set of classes . A common way to train FSL methods is through episodic meta-training ( Vinyals et al. , 2017 ) with the model repeatedly exposed to batches of tasks sampled from a task-distribution and then tested on a different but similar distribution in the meta-testing phase . The prefix “ meta ” is commonly used to distinguish the highlevel training and evaluation routines of meta-learning ( outer loop ) , from the training and evaluation routines at the single-task level ( inner loop ) . Limitations . Standard meta-training overlooks many challenges stemming from real-world dynamics , such as class-imbalance ( CI ) . The standard setting assumes that all classes in the support set contain the same number of data points , whereas in many practical applications , the number of samples for each class may vary ( Buda et al. , 2018 ; Leevy et al. , 2018 ) . Given the limited amount of data used in FSL , a small difference in the number of samples between classes could already introduce significant levels of imbalance . Most FSL methods are not designed to cope with these more challenging settings . Figure 1 exemplifies these considerations by showing that several state-of-the-art FSL methods underperform when tested under three CI regimes ( linear , step , random ) . Previous work . Previous work mainly focuses on the single imbalance case or grouping several settings into one task , offering limited insights into the effects of CI on FSL and making it challenging to quantify its effects ( Guan et al. , 2020 ; Triantafillou et al. , 2020 ; Lee et al. , 2019 ; Chen et al. , 2020 ) . A common approach to mitigate imbalance is Random-Shot meta-training ( Triantafillou et al. , 2020 ) , which exposes the model to imbalanced tasks during meta-training . However , previous work provides little insight into the effectiveness of this procedure on the imbalanced FSL evaluation task . Furthermore , minimal work exists that investigates meta-training outcomes under an imbalanced distribution of classes at the ( meta- ) dataset level , while this case is common in recent FSL applications ( Ochal et al. , 2020 ; Guan et al. , 2020 ) and meta-learning benchmarks ( Triantafillou et al. , 2020 ) . The CI problem is well-known within the supervised learning community , which has systematically produced strategies to deal with the problem , such as the popular Random Over-Sampling ( Japkowicz & Stephen , 2002 ) that aims at rebalancing minority classes by uniform sampling . While such strategies have been extensively studied on many supervised learning problems , there is little understanding of how they behave with the recently proposed FSL methods in the low-data regime . Our work and main contributions . In this paper , we provide , for the first time , a detailed analysis of the CI problem within the FSL framework . Our results show that even small CI levels can introduce a significant performance drop for all the methods considered . Moreover , we find that only a few models benefit from Random-Shot meta-training ( Triantafillou et al. , 2020 ; Lee et al. , 2019 ; Chen et al. , 2020 ) over the classical ( balanced ) episodic meta-training ( Vinyals et al. , 2017 ) ; while pairing the meta-training procedures with Random Over-Sampling offers a substantial advantage . The experimental results show that imbalance severity at the dataset level depends on the size of the dataset . Our contributions can be summarized as follows : 1 . A systematic , comprehensive and in-depth study of the effects of CI within the FSL framework along three axes : ( i ) dataset vs. support set imbalance , ( ii ) effect of different imbalance distributions ( linear , step , random ) , ( iii ) effect of rebalancing techniques , such as random over-sampling and the recently proposed Random-Shot meta-training ( Triantafillou et al. , 2020 ) . 2 . We reveal novel insights into the meta-learning and support set adaptation capabilities to the CI regime , supported by extensive results on over 10 FSL methods with different imbalance settings , backbones , support set sizes , and datasets . 3 . We provide insight into the previously unaddressed CI problem in the ( meta- ) training dataset , showing that the effects of imbalance at the dataset level are less significant than the effects at the support set level . 2 RELATED WORK . 2.1 CLASS IMBALANCE . In classification , imbalance occurs when at least one class ( the majority class ) contains a higher number of samples than the others . The classes with the lowest number of samples are called minority classes . If uncorrected , conventional supervised loss functions , such as ( multi-class ) cross-entropy , skew the learning process in favor of the majority class , introducing bias and poor generalization toward the minority class samples ( Buda et al. , 2018 ; Leevy et al. , 2018 ) . Imbalance approaches are categorized into three groups : data-level , algorithm-level , and hybrid . Data-level strategies manipulate and create new data points to equalize data sampling . Popular data-level methods include Random Over-Sampling ( ROS ) and Random Under-Sampling ( RUS ) ( Japkowicz & Stephen , 2002 ) . ROS randomly resamples data points from the minority classes , while RUS randomly leaves out a randomly selected portion of the majority classes to decrease imbalance levels . Algorithm-level strategies use regularization or minimization of loss/cost functions . Weighted loss is a common approach where each sample ’ s loss is weighted by the inverse frequency of that sample ’ s class . Focal loss ( Lin et al. , 2017 ) is another type of cost function that has seen wide success . Hybrid methods combine one or more types of strategies ( e.g . Two-Phase Training , Havaei et al . ( 2017 ) ) . Modeling Imbalance . The object recognition community studies class imbalance using realworld datasets or distributions that approximate real-world imbalance ( Buda et al. , 2018 ; Johnson & Khoshgoftaar , 2019 ; Liu et al. , 2019 ) . Buda et al . ( 2018 ) note that two distributions can be used : linear and step imbalance ( defined in our methodology Section 3 ) . At large-scale , datasets with many samples and classes tend to follow a long-tail distribution ( Liu et al. , 2019 ; Salakhutdinov et al. , 2011 ; Reed , 2001 ) , with most of the classes occurring with small frequency and a few classes occurring with high frequency . Our work primarily focuses on the tail-end of the distribution and does not consider the case of large sample size . Therefore , we do not examine the long-tail mechanisms . 2.2 FEW-SHOT LEARNING . FSL methods can be broadly categorized into metric-learning , optimization-based , hallucination , data-adaptation , and probabilistic approaches ( Chen et al. , 2019 ) . Metric-learning approaches such as Prototypical Networks ( Snell et al. , 2017 ) , Relation Networks ( Sung et al. , 2017 ) , the Neural Statistician ( Edwards & Storkey , 2017 ) and Matching Networks ( Vinyals et al. , 2017 ) , learn a feature extractor capable of parameterizing images into embeddings , and then use distance metrics to classify mapped query samples based on their distance to support points . Optimization-based approaches such as MAML ( Finn et al. , 2017 ) and Meta-Learner LSTM ( Ravi & Larochelle , 2016 ) ) , are meta-trained to use guided optimization steps on the support set for quick adaptation . Hallucination or data augmentation techniques perform affine and color transformations on the support set to create additional data points ( Zhang et al. , 2018 ) . Probabilistic methods use Bayesian inference to learn and classify samples , for example , the recently proposed Deep Kernel Transfer ( DKT ) ( Patacchiola et al. , 2020 ) , which uses Gaussian Process at inference time . We use the term domain adaptation to represent those approaches using standard transfer-learning with a pre-training stage on a large set of classes and a fine-tune stage on the support set – examples are Baseline and Baseline++ from Chen et al . ( 2019 ) , and the recently proposed Transductive Fine-Tuning from Dhillon et al . ( 2020 ) . The details of the methods used in our experiments are reported in Appendix A . For completeness , it is worth mentioning Incremental Few-Shot Learning ( Ren et al. , 2018 ; Gidaris & Komodakis , 2018 ; Hariharan & Girshick , 2017 ) which is an extension of FSL . It considers maintaining performance on base classes ( meta-training dataset ) while incrementally learning about novel classes using limited data , typically without re-training from scratch on all data . Here , we focus on studying how imbalance affects the learning of novel classes only ; therefore , we will not consider incremental FSL further . 2.3 IMBALANCE IN FEW-SHOT AND META LEARNING . Class Imbalance in the low-data regime has received some attention , although the current work is not comprehensive ( Guan et al. , 2020 ; Triantafillou et al. , 2020 ; Lee et al. , 2019 ; Chen et al. , 2020 ) . We identify that in FSL , class-imbalance occurs at two levels : the task-level and the meta-dataset level . At the task level , class-imbalance occurs in the support set or the query-set , directly affecting learning and evaluation procedures . Class imbalance at the meta-dataset level is caused by imbalanced dataset classes in one ( or more ) of the three data splits : meta-training , meta-validation , meta-testing . This disproportion affects the distribution of tasks that a model is exposed to during meta-training , affecting their ability to generalize to new tasks . In Figure 2 , we highlight the differences between imbalanced task and imbalanced meta-dataset . Related to , but distinct from , these two class-imbalance types is task-distribution imbalance ( Lee et al. , 2019 ) ; skewed task-distribution can occur as a result of meta-dataset level class-imbalance or as a result of the task-sampling procedure . In extreme cases , task-distribution imbalance can lead to out-of-distribution tasks during meta-evaluation . Task-distribution imbalance has already received some attention ( Lee et al. , 2019 ; Cao et al. , 2020 ) ; therefore , it will not be considered in this work . Class Imbalance in Tasks . Triantafillou et al . ( 2020 ) uses imbalanced support sets to create a more realistic and challenging benchmark for meta-learning . The authors use random-shot tasks with randomly selected classes ( way ) and samples ( shot ) , which replace the balanced task during meta-training and meta-evaluation . A similar idea is explored by Lee et al . ( 2019 ) , Chen et al . ( 2020 ) , and Guan et al . ( 2020 ) with the last two using a fixed number of classes ( way ) . However , none of these works quantify the impact of class-imbalance on the FSL task nor the advantages of Random-Shot meta-training . Lee et al . ( 2019 ) explores a small range of class-imbalance in the support set . However , details into the effects of class-imbalance are lost when combined with taskdistribution imbalance , making it challenging to attribute any performance changes caused by classimbalance . Chen et al . ( 2020 ) explore a pure class-imbalance problem on the support set , but their analysis is limited to just two methods ( their proposal and MAML ) . In Guan et al . ( 2020 ) , metalearning is applied on aerial imagery , exploring step imbalance ranging from 5 to 140 samples per class ( shot ) ; however , only two FSL methods are compared ( Prototypical Networks and their RFMML method ) . Previous work provides limited insight into class-imbalance at the task level . Class Imbalance in the Meta-Dataset . Standard meta-datasets ( e.g . Mini-ImageNet ) can be swapped for other domain-specific datasets , such as CUB ( Wah et al. , 2011 ) , VGG Flowers ( Nilsback & Zisserman , 2008 ) , and others ( Triantafillou et al. , 2020 ) . These datasets sometimes contain an unequal number of class samples , but previous work has never reported the effects of classimbalance in the meta-training dataset ( Guan et al. , 2020 ; Triantafillou et al. , 2020 ; Lee et al. , 2019 ; Chen et al. , 2020 ) . We emphasize that studying the impact of imbalance at this level is important since imbalanced domain-specific meta-datasets are common in real-world applications ( Guan et al. , 2020 ; Ochal et al. , 2020 ) and recent benchmarks ( Triantafillou et al. , 2020 ) . Our work is the first to provide quantitative insights into this setting .

Summary: The authors present a detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques. The authors extensively compare over 10 state-of-the-art few-shot learning methods using backbones of different depths on multiple datasets. The analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to 18.0% for optimization-based methods, although feature-transfer and metric-based methods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can be adapted to the few-shot case resulting in better performances, 3) the effects of imbalance at the dataset level are less significant than the effects at the support set level.