Generalized Sampling Method for Few Shot Learning

1 INTRODUCTION . Few-shot learning ( FSL ) refers the problem of learning from datasets where only a limited number of examples ( typically , one to tens per class or a problem in general ) are available for training a machine learning model . FSL has gained importance over the years since obtaining large labelled datasets requires a significant investment of time and resources . There is a rich history of research into various methodologies for FSL , two primary ones being model development approaches and representation learning . ( Wang et al. , 2020 ) . Model development approaches aim at capturing the data distribution so that new points can be sampled to improve few-shot classification accuracy ( Park et al. , 2020 ) , ( Wang et al. , 2019 ) , ( Chen et al. , 2019b ) , ( Zhang et al. , 2019 ) . They have typically relied on complex models and loss functions to understand data from only a few examples , which limits the interpretability of the model and hampers generalization . Representation learning , on the other hand , aims at identifying feature transformations which can allow simple statistical techniques like nearest neighbor and bayesian classification generalize on few-shot tasks for novel classes ( Chen et al. , 2020 ) , ( Xue & Wang , 2020 ) . Starting from early works ( Miller et al. , 2000 ) and building to recent methodologies ( Yang et al. , 2021 ) , ( Zhang et al. , 2021b ) , representation learning strategies have relied on simple statistical assumptions that may not hold across diverse datasets . Recently , ( Yang et al. , 2021 ) showed that classes which are semantically similar in meaning are also correlated in the means and covariances of their feature distributions . They used the statistics of classes with plentiful datapoints ( called base classes ) to learn the distribution of classes with a few datapoints ( called novel classes ) . Despite some limiting assumptions , their method , called Distribution Calibration , outperformed much more complex models that relied on non-parametric optimization and generative models ( Park et al. , 2020 ) , ( Wang et al. , 2019 ) , ( Chen et al. , 2019b ) , ( Zhang et al. , 2019 ) . Building on this idea and inspired by traditional statistics , we present a rigorous generalized sampling method for Few-Shot Learning , called DC+ that outperforms existing stateof-the-art classification accuracy without introducing additional statistical assumptions or complex generative models . Our main contributions are : 1 . Introducing a principled approach to estimate novel class mean and covariance from the moments of a random variable weighted by the distance between the novel and the base classes , 2 . Incorporating the statistical technique of variance shrinkage , which not only helps increase the accuracy but also stabilizes covariance estimation in cases when the feature extractor is over-parametrized ( a common occurrence in modern deep learning models ) , 3 . Extending the applicability of the statistical sampling approach to arbitrary feature extractors by introducing general Gaussianization transformations , 4 . Presenting a single scaling parameter in Euclidean distance weighting that mitigates the need to search among Euclidean , Mahalanobis , and generalized distances for novel class estimation , and Combined , our contributions put statistical sampling approach on a sound foundation , close open questions in earlier research , and demonstrate 3 % to 5 % improvement over competitive states-ofthe-art for 5way-1shot and 5way-5shot tasks for miniImageNet ( Ravi & Larochelle , 2017 ) , CUB ( Welinder et al. , 2010 ) , StanfordDogs ( Khosla et al. , 2011 ) , highest level classes of tieredImagenet ( Ren et al. , 2018 ) and 1 % improvement Cross Domain 5way-1shot task of miniImagenet −→ CUB . 2 RELATED WORKS . Learning good features or manipulating the features to help generalize few-shot tasks is an active research area ( Hou et al. , 2019 ) ( Hao et al. , 2019 ) , ( Li et al. , 2019a ) . Miller et al . ( 2000 ) proposed a congealing process which learned a joint distribution among all available classes . This distribution could then be used as a prior knowledge for constructing efficient few-shot classifiers . Chen et al . ( 2020 ) showed that by pre-training first on entire base classes , few shot classification accuracies on novel classes could be improved with a simple meta-learning on nearest-centroid based classification algorithm . Their main focus was on the pretraining methods which could improve a cosine based classifier on the centroids of the extracted features . DC+ can be applied on top of these feature extractor techniques to further explore improvements . To correct bias in centroid or prototype estimations , several rectification methods were proposed— RestoreNet ( Xue & Wang , 2020 ) transforms the feature space to move the images closer to their true centroids . Our proposed method approximates this transformation with scaled Euclidean distances and weighted random variables for novel classes , without any additional learnable parameters . Zhang et al . ( 2021a ) proposed a 4-step method consisting of learning base class details as priors and then using these priors for correcting bias in novel prototype estimation . They then jointly fine tuned the feature extractor and bias corrector . Our method does not have this multi-step process and works with off-the-shelf feature extractors . Liu et al . ( 2020c ) attempted to reduce the bias in distance estimation through reducing intra-class bias by label propagation and cross-class bias through shifting features . Their method works in the transductive setting where entire data , including the query set is consumed without label information . Our method does not need additional unlabelled data from the query set . Again their method improves upon Prototypical Networks ( Snell et al. , 2017 ) whereas we show that our method can be applied with any feature extractor . DC+ can be broadly categorized as a data augmentation method . Several methods have been proposed earlier in this space . Antoniou & Storkey ( 2019 ) learn few-shot tasks by randomly labelling a subset of images and then augmenting this subset with different techniques like random crops , flips etc . Park et al . ( 2020 ) tried to transfer variance between different classes in order to simulate the query examples that can be encountered during test . Other works like Wang et al . ( 2019 ) and Liu et al . ( 2020b ) utilized the intra-class variance to perform augmentation . These methods leverage complex neural networks with large number of learnable parameters to generate new examples . Our method does not require any additional learnable parameter and uses simple statistical techniques while still outperforming all previous deep learning methods . Closest to DC+ , Yang et al . ( 2021 ) estimated the novel class distributions based on their similarity with the base classes . Their method implicitly assumed that the base classes were semantically independent of each other when constructing covariance estimates , did not consider the similarity strength between the base and novel classes when estimating novel class statistics , and could not be applied to arbitrary off-the-shelf feature extractors ( with activation functions different from relu ) and large feature dimensions often capable of producing ill-defined covariances . Our method does not make independence assumptions , leverages similarity information in the base classes , and can be applied to any off-the-shelf feature extractor . 3 PROPOSED APPROACH . 3.1 PROBLEM DEFINITION . Few-shot classification problems are defined as Nway-Kshot classification tasks T ( Vinyals et al. , 2016 ) where given a small support set S of features x̃ and labels y , S = { ( x̃i , yi ) } N×Ki=1 , x̃i ∈ Rd , yi ∈ C , consisting of K points from N classes , the model should correctly classify a query set Q = { ( x̃i , yi ) } N×K+N×qi=N×K+1 with q points from each of the N classes in the support set . The entire dataset D , is divided into Cb base , Cv validation and Cn novel classes such that Cb ∩ Cv ∩ Cn = φ and Cb ∪ Cv ∪ Cn = C. The goal is to train a model with tasks T sampled from Cb and use Cv for hyperparameter tuning , where each task T is an Nway-Kshot classification problem on N unique classes of the set under consideration , for example base , validation set Cb , Cv here . The performance of few-shot learning algorithms is reported as the average accuracy on the query set Q of tasks T sampled from Cn . 3.2 ALGORITHM . Our proposed methodology , DC+ , is outlined in Algorithm 1 . In the following subsections , we incrementally go through the steps of DC+ in detail . 3.2.1 GAUSSIANIZATION OF THE DATA . Following Yang et al . ( 2021 ) , our sampling methodology assumes that the input features follow a multivariate normal distribution . There are many methods of data gaussianization like Tukey ’ s Ladder of Powers ( Tukey , 1977 ) , Yeo-Johnson Transformation ( Weisberg , 2001 ) , and Iterative Gaussianization ( Chen & Gopinath , 2000 ) . In our experiments , we observed that Tukey ’ s Ladder of Powers outperformed other methods , but the fractional powers and log transform could only be applied to non-negative features ( feature extractors which have relu and equivalent activation functions in their final layers ) . Expanding the applicability of the sampling method to arbitrary feature extractors and activation functions in deep learning models , we make a choice on the transformation of input features denoted by random variable x as per equation 1 below , x̂ = { tukey ( x ) if x ≥ 0 always yeo johnson ( x ) otherwise ( 1 ) We give the definitions of tukey and yeo johnson in Appendix A . 3.2.2 PROPOSED RANDOM VARIABLE . We extrapolate the distribution of a given novel class as a weighted average of the distributions of k closest base classes . Formally , if x̃ ∈ Rd is a d dimensional support point from a novel class , and Xi ∈ Rd is a random variable denoting points of base class i , then we compose a random variable X′ representing our estimate of that novel class as X′ = x̃ + ∑ i∈Sk wiXi 1 + ∑ i∈Sk wi , ( 2 ) where wi are the weights assigned to the closest base classes in Sk . The associated mean and the covariance of this random variable are ( since x̃ is a constant vector , it does not affect the covariance in Σ′ ) , µ′ = x̃ + ∑ i∈Sk wiµi 1 + ∑ i∈Sk wi Σ′ = cov ( X′ ) ( 3 ) where µi = E [ Xi ] . There are many ways of estimating wi . One of the simplest is to look at the distance of the novel point from the base classes . In particular , we find the k closest base classes to x̃ in Sk , di = ||x̃− µi||2 , i ∈ Cb ( 4 ) Sk = { i| − di ∈ topk ( −di ) , i ∈ Cb } ( 5 ) Based on di ( for alternative distance formulations , see Appendix B ) , we construct the weights wi of each base class in Sk as , wi = 1 1 + dmi , i ∈ Sk ( 6 ) where m is a hyperparameter that helps in decaying wi as a function of di and gives us control in the relative weights of the classes in Sk . This form of weighted variable estimation is reminiscent of ( though not the same as ) inverse distance weighting , which is widely used to estimate unknown functions at interpolated points ( Shepard , 1968 ) , ( Łukaszyk , 2004 ) It is worth noting that limdi→0 wi = 1 . Hence as the base class i moves closer to the support point x̃ , it gains increasing weight until µi overlaps with x̃ , at which point the weight is 1 , same as DC method .

Summary: This paper suggests a data augmentation method for few-shot learning. They generalize the similar method, DC (Yang et al., 2021) introducing a couple of hyperparameters to tune. Like DC, without introducing any learnable parameters, they suggest a way to augment few-shot data by extrapolating its distribution base on weighted empirical distributions of base classes. In addition, in a case when the number of data is smaller, they suggest a covariance shrinkage technique. The suggested method is empirically supported with experiments where the suggested method outperforms other baselines.  (Yang et al., 2021) Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id= JWOiYxMG92s