Generalized Sampling Method for Few Shot Learning

1 INTRODUCTION . Few-shot learning ( FSL ) refers the problem of learning from datasets where only a limited number of examples ( typically , one to tens per class or a problem in general ) are available for training a machine learning model . FSL has gained importance over the years since obtaining large labelled datasets requires a significant investment of time and resources . There is a rich history of research into various methodologies for FSL , two primary ones being model development approaches and representation learning . ( Wang et al. , 2020 ) . Model development approaches aim at capturing the data distribution so that new points can be sampled to improve few-shot classification accuracy ( Park et al. , 2020 ) , ( Wang et al. , 2019 ) , ( Chen et al. , 2019b ) , ( Zhang et al. , 2019 ) . They have typically relied on complex models and loss functions to understand data from only a few examples , which limits the interpretability of the model and hampers generalization . Representation learning , on the other hand , aims at identifying feature transformations which can allow simple statistical techniques like nearest neighbor and bayesian classification generalize on few-shot tasks for novel classes ( Chen et al. , 2020 ) , ( Xue & Wang , 2020 ) . Starting from early works ( Miller et al. , 2000 ) and building to recent methodologies ( Yang et al. , 2021 ) , ( Zhang et al. , 2021b ) , representation learning strategies have relied on simple statistical assumptions that may not hold across diverse datasets . Recently , ( Yang et al. , 2021 ) showed that classes which are semantically similar in meaning are also correlated in the means and covariances of their feature distributions . They used the statistics of classes with plentiful datapoints ( called base classes ) to learn the distribution of classes with a few datapoints ( called novel classes ) . Despite some limiting assumptions , their method , called Distribution Calibration , outperformed much more complex models that relied on non-parametric optimization and generative models ( Park et al. , 2020 ) , ( Wang et al. , 2019 ) , ( Chen et al. , 2019b ) , ( Zhang et al. , 2019 ) . Building on this idea and inspired by traditional statistics , we present a rigorous generalized sampling method for Few-Shot Learning , called DC+ that outperforms existing stateof-the-art classification accuracy without introducing additional statistical assumptions or complex generative models . Our main contributions are : 1 . Introducing a principled approach to estimate novel class mean and covariance from the moments of a random variable weighted by the distance between the novel and the base classes , 2 . Incorporating the statistical technique of variance shrinkage , which not only helps increase the accuracy but also stabilizes covariance estimation in cases when the feature extractor is over-parametrized ( a common occurrence in modern deep learning models ) , 3 . Extending the applicability of the statistical sampling approach to arbitrary feature extractors by introducing general Gaussianization transformations , 4 . Presenting a single scaling parameter in Euclidean distance weighting that mitigates the need to search among Euclidean , Mahalanobis , and generalized distances for novel class estimation , and Combined , our contributions put statistical sampling approach on a sound foundation , close open questions in earlier research , and demonstrate 3 % to 5 % improvement over competitive states-ofthe-art for 5way-1shot and 5way-5shot tasks for miniImageNet ( Ravi & Larochelle , 2017 ) , CUB ( Welinder et al. , 2010 ) , StanfordDogs ( Khosla et al. , 2011 ) , highest level classes of tieredImagenet ( Ren et al. , 2018 ) and 1 % improvement Cross Domain 5way-1shot task of miniImagenet −→ CUB . 2 RELATED WORKS . Learning good features or manipulating the features to help generalize few-shot tasks is an active research area ( Hou et al. , 2019 ) ( Hao et al. , 2019 ) , ( Li et al. , 2019a ) . Miller et al . ( 2000 ) proposed a congealing process which learned a joint distribution among all available classes . This distribution could then be used as a prior knowledge for constructing efficient few-shot classifiers . Chen et al . ( 2020 ) showed that by pre-training first on entire base classes , few shot classification accuracies on novel classes could be improved with a simple meta-learning on nearest-centroid based classification algorithm . Their main focus was on the pretraining methods which could improve a cosine based classifier on the centroids of the extracted features . DC+ can be applied on top of these feature extractor techniques to further explore improvements . To correct bias in centroid or prototype estimations , several rectification methods were proposed— RestoreNet ( Xue & Wang , 2020 ) transforms the feature space to move the images closer to their true centroids . Our proposed method approximates this transformation with scaled Euclidean distances and weighted random variables for novel classes , without any additional learnable parameters . Zhang et al . ( 2021a ) proposed a 4-step method consisting of learning base class details as priors and then using these priors for correcting bias in novel prototype estimation . They then jointly fine tuned the feature extractor and bias corrector . Our method does not have this multi-step process and works with off-the-shelf feature extractors . Liu et al . ( 2020c ) attempted to reduce the bias in distance estimation through reducing intra-class bias by label propagation and cross-class bias through shifting features . Their method works in the transductive setting where entire data , including the query set is consumed without label information . Our method does not need additional unlabelled data from the query set . Again their method improves upon Prototypical Networks ( Snell et al. , 2017 ) whereas we show that our method can be applied with any feature extractor . DC+ can be broadly categorized as a data augmentation method . Several methods have been proposed earlier in this space . Antoniou & Storkey ( 2019 ) learn few-shot tasks by randomly labelling a subset of images and then augmenting this subset with different techniques like random crops , flips etc . Park et al . ( 2020 ) tried to transfer variance between different classes in order to simulate the query examples that can be encountered during test . Other works like Wang et al . ( 2019 ) and Liu et al . ( 2020b ) utilized the intra-class variance to perform augmentation . These methods leverage complex neural networks with large number of learnable parameters to generate new examples . Our method does not require any additional learnable parameter and uses simple statistical techniques while still outperforming all previous deep learning methods . Closest to DC+ , Yang et al . ( 2021 ) estimated the novel class distributions based on their similarity with the base classes . Their method implicitly assumed that the base classes were semantically independent of each other when constructing covariance estimates , did not consider the similarity strength between the base and novel classes when estimating novel class statistics , and could not be applied to arbitrary off-the-shelf feature extractors ( with activation functions different from relu ) and large feature dimensions often capable of producing ill-defined covariances . Our method does not make independence assumptions , leverages similarity information in the base classes , and can be applied to any off-the-shelf feature extractor . 3 PROPOSED APPROACH . 3.1 PROBLEM DEFINITION . Few-shot classification problems are defined as Nway-Kshot classification tasks T ( Vinyals et al. , 2016 ) where given a small support set S of features x̃ and labels y , S = { ( x̃i , yi ) } N×Ki=1 , x̃i ∈ Rd , yi ∈ C , consisting of K points from N classes , the model should correctly classify a query set Q = { ( x̃i , yi ) } N×K+N×qi=N×K+1 with q points from each of the N classes in the support set . The entire dataset D , is divided into Cb base , Cv validation and Cn novel classes such that Cb ∩ Cv ∩ Cn = φ and Cb ∪ Cv ∪ Cn = C. The goal is to train a model with tasks T sampled from Cb and use Cv for hyperparameter tuning , where each task T is an Nway-Kshot classification problem on N unique classes of the set under consideration , for example base , validation set Cb , Cv here . The performance of few-shot learning algorithms is reported as the average accuracy on the query set Q of tasks T sampled from Cn . 3.2 ALGORITHM . Our proposed methodology , DC+ , is outlined in Algorithm 1 . In the following subsections , we incrementally go through the steps of DC+ in detail . 3.2.1 GAUSSIANIZATION OF THE DATA . Following Yang et al . ( 2021 ) , our sampling methodology assumes that the input features follow a multivariate normal distribution . There are many methods of data gaussianization like Tukey ’ s Ladder of Powers ( Tukey , 1977 ) , Yeo-Johnson Transformation ( Weisberg , 2001 ) , and Iterative Gaussianization ( Chen & Gopinath , 2000 ) . In our experiments , we observed that Tukey ’ s Ladder of Powers outperformed other methods , but the fractional powers and log transform could only be applied to non-negative features ( feature extractors which have relu and equivalent activation functions in their final layers ) . Expanding the applicability of the sampling method to arbitrary feature extractors and activation functions in deep learning models , we make a choice on the transformation of input features denoted by random variable x as per equation 1 below , x̂ = { tukey ( x ) if x ≥ 0 always yeo johnson ( x ) otherwise ( 1 ) We give the definitions of tukey and yeo johnson in Appendix A . 3.2.2 PROPOSED RANDOM VARIABLE . We extrapolate the distribution of a given novel class as a weighted average of the distributions of k closest base classes . Formally , if x̃ ∈ Rd is a d dimensional support point from a novel class , and Xi ∈ Rd is a random variable denoting points of base class i , then we compose a random variable X′ representing our estimate of that novel class as X′ = x̃ + ∑ i∈Sk wiXi 1 + ∑ i∈Sk wi , ( 2 ) where wi are the weights assigned to the closest base classes in Sk . The associated mean and the covariance of this random variable are ( since x̃ is a constant vector , it does not affect the covariance in Σ′ ) , µ′ = x̃ + ∑ i∈Sk wiµi 1 + ∑ i∈Sk wi Σ′ = cov ( X′ ) ( 3 ) where µi = E [ Xi ] . There are many ways of estimating wi . One of the simplest is to look at the distance of the novel point from the base classes . In particular , we find the k closest base classes to x̃ in Sk , di = ||x̃− µi||2 , i ∈ Cb ( 4 ) Sk = { i| − di ∈ topk ( −di ) , i ∈ Cb } ( 5 ) Based on di ( for alternative distance formulations , see Appendix B ) , we construct the weights wi of each base class in Sk as , wi = 1 1 + dmi , i ∈ Sk ( 6 ) where m is a hyperparameter that helps in decaying wi as a function of di and gives us control in the relative weights of the classes in Sk . This form of weighted variable estimation is reminiscent of ( though not the same as ) inverse distance weighting , which is widely used to estimate unknown functions at interpolated points ( Shepard , 1968 ) , ( Łukaszyk , 2004 ) It is worth noting that limdi→0 wi = 1 . Hence as the base class i moves closer to the support point x̃ , it gains increasing weight until µi overlaps with x̃ , at which point the weight is 1 , same as DC method .

Summary: The authors propose a statistical method for estimating the data distribution for few-shot learning. The method is built on a previous work (DC) with minor fixes but have non-trivial performance improvements. Specifically, the main differences lie in the transformation function, weighted random variables and the covariance shrinkage.