Free Lunch for Few-shot Learning:  Distribution Calibration

1 INTRODUCTION Table 1 : The class mean similarity ( “ mean sim ” ) and class variance similarity ( “ var sim ” ) between Arctic fox and different classes . Arctic fox mean sim var sim white wolf 97 % 97 % malamute 85 % 78 % lion 81 % 70 % meerkat 78 % 70 % jellyfish 46 % 26 % orange 40 % 19 % beer bottle 34 % 11 % Learning from a limited number of training samples has drawn increasing attention due to the high cost of collecting and annotating a large amount of data . Researchers have developed algorithms to improve the performance of models that have been trained with very few data . Finn et al . ( 2017 ) ; Snell et al . ( 2017 ) train models in a meta-learning fashion so that the model can adapt quickly on tasks with only a few training samples available . Hariharan & Girshick ( 2017 ) ; Wang et al . ( 2018 ) try to synthesize data or features by learning a generative model to alleviate the data insufficiency problem . Ren et al . ( 2018 ) propose to leverage unlabeled data and predict pseudo labels to improve the performance of fewshot learning . While most previous works focus on developing stronger models , scant attention has been paid to the property of the data itself . It is natural that when the number of data grows , the ground truth distribution can be more accurately uncovered . Models trained with a wide coverage of data can generalize well during evaluation . On the other hand , when training a model with only a few training data , the model tends to overfit on these few samples by minimizing the training loss over these samples . These phenomena are illustrated in Figure 1 . This biased distribution based on a few examples can damage the generalization ability of the model since it is far from mirroring the ground truth distribution from which test cases are sampled during evaluation . ∗Corresponding author . Here , we consider calibrating this biased distribution into a more accurate approximation of the ground truth distribution . In this way , a model trained with inputs sampled from the calibrated distribution can generalize over a broader range of data from a more accurate distribution rather than only fitting itself to those few samples . Instead of calibrating the distribution of the original data space , we try to calibrate the distribution in the feature space , which has much lower dimensions and is easier to calibrate ( Xian et al . ( 2018 ) ) . We assume every dimension in the feature vectors follows a Gaussian distribution and observe that similar classes usually have similar mean and variance of the feature representations , as shown in Table 1 . Thus , the mean and variance of the Gaussian distribution can be transferred across similar classes ( Salakhutdinov et al . ( 2012 ) ) . Meanwhile , the statistics can be estimated more accurately when there are adequate samples for this class . Based on these observations , we reuse the statistics from many-shot classes and transfer them to better estimate the distribution of the few-shot classes according to their class similarity . More samples can be generated according to the estimated distribution which provides sufficient supervision for training the classification model . In the experiments , we show that a simple logistic regression classifier trained with our strategy can achieve state-of-the-art accuracy on three datasets . Our distribution calibration strategy can be paired with any classifier and feature extractor with no extra learnable parameters . Training with samples selected from the calibrated distribution can achieve 12 % accuracy gain compared to the baseline which is only trained with the few samples given in a 5way1shot task . We also visualize the calibrated distribution and show that it is an accurate approximation of the ground truth that can better cover the test cases . 2 RELATED WORKS . Few-shot classification is a challenging machine learning problem and researchers have explored the idea of learning to learn or meta-learning to improve the quick adaptation ability to alleviate the few-shot challenge . One of the most general algorithms for meta-learning is the optimizationbased algorithm . Finn et al . ( 2017 ) and Li et al . ( 2017 ) proposed to learn how to optimize the gradient descent procedure so that the learner can have a good initialization , update direction , and learning rate . For the classification problem , researchers proposed simple but effective algorithms based on metric learning . MatchingNet ( Vinyals et al. , 2016 ) and ProtoNet ( Snell et al. , 2017 ) learned to classify samples by comparing the distance to the representatives of each class . Our distribution calibration and feature sampling procedure does not include any learnable parameters and the classifier is trained in a traditional supervised learning way . Another line of algorithms is to compensate for the insufficient number of available samples by generation . Most methods use the idea of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) or autoencoder ( Rumelhart et al. , 1986 ) to generate samples ( Zhang et al . ( 2018 ) ; Chen et al . ( 2019b ) ; Schwartz et al . ( 2018 ) ; Gao et al . ( 2018 ) ) or features ( Xian et al . ( 2018 ) ; Zhang et al . ( 2019 ) ) to augment the training set . Specifically , Zhang et al . ( 2018 ) and Xian et al . ( 2018 ) proposed to synthesize data by introducing an adversarial generator conditioned on tasks . Zhang et al . ( 2019 ) tried to learn a variational autoencoder to approximate the distribution and predict labels based on the estimated statistics . The autoencoder can also augment samples by projecting between the visual space and the semantic space ( Chen et al. , 2019b ) or encoding the intra-class deformations ( Schwartz et al. , 2018 ) . Liu et al . ( 2019b ) and Liu et al . ( 2019a ) propose to generate features through the class hierarchy . While these methods can generate extra samples or features for training , they require the design of a complex model and loss function to learn how to generate . However , our distribution calibration strategy is simple and does not need extra learnable parameters . Data augmentation is a traditional and effective way of increasing the number of training samples . Qin et al . ( 2020 ) and Antoniou & Storkey ( 2019 ) proposed the used of the traditional data augmentation technique to construct pretext tasks for unsupervised few-shot learning . Wang et al . ( 2018 ) and Hariharan & Girshick ( 2017 ) leveraged the general idea of data augmentation , they designed a hallucination model to generate the augmented version of the image with different choices for the model ’ s input , i.e. , an image and a noise ( Wang et al. , 2018 ) or the concatenation of multiple features ( Hariharan & Girshick , 2017 ) . Park et al . ( 2020 ) ; Wang et al . ( 2019 ) ; Liu et al . ( 2020b ) tried to augment feature representations by leveraging intra-class variance . These methods learn to augment from the original samples or their feature representation while we try to estimate the classlevel distribution and thus can eliminate the inductive bias from a single sample and provide more diverse generations from the calibrated distribution . 3 MAIN APPROACH . In this section , we introduce the few-shot classification problem definition in Section 3.1 and details of our proposed approach in Section 3.2 . 3.1 PROBLEM DEFINITION . We follow a typical few-shot classification setting . Given a dataset with data-label pairs D = { ( xi , yi ) } where xi ∈ Rd is the feature vector of a sample and yi ∈ C , where C denotes the set of classes . This set of classes is divided into base classes Cb and novel classes Cn , where Cb∩Cn = ∅ and Cb ∪ Cn = C. The goal is to train a model on the data from the base classes so that the model can generalize well on tasks sampled from the novel classes . In order to evaluate the fast adaptation ability or the generalization ability of the model , there are only a few available labeled samples for each task T . The most common way to build a task is called an N-way-K-shot task ( Vinyals et al . ( 2016 ) ) , where N classes are sampled from the novel set and only K ( e.g. , 1 or 5 ) labeled samples are provided for each class . The few available labeled data are called support set S = { ( xi , yi ) } N×Ki=1 and the model is evaluated on another query setQ = { ( xi , yi ) } N×K+N×qi=N×K+1 , where every class in the task has q test cases . Thus , the performance of a model is evaluated as the averaged accuracy on ( the query set of ) multiple tasks sampled from the novel classes . 3.2 DISTRIBUTION CALIBRATION . As introduced in Section 3.1 , the base classes have a sufficient amount of data while the evaluation tasks sampled from the novel classes only have a limited number of labeled samples . The statistics of the distribution for the base class can be estimated more accurately compared to the estimation based on few-shot samples , which is an ill-posed problem . As shown in Table 1 , we observe that if we assume the feature distribution is Gaussian , the mean and variance with respect to each class are correlated to the semantic similarity of each class . With this in mind , the statistics can be transferred from the base classes to the novel classes if we learn how similar the two classes are . In the following sections , we discuss how we calibrate the distribution estimation of the classes with only a few samples ( Section 3.2.2 ) with the help of the statistics of the base classes ( Section 3.2.1 ) . We will also elaborate on how do we leverage the calibrated distribution to improve the performance of few-shot learning ( Section 3.2.3 ) . Note that our distribution calibration strategy is over the feature-level and is agnostic to any feature extractor . Thus , it can be built on top of any pretrained feature extractors without further costly finetuning . In our experiments , we use the pretrained WideResNet Zagoruyko & Komodakis ( 2016 ) following previous work ( Mangla et al . ( 2020 ) ) . The WideResNet is trained to classify the base classes , along with a self-supervised pretext task to learn the general-purpose representations suitable for image understanding tasks . Please refer to their paper for more details on training the feature extractor . Algorithm 1 Training procedure for an N-way-K-shot task Require : Support set features S = ( xi , y ) N×Ki=1 Require : Base classes ’ statistics { µi } |Cb|i=1 , { Σi } |Cb| i=1 1 : Transform ( xi ) N×Ki=1 with Tukey ’ s Ladder of Powers as Equation 3 2 : for ( xi , yi ) ∈ S do 3 : Calibrate the mean µ′ and the covariance Σ′ for class yi using xi with Equation 6 4 : Sample features for class yi from the calibrated distribution as Equation 7 5 : end for 6 : Train a classifier using both support set features and all sampled features as Equation 8

Summary: This paper identifies the problem of biased distributions in few-shot learning and proposes to fix it. In few-shot learning, only a few samples per class are available; this makes estimating the class distribution difficult. The paper proposes a distribution calibration algorithm that makes use of the meta-train class distributions to calibrate the few-shot class distributions. Once calibrated, more samples are drawn from this distribution to learn a classifier that generalizes better. This approach does not require additional learnable parameters and can be (potentially) built on-top of any pre-trained feature extractor. Empirical results show that this approach achieves state-of-the-art results on Mini-ImageNet and CUB.