Free Lunch for Few-shot Learning:  Distribution Calibration

1 INTRODUCTION Table 1 : The class mean similarity ( “ mean sim ” ) and class variance similarity ( “ var sim ” ) between Arctic fox and different classes . Arctic fox mean sim var sim white wolf 97 % 97 % malamute 85 % 78 % lion 81 % 70 % meerkat 78 % 70 % jellyfish 46 % 26 % orange 40 % 19 % beer bottle 34 % 11 % Learning from a limited number of training samples has drawn increasing attention due to the high cost of collecting and annotating a large amount of data . Researchers have developed algorithms to improve the performance of models that have been trained with very few data . Finn et al . ( 2017 ) ; Snell et al . ( 2017 ) train models in a meta-learning fashion so that the model can adapt quickly on tasks with only a few training samples available . Hariharan & Girshick ( 2017 ) ; Wang et al . ( 2018 ) try to synthesize data or features by learning a generative model to alleviate the data insufficiency problem . Ren et al . ( 2018 ) propose to leverage unlabeled data and predict pseudo labels to improve the performance of fewshot learning . While most previous works focus on developing stronger models , scant attention has been paid to the property of the data itself . It is natural that when the number of data grows , the ground truth distribution can be more accurately uncovered . Models trained with a wide coverage of data can generalize well during evaluation . On the other hand , when training a model with only a few training data , the model tends to overfit on these few samples by minimizing the training loss over these samples . These phenomena are illustrated in Figure 1 . This biased distribution based on a few examples can damage the generalization ability of the model since it is far from mirroring the ground truth distribution from which test cases are sampled during evaluation . ∗Corresponding author . Here , we consider calibrating this biased distribution into a more accurate approximation of the ground truth distribution . In this way , a model trained with inputs sampled from the calibrated distribution can generalize over a broader range of data from a more accurate distribution rather than only fitting itself to those few samples . Instead of calibrating the distribution of the original data space , we try to calibrate the distribution in the feature space , which has much lower dimensions and is easier to calibrate ( Xian et al . ( 2018 ) ) . We assume every dimension in the feature vectors follows a Gaussian distribution and observe that similar classes usually have similar mean and variance of the feature representations , as shown in Table 1 . Thus , the mean and variance of the Gaussian distribution can be transferred across similar classes ( Salakhutdinov et al . ( 2012 ) ) . Meanwhile , the statistics can be estimated more accurately when there are adequate samples for this class . Based on these observations , we reuse the statistics from many-shot classes and transfer them to better estimate the distribution of the few-shot classes according to their class similarity . More samples can be generated according to the estimated distribution which provides sufficient supervision for training the classification model . In the experiments , we show that a simple logistic regression classifier trained with our strategy can achieve state-of-the-art accuracy on three datasets . Our distribution calibration strategy can be paired with any classifier and feature extractor with no extra learnable parameters . Training with samples selected from the calibrated distribution can achieve 12 % accuracy gain compared to the baseline which is only trained with the few samples given in a 5way1shot task . We also visualize the calibrated distribution and show that it is an accurate approximation of the ground truth that can better cover the test cases . 2 RELATED WORKS . Few-shot classification is a challenging machine learning problem and researchers have explored the idea of learning to learn or meta-learning to improve the quick adaptation ability to alleviate the few-shot challenge . One of the most general algorithms for meta-learning is the optimizationbased algorithm . Finn et al . ( 2017 ) and Li et al . ( 2017 ) proposed to learn how to optimize the gradient descent procedure so that the learner can have a good initialization , update direction , and learning rate . For the classification problem , researchers proposed simple but effective algorithms based on metric learning . MatchingNet ( Vinyals et al. , 2016 ) and ProtoNet ( Snell et al. , 2017 ) learned to classify samples by comparing the distance to the representatives of each class . Our distribution calibration and feature sampling procedure does not include any learnable parameters and the classifier is trained in a traditional supervised learning way . Another line of algorithms is to compensate for the insufficient number of available samples by generation . Most methods use the idea of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) or autoencoder ( Rumelhart et al. , 1986 ) to generate samples ( Zhang et al . ( 2018 ) ; Chen et al . ( 2019b ) ; Schwartz et al . ( 2018 ) ; Gao et al . ( 2018 ) ) or features ( Xian et al . ( 2018 ) ; Zhang et al . ( 2019 ) ) to augment the training set . Specifically , Zhang et al . ( 2018 ) and Xian et al . ( 2018 ) proposed to synthesize data by introducing an adversarial generator conditioned on tasks . Zhang et al . ( 2019 ) tried to learn a variational autoencoder to approximate the distribution and predict labels based on the estimated statistics . The autoencoder can also augment samples by projecting between the visual space and the semantic space ( Chen et al. , 2019b ) or encoding the intra-class deformations ( Schwartz et al. , 2018 ) . Liu et al . ( 2019b ) and Liu et al . ( 2019a ) propose to generate features through the class hierarchy . While these methods can generate extra samples or features for training , they require the design of a complex model and loss function to learn how to generate . However , our distribution calibration strategy is simple and does not need extra learnable parameters . Data augmentation is a traditional and effective way of increasing the number of training samples . Qin et al . ( 2020 ) and Antoniou & Storkey ( 2019 ) proposed the used of the traditional data augmentation technique to construct pretext tasks for unsupervised few-shot learning . Wang et al . ( 2018 ) and Hariharan & Girshick ( 2017 ) leveraged the general idea of data augmentation , they designed a hallucination model to generate the augmented version of the image with different choices for the model ’ s input , i.e. , an image and a noise ( Wang et al. , 2018 ) or the concatenation of multiple features ( Hariharan & Girshick , 2017 ) . Park et al . ( 2020 ) ; Wang et al . ( 2019 ) ; Liu et al . ( 2020b ) tried to augment feature representations by leveraging intra-class variance . These methods learn to augment from the original samples or their feature representation while we try to estimate the classlevel distribution and thus can eliminate the inductive bias from a single sample and provide more diverse generations from the calibrated distribution . 3 MAIN APPROACH . In this section , we introduce the few-shot classification problem definition in Section 3.1 and details of our proposed approach in Section 3.2 . 3.1 PROBLEM DEFINITION . We follow a typical few-shot classification setting . Given a dataset with data-label pairs D = { ( xi , yi ) } where xi ∈ Rd is the feature vector of a sample and yi ∈ C , where C denotes the set of classes . This set of classes is divided into base classes Cb and novel classes Cn , where Cb∩Cn = ∅ and Cb ∪ Cn = C. The goal is to train a model on the data from the base classes so that the model can generalize well on tasks sampled from the novel classes . In order to evaluate the fast adaptation ability or the generalization ability of the model , there are only a few available labeled samples for each task T . The most common way to build a task is called an N-way-K-shot task ( Vinyals et al . ( 2016 ) ) , where N classes are sampled from the novel set and only K ( e.g. , 1 or 5 ) labeled samples are provided for each class . The few available labeled data are called support set S = { ( xi , yi ) } N×Ki=1 and the model is evaluated on another query setQ = { ( xi , yi ) } N×K+N×qi=N×K+1 , where every class in the task has q test cases . Thus , the performance of a model is evaluated as the averaged accuracy on ( the query set of ) multiple tasks sampled from the novel classes . 3.2 DISTRIBUTION CALIBRATION . As introduced in Section 3.1 , the base classes have a sufficient amount of data while the evaluation tasks sampled from the novel classes only have a limited number of labeled samples . The statistics of the distribution for the base class can be estimated more accurately compared to the estimation based on few-shot samples , which is an ill-posed problem . As shown in Table 1 , we observe that if we assume the feature distribution is Gaussian , the mean and variance with respect to each class are correlated to the semantic similarity of each class . With this in mind , the statistics can be transferred from the base classes to the novel classes if we learn how similar the two classes are . In the following sections , we discuss how we calibrate the distribution estimation of the classes with only a few samples ( Section 3.2.2 ) with the help of the statistics of the base classes ( Section 3.2.1 ) . We will also elaborate on how do we leverage the calibrated distribution to improve the performance of few-shot learning ( Section 3.2.3 ) . Note that our distribution calibration strategy is over the feature-level and is agnostic to any feature extractor . Thus , it can be built on top of any pretrained feature extractors without further costly finetuning . In our experiments , we use the pretrained WideResNet Zagoruyko & Komodakis ( 2016 ) following previous work ( Mangla et al . ( 2020 ) ) . The WideResNet is trained to classify the base classes , along with a self-supervised pretext task to learn the general-purpose representations suitable for image understanding tasks . Please refer to their paper for more details on training the feature extractor . Algorithm 1 Training procedure for an N-way-K-shot task Require : Support set features S = ( xi , y ) N×Ki=1 Require : Base classes ’ statistics { µi } |Cb|i=1 , { Σi } |Cb| i=1 1 : Transform ( xi ) N×Ki=1 with Tukey ’ s Ladder of Powers as Equation 3 2 : for ( xi , yi ) ∈ S do 3 : Calibrate the mean µ′ and the covariance Σ′ for class yi using xi with Equation 6 4 : Sample features for class yi from the calibrated distribution as Equation 7 5 : end for 6 : Train a classifier using both support set features and all sampled features as Equation 8

Summary: The paper proposes a method to calibrate the underlying distribution of a few samples in the few-shot classification scenario. The idea is to estimate a feature distribution of a few samples of a novel class from base class distributions. The authors assume that every dimension in the feature vector follows a Gaussian distribution. Based on the observation that the mean and variance of the distribution with respect to each class are correlated to the semantic similarity of each class, base class distribution can be transferred to the novel class distribution. After distribution calibration, features can be generated from the calibrated distribution and the generated features are used to train classifiers. SVM and logistic regression classifier are used to verify the approach on the mini-imagenet and CUB datasets.