Year: 2021

Abstract:
Few shot learning is an important problem in machine learning as large labelled datasets take considerable time and effort to assemble. Most few-shot learning algorithms suffer from one of two limitations— they either require the design of sophisticated models and loss functions, thus hampering interpretability; or employ statistical techniques but make assumptions that may not hold across different datasets or features. Developing on recent work in extrapolating distributions of small sample classes from the most similar larger classes, we propose a Generalized sampling method that learns to estimate few-shot distributions for classification as weighted random variables of all large classes. We use a form of covariance shrinkage to provide robustness against singular covariances due to overparameterized features or small datasets. We show that our sampled points are close to few-shot classes even in cases when there are no similar large classes in the training set. Our method works with arbitrary off-the-shelf feature extractors and outperforms existing state-of-the-art on miniImagenet, CUB and Stanford Dogs datasets by 3% to 5% on 5way-1shot and 5way-5shot tasks and by 1% in challenging cross domain tasks.

heading: 1 INTRODUCTION
text: Few-shot learning (FSL) refers the problem of learning from datasets where only a limited number of examples (typically, one to tens per class or a problem in general) are available for training a machine learning model. FSL has gained importance over the years since obtaining large labelled datasets requires a significant investment of time and resources. There is a rich history of research into various methodologies for FSL, two primary ones being model development approaches and representation learning. (Wang et al., 2020).
Model development approaches aim at capturing the data distribution so that new points can be sampled to improve few-shot classification accuracy (Park et al., 2020), (Wang et al., 2019), (Chen et al., 2019b), (Zhang et al., 2019). They have typically relied on complex models and loss functions to understand data from only a few examples, which limits the interpretability of the model and hampers generalization. Representation learning, on the other hand, aims at identifying feature transformations which can allow simple statistical techniques like nearest neighbor and bayesian classification generalize on few-shot tasks for novel classes (Chen et al., 2020), (Xue & Wang, 2020). Starting from early works (Miller et al., 2000) and building to recent methodologies (Yang et al., 2021), (Zhang et al., 2021b), representation learning strategies have relied on simple statistical assumptions that may not hold across diverse datasets.
Recently, (Yang et al., 2021) showed that classes which are semantically similar in meaning are also correlated in the means and covariances of their feature distributions. They used the statistics of classes with plentiful datapoints (called base classes) to learn the distribution of classes with a few datapoints (called novel classes). Despite some limiting assumptions, their method, called Distribution Calibration, outperformed much more complex models that relied on non-parametric optimization and generative models (Park et al., 2020), (Wang et al., 2019), (Chen et al., 2019b), (Zhang et al., 2019). Building on this idea and inspired by traditional statistics, we present a rigorous generalized sampling method for Few-Shot Learning, called DC+ that outperforms existing stateof-the-art classification accuracy without introducing additional statistical assumptions or complex generative models. Our main contributions are:
1. Introducing a principled approach to estimate novel class mean and covariance from the moments of a random variable weighted by the distance between the novel and the base classes,
2. Incorporating the statistical technique of variance shrinkage, which not only helps increase the accuracy but also stabilizes covariance estimation in cases when the feature extractor is over-parametrized (a common occurrence in modern deep learning models),
3. Extending the applicability of the statistical sampling approach to arbitrary feature extractors by introducing general Gaussianization transformations,
4. Presenting a single scaling parameter in Euclidean distance weighting that mitigates the need to search among Euclidean, Mahalanobis, and generalized distances for novel class estimation, and
Combined, our contributions put statistical sampling approach on a sound foundation, close open questions in earlier research, and demonstrate 3% to 5% improvement over competitive states-ofthe-art for 5way-1shot and 5way-5shot tasks for miniImageNet (Ravi & Larochelle, 2017), CUB (Welinder et al., 2010), StanfordDogs (Khosla et al., 2011), highest level classes of tieredImagenet (Ren et al., 2018) and 1% improvement Cross Domain 5way-1shot task of miniImagenet −→ CUB.

heading: 2 RELATED WORKS
text: Learning good features or manipulating the features to help generalize few-shot tasks is an active research area (Hou et al., 2019) (Hao et al., 2019), (Li et al., 2019a). Miller et al. (2000) proposed a congealing process which learned a joint distribution among all available classes. This distribution could then be used as a prior knowledge for constructing efficient few-shot classifiers. Chen et al. (2020) showed that by pre-training first on entire base classes, few shot classification accuracies on novel classes could be improved with a simple meta-learning on nearest-centroid based classification algorithm. Their main focus was on the pretraining methods which could improve a cosine based classifier on the centroids of the extracted features. DC+ can be applied on top of these feature extractor techniques to further explore improvements.
To correct bias in centroid or prototype estimations, several rectification methods were proposed— RestoreNet (Xue & Wang, 2020) transforms the feature space to move the images closer to their true centroids. Our proposed method approximates this transformation with scaled Euclidean distances and weighted random variables for novel classes, without any additional learnable parameters. Zhang et al. (2021a) proposed a 4-step method consisting of learning base class details as priors and then using these priors for correcting bias in novel prototype estimation. They then jointly fine tuned the feature extractor and bias corrector. Our method does not have this multi-step process and works with off-the-shelf feature extractors. Liu et al. (2020c) attempted to reduce the bias in distance estimation through reducing intra-class bias by label propagation and cross-class bias through shifting features. Their method works in the transductive setting where entire data, including the query set is consumed without label information. Our method does not need additional unlabelled data from the query set. Again their method improves upon Prototypical Networks (Snell et al., 2017) whereas we show that our method can be applied with any feature extractor.
DC+ can be broadly categorized as a data augmentation method. Several methods have been proposed earlier in this space. Antoniou & Storkey (2019) learn few-shot tasks by randomly labelling a subset of images and then augmenting this subset with different techniques like random crops, flips etc. Park et al. (2020) tried to transfer variance between different classes in order to simulate the query examples that can be encountered during test. Other works like Wang et al. (2019) and Liu et al. (2020b) utilized the intra-class variance to perform augmentation. These methods leverage complex neural networks with large number of learnable parameters to generate new examples. Our method does not require any additional learnable parameter and uses simple statistical techniques while still outperforming all previous deep learning methods.
Closest to DC+, Yang et al. (2021) estimated the novel class distributions based on their similarity with the base classes. Their method implicitly assumed that the base classes were semantically independent of each other when constructing covariance estimates, did not consider the similarity strength between the base and novel classes when estimating novel class statistics, and could not be applied to arbitrary off-the-shelf feature extractors (with activation functions different from relu)
and large feature dimensions often capable of producing ill-defined covariances. Our method does not make independence assumptions, leverages similarity information in the base classes, and can be applied to any off-the-shelf feature extractor.

heading: 3 PROPOSED APPROACH

heading: 3.1 PROBLEM DEFINITION
text: Few-shot classification problems are defined as Nway-Kshot classification tasks T (Vinyals et al., 2016) where given a small support set S of features x̃ and labels y, S = {(x̃i, yi)}N×Ki=1 , x̃i ∈ Rd, yi ∈ C, consisting of K points from N classes, the model should correctly classify a query set Q = {(x̃i, yi)}N×K+N×qi=N×K+1 with q points from each of the N classes in the support set. The entire dataset D, is divided into Cb base, Cv validation and Cn novel classes such that Cb ∩ Cv ∩ Cn = φ and Cb ∪ Cv ∪ Cn = C. The goal is to train a model with tasks T sampled from Cb and use Cv for hyperparameter tuning, where each task T is an Nway-Kshot classification problem on N unique classes of the set under consideration, for example base, validation set Cb,Cv here. The performance of few-shot learning algorithms is reported as the average accuracy on the query set Q of tasks T sampled from Cn.

heading: 3.2 ALGORITHM
text: Our proposed methodology, DC+, is outlined in Algorithm 1. In the following subsections, we incrementally go through the steps of DC+ in detail.

heading: 3.2.1 GAUSSIANIZATION OF THE DATA
text: Following Yang et al. (2021), our sampling methodology assumes that the input features follow a multivariate normal distribution. There are many methods of data gaussianization like Tukey’s Ladder of Powers (Tukey, 1977), Yeo-Johnson Transformation (Weisberg, 2001), and Iterative Gaussianization (Chen & Gopinath, 2000). In our experiments, we observed that Tukey’s Ladder of Powers outperformed other methods, but the fractional powers and log transform could only be applied to non-negative features (feature extractors which have relu and equivalent activation functions in their final layers). Expanding the applicability of the sampling method to arbitrary feature extractors and activation functions in deep learning models, we make a choice on the transformation of input features denoted by random variable x as per equation 1 below,
x̂ = { tukey(x) if x ≥ 0 always yeo johnson(x) otherwise
(1)
We give the definitions of tukey and yeo johnson in Appendix A.

heading: 3.2.2 PROPOSED RANDOM VARIABLE
text: We extrapolate the distribution of a given novel class as a weighted average of the distributions of k closest base classes. Formally, if x̃ ∈ Rd is a d dimensional support point from a novel class, and Xi ∈ Rd is a random variable denoting points of base class i, then we compose a random variable X′ representing our estimate of that novel class as
X′ = x̃ +
∑ i∈Sk wiXi
1 + ∑ i∈Sk wi , (2)
where wi are the weights assigned to the closest base classes in Sk. The associated mean and the covariance of this random variable are (since x̃ is a constant vector, it does not affect the covariance in Σ′),
µ′ = x̃ +
∑ i∈Sk wiµi
1 + ∑ i∈Sk wi
Σ′ = cov(X′) (3)
where µi = E[Xi]. There are many ways of estimating wi. One of the simplest is to look at the distance of the novel point from the base classes. In particular, we find the k closest base classes to
x̃ in Sk,
di = ||x̃− µi||2, i ∈ Cb (4) Sk = {i| − di ∈ topk(−di), i ∈ Cb} (5)
Based on di (for alternative distance formulations, see Appendix B), we construct the weights wi of each base class in Sk as,
wi = 1
1 + dmi , i ∈ Sk (6)
where m is a hyperparameter that helps in decaying wi as a function of di and gives us control in the relative weights of the classes in Sk. This form of weighted variable estimation is reminiscent of (though not the same as) inverse distance weighting, which is widely used to estimate unknown functions at interpolated points (Shepard, 1968), (Łukaszyk, 2004) It is worth noting that limdi→0 wi = 1. Hence as the base class i moves closer to the support point x̃, it gains increasing weight until µi overlaps with x̃, at which point the weight is 1, same as DC method.

heading: 3.2.3 SHRINKING THE COVARIANCE
text: When the number of data points in base class Xi is less than the feature dimensions, i.e. |Xi| < d, cov(Xi) = Σi is in general non-invertible, which prohibits constructing a normal distribution with Σ′ and poses a serious limitation since most off-the-shelf deep learning feature extractors have large feature dimensions (Zagoruyko & Komodakis, 2016), (Szegedy et al., 2015). We propose a variant of covariance shrinkage (Van Ness, 1980), (Friedman, 1989) to stabilize the Σ′ in equation 3 against singularities by introducing two hyperparameters that dictate the relative strength of the dimension variances and the off-diagonal covariance interactions as,
Σ′s = Σ ′ + α1σ1I + α2σ2(1− I), (7)
where σ1 is the average diagonal variance and σ2 is the average off-diagonal covariance of Σ′.

heading: 3.2.4 SAMPLING THE NOVEL CLASS
text: With µ′,Σ′s now formulated, we have both the mean and covariance to represent the ground truth distribution of the novel class associated with the support point x̃. Hence we can sample n points from this extrapolated distribution and append them to the existing support set,
Dy = {(x, y)|x ∼ N (µ′,Σ′s)} (8)
where y denotes the class of x̃. Steps from Section 3.2.1 to 3.2.4 are repeated for every point x̃ in the support set S, which has N classes with K points in each class constituting an Nway-Kshot classification task T . Hence the total dataset D after augmentation can be written as,
D = ∪{Dy∀(x̃, y) ∈ S} ∪ S (9)
D can be used to construct a classifier like Logistic Regression, SVM etc. and the performance is reported as the average accuracy on the query set Q in T .

heading: 4 EXPERIMENTS
text: In this section we compare the performance of DC+ with other states-of-the-art, show that our sampled points are closer to query data than DC, give theoretical insights with empirical results on the generalization improvement of DC+, and perform an ablation study to show the effectiveness of each component in our method.

heading: 4.1 IMPLEMENTATION DETAILS

heading: 4.1.1 DATASETS
text: We compare our proposed method with existing states-of-the-art on miniImagenet (Ravi & Larochelle, 2017), CUB (Welinder et al., 2010) and Stanford Dogs (Khosla et al., 2011). The details of the train/test/validation splits of each dataset is given in Appendix D. To show that our
Algorithm 1 DC+: Generalized Sampling Method for Few-shot learning Require: Base class features Xi ∈ Rd, i ∈ Cb Require: Support set features S = {(x̃j , yj)}N×Kj=1 : x̃j ∈ Rd, yj ∈ Cn
for (x̃j , yj) ∈ S do Gaussianize x̃ with equation 1 Find the nearest k base classes using di, Sk (equation 4, 5) Calculate weights wi for each base class i ∈ Sk (equation 6) Use wi to find the weighted random variable X′ (equation 2) Calculate µ′,Σ′ (equation 3) Shrink Σ′ to get Σ′s (equation 7) Sample features for class yj using {(x, yj)|x ∼ N (µ′,Σ′s)} (equation 8) end for Construct D by appending all sampled features to the support set S (equation 9) Train a logistic regression classifier on D (Details in Appendix C) Report the accuracy on query set Q = {(x̃j , yj)}N×K+N×qj=N×K+1
method gives superior performance when the base classes are dissimilar to the novel classes, we evaluate our method on Cross Domain dataset by training on tasks sampled from one distribution and evaluating on a different distribution. Specifically, we follow Patacchiola et al. (2020) and show results on miniImagenet −→ CUB, i.e. train split from miniImagenet and test/validation split from CUB. We also compare our method with DC (Yang et al., 2021) on a meta-tieredImagenet dataset of the 34 broad categories from tieredImagenet, split into 20 base, 8 novel and 6 validation classes, as laid out in Ren et al. (2018). Note that there is a high dissimilarity between the base and novel/validation classes in this meta-tieredImagenet as seen in Table 6 of Appendix D.

heading: 4.1.2 FEATURE EXTRACTOR
text: We used a WRN-28-10 (Zagoruyko & Komodakis, 2016) feature extractor trained using S2M2 Method (Mangla et al., 2020) for miniImagenet, CUB, Stanford Dogs and meta-tieredImagenet experiments. For our cross domain results on miniImagenet −→ CUB, we used a Conv-4 backbone to compare our results with other states-of-the-art as done in Patacchiola et al. (2020). Details of our feature extractor training can be found in Appendix E.
The extracted features were 640 dimensional for miniImagenet and CUB which had 600 and 44 points in each base class. Feature dimensions were 1600 for miniImagenet −→ CUB experiments with 600 points in each base class. Note that such mismatch between the feature dimensions and the number of datapoints leads to singularities when trying to estimate the covariance matrix. Our shrinkage method takes care of these pathological situations. To show that our proposed algorithm works equally well for non-singular cases, we extracted 64 dimensional representation for Stanford Dogs and meta-tieredImagenet which had 148 and 12950 points in each of their base classes.
To demonstrate that our gaussianization approach (Section 3.2.1) generalizes, our features in miniImagenet and CUB come from a penultimate relu layer hence are all non negative. For Stanford Dogs and meta-tieredImagenet, we project 640 dimensional features from WRN-28-10 to 64 dimensions in the penultimate layer and hence the features span in both positive and negative regions.

heading: 4.1.3 HYPERPARAMETER SEARCH
text: To circumvent combinatorial explosion from grid search of a larger number of hyperparameters, we used optuna (Akiba et al., 2019) library for tuning our hyperparameters β,m, k, α1, α2, n. All tunings are 1000 trials long with a Median Pruner. We give complete details of our optuna setting and picking the best hyperparameters in Appendix F.

heading: 4.2 COMPARISON WITH STATES-OF-THE-ART
text: Tables 1 and 2 summarize the performance of our proposed DC+ with existing states-of-the-arts. We report the average classification accuracy of 5000 random tasks T sampled from the novel classes
Cn along with their 95% confidence intervals. The exact value of each hyperparameter that give the reported accuracy along with other candidate hyperparameters are given in Appendix G.
We observe that a logistic regression constructed on our sampled data achieves consistent performance improvements of 3% (CUB 5way-5shot) to 5% (CUB 5way-1shot) compared to the 2nd best method. We also achieve 1% improvement on challenging cross domain 5way-1shot classification of miniImagenet −→ CUB. Finally, for the case when base classes and novel classes share very little similarity (as is the case for the meta-tieredImagenet), we outperform DC by more than 9% for 5way-5shot task as shown in Table 3.
Note that these accuracy improvements are derived from a simple statistical model that leverages just 3 additional hyperparameters compared to the 2nd best method of DC Yang et al. (2021). We also do not make any assumptions about the feature values, the feature dimensions, and show these improvements across different cases of singular (miniImagenet, CUB, miniImagenet −→ CUB) and non-singular covariances (Stanford Dogs, meta-tieredImagenet).

heading: 4.3 VISUALIZATION OF SAMPLED POINTS
text: In Figure 1 we compare the t-SNE representation (van der Maaten & Hinton, 2008) of sampled points for both DC (Yang et al., 2021) and our method DC+ to visualize whether our sampled points are closer to the ground truth as indicated quantitatively from results in Tables 1, 2, 3. We can see that our method produces clusters of sampled points which are more compact and overlap with more query points than the sampled points of DC. We analyse this in the next section 4.4. For more visualizations refer to Appendix H.
4.4 WHY SHOULD DC+ GENERALIZE BETTER THAN DC?
To motivate the generalization and performance improvements of DC+, we start with a comparison of the weighting scheme between DC and DC+. Figure 2 (left) shows the estimated position of X′ as the closest base class moves away from x̃ for the simple case of one base class. The unweighted random variable X′ in DC can be written as X′ = (x̃+ ∑ i∈Sk Xi)/(1+k) = (x̃+X1)/2, for k=1.
Hence as µi = E(X1) moves away from x̃, X′ moves further apart in DC as seen in Figure 2. Thus even though the method borrows stable values of mean and covariance from the base class in question, the values themselves are far from the support point, which acts as a proxy location for the query points during training. In contrast, with a higher m in DC+, the weight wi assigned to base class i drops polynomially with distance from x̃ and X′ saturates at x̃ when the base class is far from x̃. Hence our sampled points should be much closer to x̃ giving us better generalization than DC.
To confirm this hypothesis, we visualize the sampled points for k = 1 in miniImagenet −→ CUB experiment in Figure 2. We chose this dataset to also show how a higher m can help offset noise from base classes by minimizing their wi in estimating the distribution X′ when they are not similar to the novel class of x̃. We can see that points produced by DC andm = 0, 1 are far from the support point x̃. As m increases, the cluster moves closer to x̃ where at m = 4 the cluster overlaps with x̃. Our peformance improvement of 5% on 5way-1shot and 4% on 5way-5shot tasks of Cross Domain miniImagenet −→ CUB experiments compared to DC further confirms this hypothesis. This same trend can also be seen in the results of meta-tieredImagenet in Table 3 where DC+ converges on a high value ofm (m = 2.25 in 5way-1shot andm = 2 in 5way-5shot) giving weights to base classes that are 10 to 100 smaller than miniImagenet, and 100 times smaller than DC. We further analyse this case and visualize the histogram of these weights comparing them with DC, and weights from miniImagenet experiments in Appendix I.

heading: 4.5 ABLATION STUDY
text: We ran a 5way-1shot ablation study on the Stanford Dogs dataset to estimate the effect of each hyperparameter of our method. For each experiment, the hyperparameter range were defined as,
m ∈[low = 0, high = 3, step = 0.25] k ∈[low = 1, high = 20, step = 1] α1 ∈[low = 0, high = 600, step = 100] α2 ∈[low = 0× α1, high = 600× α1, step = 100]
We fixed n = 750, and the total number of trials in our optuna hyperparameter search to 100 so that the accuracy improvements with the addition of each hyperparameter were independent of the search time. For each trial, the number of tasks T was also fixed to 200. Table 4 summarizes the results of our study and shows that each component of DC+ is important in giving improvements over state-of-the-art. For the Gaussianization step, the final logistic regression model was trained on one data point from each novel class in a 5way classification task. Notice that β = 1 in our study implies that raw features from 64 dimensional linear layer give best accuracy. The results for m = 1 show that introducing weighted random variables to model novel class distributions helps even without tuningm. Further control on decayingwi as a function of di as discussed in Section 4.4 with a tuned m helps in achieving state-of-the-art.
4.5.1 EFFECT OF k
We investigated whether the weight decay parameter m can remove the need for the hyperparameter k, since increasing m reduces the weight wi of a base class that is farther from a given novel point. In Table 4 we see that the best k equals 10 for the case when all hyperparameters are tuned together. This indicates that k and m work together to estimate novel class statistics (otherwise the best value would be k = 30 = |Cb|, number of base classes in Stanford Dogs dataset). To see how much the introduction of k is helping the accuracy, we tuned all hyperparameters in Stanford Dogs 5way1shot task and find that without k, the accuracy was 63.46% with m = 1.5, α1 = 600, α2 = 0. This was 0.9% lower than the best result with all k, α1, α2,m tuned in the last row of Table 4.
4.5.2 EFFECT OF n, α1, α2
The value of n, the number of sampled points, was fixed to 750 in all our experiments. Figure 3 shows the trend of accuracy v/s n in our ablation study on Stanford Dogs 5way-1shot task. We can see that the accuracy keeps increasing with slight noise until n = 750 after which it saturates. It indicates that the sampling more points after a certain number does not cover additional query points in this specific feature space.
From Table 4 we see that α1, α2 play a key role in improving accuracy. In Figure 3 the heatmap of accuracy versus α1 and α2 shows that accuracy increases with increasing α1, α2 and then starts dropping beyond α1 > 200. This could be due to overlapping clusters of sampled points with a higher covariance, where the decision boundaries start overlapping.

heading: 5 CONCLUSION
text: In this work, we have proposed a principled approach to estimate novel class distributions by formulating a similarity-based weighted random variable of closest base classes. We showed that incorporating statistical techniques of covariance shrinkage and Gaussianization not only generalize our method (DC+) to arbitrary pretrained feature extractors, but also increase the accuracy over state-of-the-art significantly. Our experiments effectively demonstrate cumulative performance improvements of 1% to 9% over DC including challenging cross domain tasks. Exploring the trade-off between more hyperparameters and accuracy along with generalizations to different tasks like nonGaussian distributions can be productive avenues to pursue.

heading: A TUKEY AND YEO-JOHNSON
text: Tukey transformation on a random vector x with parameter β is defined as,
tukey(x) = { xβ β 6= 0 log(x) β = 0
(10)
Yeo-Johnson transformation on a random vector x with parameter β is defined as,
yeo johnson(x) =  ((x + 1)β − 1)/β β 6= 0,x ≥ 0 log(x + 1) β = 0,x ≥ 0 −[(−x + 1)2−β − 1]/(2− β) β 6= 2,x < 0 − log(−x + 1) β = 2,x < 0
(11)

heading: B EFFECT OF DISTANCES
text: For normally distributed features, the probability of a novel point x̃ being in class i with distribution N (µi,Σi) can be written quite generally as,
p(class = i|x̃) ≈ p(i) 1√ | Σi | exp −1 2 (x̃− µi)TΣ−1i (x̃− µi), (12)
Taking ln on both sides of this probability, and assuming that each base class is equally probable in priors p(i), we get a natural distance between x̃ and µi as:
di = ln | Σi |+ (x̃− µi)TΣ−1i (x̃− µi) (13)
equation 13 is the squared Mahalanobis distance between x̃ and µi along with an added ln term. Replacing Σ−1i = I for all i, our di reduces to squared Euclidean distance,
di = (x̃− µi)T (x̃− µi) (14)
The above formulations assume that both x̃,µi come from the same distribution. Keeping this in mind, we also derive a metric called squared δ distance with δ as a hyperparameter in Formulation 1 (Section B.1) as,
di = (Ix̃− δIµi)T (Ix̃− δIµi) (15)
Experimenting with these different distance metrics, in Table 5, we observe no advantage of using Mahalanobis distance or squared δ distance over a simpler Euclidean distance when defining di in Section 3.2.2. We also see that the scaled Euclidean with m as a hyperparameter gives improved accuracy compared to Euclidean with m = 1. Hence, we propose using our scaled Euclidean distance (equation 6) instead of a searching for multiple distance metrics in the feature space to improve accuracy.
B.1 FORMULATION 1: GENERAL DISTANCE: SQUARED δ
Consider two points x1,x2 ∈ Rd sampled from different multivariate distributions N (µ1,Σ1),N (µ1,Σ1),µ1,µ2 ∈ Rd,Σ1,Σ2 ∈ Rd×d. Multiple distance measures can be constructed between the populations, e.g., the Bhattacharya distance (Bhattacharya, 1943). Here, we consider a general distance of the form
d = (L−1x1 −M−1x2)T (L−1x1 −M−1x2) (16) where, LLT = Σ1, MM T = Σ2
Here L,M ∈ Rd×d are the cholesky decompositions of Σ1,Σ2 respectively. If Σ1 = Σ2, equation 16 reduces to squared Mahalanobis distance. If x1 is the support point x̃ and x2 is the mean of base class Xi, then Σ2 = Σi =covariance of the base class i. The covariance of the distribution of x̃ is unknown since we are trying to estimate it. Assuming Σ1 = ψI, i.e. a diagonal covariance, we get,
di =(L −1x̃−M−1µi)T (L−1x̃−M−1µi) =( √ ψIx̃−M−1µi)T ( √ ψIx̃−M−1µi)
where, LLT = ψI, MMT = Σi
From our experiments we observed that the off-diagonal covariances in the base classes of miniImagenet, CUB and Stanford Dogs, were at least 2 orders of magnitudes smaller than the diagonal variance. Hence assuming Σi = σI, i.e. a diagonal matrix with constant variance, we get,
di =( √ ψIx̃− √ σIµi) T ( √ ψIx̃− √ σIµi)
=ψ(Ix̃− √ σ√ ψ Iµi) T (Ix̃− √ σ√ ψ Iµi) =ψ(Ix̃− δIµi)T (Ix̃− δIµi) , where δ = √ σ√ ψ
(17)
Since ψ is common to all base classes Xi, it does not affect our closest base class calculation. Hence dropping we ψ we get a general distance as,
di = (Ix̃− δIµi)T (Ix̃− δIµi) (18)
where δ is a hyperparameter that can be optimized for accuracy.

heading: C DETAILS OF LOGISTIC REGRESSION
text: We performed Logistic Regression using torch library. For all datasets– miniImagenet, CUB, StanfordDogs and for all experiments, the following hyperparameters were used,
batch size = 1024, epochs = 200,
learning rate = 0.08, optimizer = Stochastic Gradient Descent (torch.optim.SGD), scheduler = None,
Loss Function = Cross Entropy (torch.nn.CrossEntropyLoss), Loss Regularization = None

heading: D DATASETS
text: miniImagenet (Ravi & Larochelle, 2017) is derived from the ILSVRC-12 dataset (Russakovsky et al., 2014). It contains 100 different classes with 600 examples per class. The images sizes are 84× 84× 3. We follow the train, validation, test split of 64 base, 16 validation and 20 novel classes as done in previous work of Ravi & Larochelle (2017).
CUB (Welinder et al., 2010) is a fine grained classification dataset consisting of different bird species. Each class has varying number of examples in this dataset so we take the minimum available number of 44 examples from each of 200 classes. The image sizes are again 84 × 84 × 3. Following Chen et al. (2019a) we split train, validation, test as 100 base, 50 validation and 50 novel classes respectively.
Stanford Dogs (Khosla et al., 2011) is another fine grained classification dataset of dogs species derived from ILSVRC-12 dataset. Here again each class has varying number of points so we take 100 points from each of 120 classes. Following existing state-of-the-art results of Chen et al. (2021), we use train, validation and test splits of 70 base, 20 validation and 30 novel classes respectively.
Cross Domain datasets: To show that our method gives superior performance even when the base classes are dissimilar to the novel classes, we evaluate our proposed method by training on tasks sampled from one distribution and evaluating on a different distribution. Specifically, we follow Patacchiola et al. (2020) and show results on miniImagenet −→ CUB, i.e. train split from miniImagenet and test/val split from CUB. We also compare our method against DC (Yang et al., 2021) on a meta-tieredImagenet of 34 broad categories from tieredImagenet, split into 20 base, 8 novel and 6 validation classes, as laid out in Ren et al. (2018). Note that there is a high dissimilarity between the base and novel/validation classes in this meta-tieredImagenet as seen in Table 6.

heading: E FEATURE EXTRACTOR BACKBONE
text: We used S2M2 Method (Mangla et al., 2020) to train a WRN-28-10 (Zagoruyko & Komodakis, 2016) feature extractor for miniImagenet, CUB, Stanford Dogs and meta-tieredImagenet. The backbone was first allowed to overfit for 400 epochs on all base classes to minimize the classification + self supervised rotation loss. Next the decision boundaries were smoothened using the validation classes with a cosine classifier on the feature extractor until the loss on validation classes stopped improving. For our cross domain results on miniImagenet −→ CUB, we used identical setting as (Patacchiola et al., 2020) with a Conv-4 backbone.

heading: F HYPERPARAMETER SEARCH METHODOLOGY
text: We used optuna (Akiba et al., 2019) library to tune our hyperparameters β,m, k, α1, α2, n. For all datasets (miniImagenet, CUB, Stanford Dogs, meta-tieredImagenet and miniImagenet −→ CUB) the search space for β,m, k, n was,
β ∈ [low = 0, high = 10, step = 0.25], m ∈ [low = 0, high = 3, step = 0.25], k ∈ [low = 2, high = |Cb|, step = 2], n ∈ [low = 100, high = 1000, step = 50]
where |Cb| denotes the number of base classes. For miniImagenet and CUB we set
α1 ∈ [low = 0, high = 10000, step = 1000], α2 ∈ {0, 0.1, 1, 10, 100} × α1 whereas for Stanford Dogs, meta-tieredImagenet, and miniImagenet −→ CUB,
α1 ∈ [low = 0, high = 1000, step = 100], α2 ∈ [low = 0, high = 1000, step = 100]× α1 Note that a larger range of α2 was needed in Stanford Dogs owing to the average off-diagonal covariance of Σ′ (equation 3) being 2 orders of magnitude smaller than the average variance.
All hyperparameters were jointly tuned using TPESampler in optuna. For each hyperparameter setting sampled, we evaluated its average accuracy over 200 random tasks T sampled from the validation classes Cv . During this phase, we pruned any hyperparameter setting which had less than median accuracy after 100 runs using MedianPruner. Once every hyperparameter setting was validated, we picked the top-3 candidates from this experiment and ran these specific hyperparameters for 5000 random tasks sampled from the novel classes, i.e. T ∼ Cn. We report our accuracy and mean confidence of the best candidate in comparison with state-of-the-art in Tables 1, 2. Accuracies of all 3 candidates can be found in Appendix G.

heading: G TUNED HYPERPARAMETERS
text: In Table 7, we give the accuracy of each of the top 3 candidates from our optuna hyperparameter search. Note than the first row of every setting (5way-1shot or 5way-5shot) is the result we used for comparing to other methodologies in Tables 1, 2.

heading: H ADDITIONAL T-SNE VISUALIZATION

heading: I META-TIEREDIMAGENET
text: We used the same S2M2 method for training the backbone feature extractor, the feature dimensions were set to 64 with no activation function. Hyperparameter search ranges were,
β ∈ [low = 0, high = 10, step = 0.25], m ∈ [low = 0, high = 3, step = 0.25], k ∈ [low = 2, high = |Cb|, step = 2] α1 ∈ [0, 1000, step = 100], α2 ∈ [0, 1000, step = 100]× α1
In Table 3, we see that DC+ also outperforms DC and baseline (no method) on meta-tieredImagenet even when there is a high dissimilarity between the base and novel/validation classes. The hyperparameters corresponding to these results were,
DC+ 5way-1shot m = 2.25, k = 8, α1 = 100, α2 = 100, β = 1, n = 750 DC+ 5way-5shot m = 2, k = 10, α1 = 100, α2 = 100, β = 1, n = 750 DC 5way-1shot k = 2, α = 1000, β = 1, n = 750 DC 5way-5shot k = 2, α = 900, β = 1, n = 750
An explanation as to why our method outperforms DC on this meta dataset is because of the high m (m = 2.25 in 5way-1shot and m = 2 in 5way-5shot) our method discovers. With such high m,

heading: DC [0.33, 0.33] [0.33, 0.33]
text: our wi is 100 times smaller than the weights assigned by DC method as seen in Figure 5. Hence very small contribution of the k base classes are extrapolated in DC+ method while calculating the calibrated µ′,Σ′, compared to DC where a hard contribution of 1/(k+1) exists.
Figure 5 and Table 8 shows the weights of the k base classes calculated by our method DC+ and DC method averaged over 5 random selection of 5way-K shot task (random selection of novel classes and support points)
To put these wi into perspective, the wi for best hyperparameters in miniImagenet are 10 times larger as shown in Figure 5 and Table 9, (m = 1, k = 8 for 5way-1shot, and m = 3, k = 30 for 5way-5shot) showing that the classes are much more similar in miniImagenet than in the current meta-tieredImagenet.

Review summaries:
This paper suggests a data augmentation method for few-shot learning. They generalize the similar method, DC (Yang et al., 2021) introducing a couple of hyperparameters to tune. Like DC, without introducing any learnable parameters, they suggest a way to augment few-shot data by extrapolating its distribution base on weighted empirical distributions of base classes. In addition, in a case when the number of data is smaller, they suggest a covariance shrinkage technique. The suggested method is empirically supported with experiments where the suggested method outperforms other baselines.  (Yang et al., 2021) Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id= JWOiYxMG92s

This paper presents a statistical sampling method to augment related extrapolated features to the novel class few-shot samples by constructing extrapolated distribution with sampled related base classes to construct much robust classifier for few-shot classification. The paper is an improved version over a previous sampling method DC for few-shot learning by Yang et al 2021 addressing the issue of strength of semantic similarity and the gaussianization technique of features for activation functions with negative outputs other than relu. The first of the above issues are dealt with a weighting scheme to have weighted average of sampled base class features along with the novel class features and consequently estimating the associated mean and the covariance for each novel class. The second issue of gaussianization for activation for negative values are handled with Yeo-Johnson transformation while keeping the same Tukey transformation for the non-negative features. Also, the covariance shrinkage method is adopted for dealing with non-singular covariance matrix formed from ill conditioned feature space with feature dimension number higher than the number of samples per class. Finally, the additional features are augmented from this extrapolated distribution per class to get rid of low data problem and simple logistic regression is utilized and classification results are reported on query samples.

The authors propose a statistical method for estimating the data distribution for few-shot learning. The method is built on a previous work (DC) with minor fixes but have non-trivial performance improvements. Specifically, the main differences lie in the transformation function, weighted random variables and the covariance shrinkage.