Free Lunch for Few-shot Learning:  Distribution Calibration
Authors: Shuo Yang
Lu Liu
Min Xu
Venue: ICLR 2021 Oral (2021)
Keywords: few-shot learning
image classification
distribution estimation

Abstract: Learning from a limited number of samples is challenging since the learned model can easily become overfitted based on the biased distribution formed by only a few training examples. In this paper, we calibrate the distribution of these few-sample classes by transferring statistics from the classes with sufficient examples. Then an adequate number of examples can be sampled from the calibrated distribution to expand the inputs to the classifier. We assume every dimension in the feature representation follows a Gaussian distribution so that the mean and the variance of the distribution can borrow from that of similar classes whose statistics are better estimated with an adequate number of samples. Our method can be built on top of off-the-shelf pretrained feature extractors and classification models without extra parameters. We show that a simple logistic regression classifier trained using the features sampled from our calibrated distribution can outperform the state-of-the-art accuracy on three datasets (~5% improvement on miniImageNet compared to the next best). The visualization of these generated features demonstrates that our calibrated distribution is an accurate estimation. 

Published as a conference paper at ICLR 2021
FREE LUNCH FOR FEW-SHOT LEARNING: DISTRIBUTION CALIBRATION
Shuo Yang', Lu Liu², Min Xu
"choool of Electrical and Data Engineering, University of Technology Sydney,
2 AAstrrllian Artificial Intelligence Institute, University of Technology Sydney
{shuo. yang, 1u. 1el-lll00p00 @etuddettttttttttteee uts. edu. au, min. xw@utsuuu....... edu. au,
ABSTRACT
Learning from a limited number of samples is challenging since the learned model can easily become overfitted based on the biased distribution formed by only a few training examples. In this paper, we calibrate the distribution of these few- sample classes by transferring statisties from the classes with sufficient examples. Then an adequate number of examples can be sampled from the calibrated dis- tribution expand the inputs the classifier. We assume every dimension in the feature representation follows a Gaussian distribution so that the mean and the variance of the distribution can borrow from that of similar classes whose statistics are better estimated with an adequate number of samples. Our method can be built on top of off-the-shelf pretrained feature extractors and classifica- tion models without extra parameters. We show that a simple logistic regression classifier trained using the features sampled from our calibrated distribution can outperform the ttate-ffthhe-rt accuracy on three datasets (5% improvement on minilmageNet compared to the next best). The visualization of these generated features demonstrates that our calibrated distribution is an accurate estimation. The code is available at: hht.ts:: /ttp::///tttiii higgtb.thuu....g.g com SSt//hhhhhannnaaa 1998/5 / Few Shot Distri ion Ca Callbaaa wwl.ibraatio
INTRODUCTION
Learning from limited number of training samples has drawn in- creasing attention due to the high cost of collecting and annotating a large amount of data. Researchers have developed algorithms to improve the performance of models that have been trained with very few data. Finn et al. (2017); Snell et al. (2017) train mod- els in a meta-learning fashion that the model can adapt quickly on tasks with only a few training samples available. HHariharan & Girshick (2017); Wang al. (2018) try to synthesize data fea- tures by learning generative model to alleviate the data insufli- ciency problem. Ree al. (2018) propose to leverage unlabeled data and predict pseudo labels to improve the performance of few- shot learning.
Table 1: The class mean similar- ity ("mean sim") and class vari- ance similarity ("var sim") be- tween Arctic fox and different classes.
Arctic fox
mean sim
var sim
white wolf malamute
97%
97%
85%
78%
lion
%
70%
meerkat
78%
70%
jellyfish
46%
26%
While most previous works focus on developing stronger models, scant attention has been paid the property of the data itself. It is natural that when the number of data grows, the ground truth distribution can be more accurately uncovered. Models trained with a wide coverage of data can generalize well during evaluation. On the other hand, when training model with only few training data, the model tends to overfit on these few samples by minimizing the training loss over these samples. These phenomena are illustrated in Figure I This biased distribution based few examples can damage the generalization ability of the model since is far from mirroring the ground truth distribution from which test cases sampled during evaluation.
orange
40%
19%
beer bottle
34%
11%
"Corresoonding author.

---

Published as a conference paper at ICLR 2021
A
ground truth distribution
calibrated distribution
A
few-shot features
features sampled from calibrated distribution
class boundary
Classifier trained with few-shot features
Classifier trained with features sampled from calibrated distribution
Figure I: Training a classifier from few-shot features makes the classifier averfit to the few examples (Left). Classifier trained with features sampled from calibrated distribution has better generalization ability (Right).
Here, we consider calibrating this biased distribution into a more accurate approximation of the ground truth distribution. In this way, model trained with inputs sampled from the calibrated distribution can generalize over a broader range data from more accurate distribution rather than only fitting itself to those few samples. Instead of calibrating the distribution of the original data space, we try to calibrate the distribution in the feature space, which has much lower dimensions and easier to calibrate (Xian al. (2018)) We assume every dimension in the feature vectors follows a Gaussian distribution and observe that similar classes usually have similar mean and variance of the feature representations, as shown in Table [ Thus, the mean and variance of the Gaussian distribution can be transferred across similar classes (Salakhutdinov et al. (2012)). Mcanwhile, the statistics can be estimated more accurately when there are adequate samples for this class. Based on these observations, we reuse the statistics from many-shot classes and transfer them to better estimate the distribution of the few-shot classes according to their class similarity. More samples can be generated according the estimated distribution which provides sufficient supervision for training the classification model.
In the experiments, we show that simple logistic regression classifier trained with our strategy can achieve state-of-the-ar accuracy on three datasets. Our distribution calibration strategy can be paired with any classifier and feature extractor with no extra learnable parameters. Training with samples selected from the calibrated distribution can achieve 12% accuracy gain compared to the baseline which is only trained with the few samples given in I Ishot task. We also visualize the calibrated distribution and show that it is an accurate approximation of the ground truth that can better cover the cases.
RELATED WORKS
Few-shot classification is challenging machine learning problem and researchers have explored the idea of learning to learn or meta-lcarning to improve the quick adaptation ability to alleviate the few-shot challenge. One of the most general algorithms for meta-learning is the optimization- based algorithm. Finn et al. (2017) and Li et al. (2017) proposed to learn how to optimize the gradient descent procedure so that the learner can have good initialization, update direction. and learning rate. For the classification problem, researchers proposed simple but effective algorithms based metric learning. MatchingNet (Vinyals et al., 2016) and ProtoNet (Snell et al., 2017) learned to classify samples by comparing the distance to the representatives of each class. Our distribution calibration and feature sampling procedure does not include any learnable parameters and the classifier is trained in traditional supervised learning way.
Another line of algorithms is compensate for the insufficient number of available samples by generation. Most methods use the idea of Generative Adversarial Networks (GANs) (Goodfellow ct al. 2014) or autoencoder (Rumelhart et al.,) 1986) to generate samples (Zhang et al. (2018) Chen al. (2019b); Schwartz al. (2018); Gao et al. (2018) features (Xian et al. (201 8 Zhang et al. (2019) )) to augment the training set. Specifically, Zhang et al. (2018) and Xian et al, (2018) 88) proposed synthesize data by introducing adversarial generator conditioned on tasks. Zhang et al. (2019) tried to learn a variational autoencoder to approximate the distribution and predict labels based on the estimated statistics. The autoencoder can also augment samples by projecting between the visual
2

---

Published as a conference paper at ICLR 2021
space and the semantic space (Chen al. 2019b) or encoding the intra-class deformations (Schwartz jet al. 2018). Liu et al. (019bb) and Liu et al. (2019a) propose to generate features through the class hierarchy. While these methods can generate extra samples or features for training, they require the design of a complex model and loss function to learn how to generate. However, our distribution calibration strategy is simple and does not need extra learnable parameters.
Data augmentation is a traditional and effective way of increasing the number of training samples. Qin et al. (2020) and Antoniou & Storkey (2019) proposed the used of the traditional data augmen- tation technique to construct pretext tasks for unsupervised few-shot learning. Wang et al. (2018) and Hariharan Girshick (2017) leveraged the general idea of data augmentation, they designed a hallucination model to generate the augmented version of the image with different choices for the model's input, i.e., an image and a noise (Wang et al. 2018) or the concatenation of multiple features (Hariharan & Girshick 2017). Park et al. (2020): Wang et al. (2019); [Liu et al. (2020b) tried augment feature representations by leveraging intra-class variance. These methods learn to augment from the original samples their feature representation while we try to estimate the class- level distribution and thus can eliminate the inductive bias from a single sample and provide more diverse generations from the calibrated distribution.
3 MAIN APPROACH
In this section, we introduce the few-shot classification problem definition in Section 3.1 and details of our proposed approach in Section 3.2
3.1 PROBLEM DEFINITION
We follow a typical sew-shot classification setting. Given a dataset with data-label pairs D {( 9:}} where x; E R is the feature vector of sample and y; C, where C' denotes the set of classes. This set of classes is divided into base classes C₁ and novel classes Cn, where C. n C, = and C. U.C, C. The goal is to train model the data from the base classes that the model can generalize well on tasks sampled from the novel classes. In order to evaluate the fast adaptation ability or the generalization ability of the model, there each task T. The most common way to build a task is called an N-way-K-sha task (Vinyals et al.) (2166), where N classes are sampled from the novel set and only K (c.g.. or 5) labeled samples are provided for each class. The few available labeled data are called support set S {(x,, y1) ) K and the model is evaluated on another query set Q = {(x,, y)),, Nx))) K-xxK-X
only few available labeled samples for
xx]=xxx++1111 K+1 where every class in the task has g test cases. Thus, the performance of a model is evaluated the averaged accuracy on (the query set of) multiple tasks sampled from the novel classes.
3.2 DISTRIBUTION CALIBRATION
As introduced in Section 3.1 the base classes have sufficient amount of data while the evaluation tasks sampled from the novel classes only have a limited number of labeled samples. The statistics of the distribution for the base class based on few-shot samples, which is an ill-posed problem. As shown in Table [ we observe that if we assume the feature distribution is Gaussian, the mean and variance with respect to each class are correlated to the semantic similarity of each class. With this in mind, the statistics can be transferred from the base classes the novel classes if we learn how similar the two classes are. In the following sections, we discuss how we calibrate the distribution estimation of the classes with only few samples (Section 3.2.2) with the help of the statistics of the base classes (Section I...I). We will also elaborate on how do we leverage the calibrated distribution to improve the performance of few-shot learning (Section 3.2.3).
be estimated more accurately compared to the estimation
Note that our distribution calibration strategy is over the feature-level and is agnostic to any feature extractor. Thus, can be built on top of any pretrained feature extractors without further costly fine- tuning. In our experiments, we use the pretrained WideResNet Zagoruyko & Komodakis (2016) following previous work (Mangla et al. (2020)). The WideResNet is trained to classify the base classes, along with self-supervised pretext task to learn the general- purppose representations suit- able for image understanding tasks. Please refer to their paper for more details on training the feature extractor.
3

---

Published as a conference paper at ICLR 2021
Algorithm Training procedure for N-way-K-sho task
Require: Support set features (æi, y) NxK
i=1
Require: Base classes' statistics { [hhh )
1: Transform (z:) (z= = with Tukey's Ladder of Powers as Equation
2: for (It, yj) do
3: Calibrate mean 1ı' the covariance E' class y using at with Equation
4: Sample features for class ys from the calibrated distribution as Equation 7
5: end for
6: Train classifier using both support set features and sampled features as Equation
3.2.1 STATISTICS OF THE BASE CLASSES
We assume the feature distribution of base classes is Gaussian. The mean of the feature vector from a base class i is calculated as the mean of every single dimension in the vector:
Ø3
(1)
where æy is feature vector of the j-th sample from the base class and ny is the total number of samples in class i. As the feature vector zj is multi-dimensional, we use covariance for a better representation of the variance between any pair of elements in the feature vector. The covariance matrix Σ; for the features from class is calculated as:
Σ
Σ
[a
ms) (2, ms)²
(2)
3.2.2 CALIBRATING STATISTICS OF THE NOVEL CLASSES
Here, consider an N-way-Ke Sway-K-shot Khohot task sampled from the novel classes.
Tukey's Ladder of Powers Transformation
To make the feature distribution more Gaussian-like, we first transform the features of the support set and query set in the target task using Tukey's Ladder of Powers transformation (Tukey (977)). )-
Tukey's 's Ladder of Powers transformation is a family of power transformations which reduce the skewness of distributions and make distributions more Gaussian-like. Tukey's 's Ladder of Powers transformation is formulated as:
if + if X = 0
log(x)
where À is a recovered by setting X as 1. Decreasing makes the distribution less positively skewed and vice versa.
to adjust how to correct the distribution. The original feature can be
Calibration through statisties transfer
Using the statistics from the base classes introduced in Section 3.2. we transfer the statistics from the base classes which are estimated more accurately on sufficient data the novel classes. The transfer is based the Euclidean distance between the feature space of the novel classes and the mean of the features from the base classes μ4; as computed Equation [ Specifically, we select the top i base classes with the closest distance to the feature of sample z from the support set:
S = 11 - 2||2 i Co). SN = { [4: 2 E topk(Sa)).
(4)
(5)
where topk(-) is an operator select the top elements from the input distance set S.. Sy stores the k nearest base classes with respect to a feature vector æ. Then, the mean and covariance of the distribution is calibrated by the statistics from the nearest base classes:
n' Eiesy + x
Eiesy Σ; + a,
E
k: +
4

---

Published as a conference paper at ICLR 2021
Table 2: 5way shot and 5way5shor classification accuracy (%) mindmageNett and CUB with 95% confi- dence intervals. The numbers bold have intersecting confidence intervals with most accurate method.
miniImageNet
CUB 5way5shot
Methods
5waylshot
5way5shot
5way Ishot
Optimization-enl
MAML (Finn et al. 2017
48.70 + 1.84
63.10 = 0.92
50.45 + 0.97
59.60 + 0.84 67.59 1 0.82
Meta-SGD (Li et al 2017)
50.47 + 1.87
64.03 = 0.94
53.34 1 0.97
LEO (Ruuu
61.76 1 0.08
77.59 - 0.12
E3BM Liu al. 2020c
63.80 0.40
80.29 0.25
Melric-basse
Matching Net Vinyals al. 2016
43.56 0.84
55.31 - 0.73
56.53
1 0.99
63.54 4 0.85
Prototypical Net (Snell al.
2017
54.16 1.82
73.68 - 0.65
72.99
+ 0.88
86.64 +: 0.51
Baseline++ Chen et al. )
75.68 0.77
67.02 0.90
83.58 + 0.54
Variational Few-shot Zhang of al.
2019
0.22
0.11
Negative-Cosine Cosine et al. 2020a
62.33 - 0.82
80.94 = 0.59
72.66 + 0.85
89.40 + 0.43
Generation-based
MetaGAN (Zhang
(2018)
52.71 0.64
68.63 0.67
Delta-Encoder (Schwartz et al. (0188)
59.9
69.7
69.8
82.6
TriNet (Chen et al, (20195))
58.12 1.37
76.92 + 0.69
69.61 + 0.46
84.10 + 0.35
Meta Variance Transter (Park al. 20200)
67.67 = 0.70
0.70
80.33 + 0.61
Maximum Likelihood with (Our)))
66.91 0.17
80.74 0.48
77.22 0.14
89.58 1 0.27
SVM with DC (Ours)
67.31 + 0.83
82.30 = 0.34
79.49 0.33
90.26 1 0.98
Logistic Regression with DC (Ours)
68.57 + 0.55
82.88 0.42
79.56 + 0.87
90.67 + 0.35
where a is paramm-p that determines the degree of dispersion of features sampled from the calibrated distribution.
For few-shot learning with more than one shot, the aforementioned procedure of the distribution cali- bration should be undertaken multiple times with each time using one feature vector from the support set. This avoids the bias provided by one specific sample and potentially achieves more diverse and accurate distribution estimation. Thus, for simplicity, we denote the calibrated distribution as a set of statistics. For a class E Cn. we denote the set of statisties as S, = {(p) Σj,,.. ...; (Mx: Σx)); where μi Σ are the calibrated mean and covariance, respectively, computed based on the i-th fea- ture in the support of class y. Here, the size of the set the value of K for N-way-K-shot task.
3.2.3 HOW LEVERAGE THE CALIBRATED DISTRIBUTION?
With set of calibrated statistics Sy for class y in a target task, we generate a set of feature vectors with label y by sampling from the calibrated Gaussian distributions:
y (xx v) x Mn, Σ). V(u, Σ) E S²).
Here, the total number of generated features per class is set as a hyperparameter and they are equally distributed for every calibrated distribution in Sy- The generated features along with the original support set features for a few-shot task is then served as the training data for a task-specific classifier. We train the classifier for task by minimizing the cross-entropy loss over both the features of its support set S and the generated features Dy:
Σ
- log Pr(y(=;0),
(8)
(UB ..ETTT
where yT is the set of classes for the task T. Š denotes the support set with features transformed by Turkey's Ladder of Powers transformation and the classifier model is parameterized by 0.
EXPERIMENTS
In this section, we answer the following questions:
How does our distribution calibration strategy perform compared to the state-of-the- the-th-art methods?
5

---

Published as a conference paper at ICLR 2021
Table 3: Swaylshot and Sway5shot classification accuracy (%) rieredImageNet Ren et al. numbers bold have intersecting confidence intervals with the most accurate method.
2018 The
tieredImageNet
Methods
5way Ishot
5way5shot
Matching Net Vinnaals al.
(0166)
68.50 0.92
80.60 = 0.71
Prototypical Net (Snell et al. 2017
65.63 + 0.92
83.40 = 0.65
LEO Rusu et al. 2019))
66.33 0.05
82.06 = 0.08
E3BM (Liu et al.
(020c
71.20 + 0.40
85.30
= 0.30
DeepEMD (Zhagg et al. 2020)
71.16 0.87
86.03 0.58
Maximum Likelihood with DC (Ours)
75.92 0.60
87.84 0.65
SVM with DC (Ours)
77.93 0.12
89.72
= 0.37
Logistic Regression with DC (Ours)
78.19 0.25
89.90 = 0.41
What does calibrated distribution look like? Is it an accurate approximation for this class?
How does Tukey's Ladder of Power transformation interact with the feature generations: How important is each in relation to performance?
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS
We evaluate our distribution calibration strategy on minilmageNeet (Ravi & Larochelle (2017). tieredImageNeet (Ren al. (2018) and CUB (Welinder et al. (2010). minilmageNeet and lieeedlmageNet have brand range of classes including various animals and objects while CUB more fine-graid fine-grained dataset that includes various species of birds. Datasets with different lev-
els of granularity may have different distributions for their feature space. We want to show the effectiveness and generality of our strategy on all three datasets.
NeteN is derived from ILSVRC-12 dataset (Russakovsky et al. 2014). It contains 100 diverse classes with 600 samples per class. The image size is 84 x 84 x 3. We follow the splits used in previous works (Ravi & Larochelle, 2017). which split the dataset into 64 base classes, 16 validation classes, and 20 novel classes.
tieredImageeNer is larger subset of ILSVRC-12 dataset (Russakovsky al. 2014), which contains 608 classes sampled from hierarchical category structure. Each class belongs to one of 34 higher- level categories sampled from the high-level nodes in the ImageNet. The average number of images in each class is 1281. We use 351, 97, and 160 classes for training, validation, and test, respectively.
CUB is fine- grained few-shot classification benchmark. It contains 200 different classes of birds with a total of 11,788 images of size 84 x 84 3. Following previous works (Chen et al.) (019a), split the dataset into 100 base classes, 50 validation classes, and 50 novel classes.
4.1.2 EVALUATION METRIC
We use the top-1 accuracy the evaluation metric to measure the performance of our method. We report the accuracy on 5way Ishot and 5way5shot settings for minilmageNet, tieredImageNet and CUB. The reported results are the averaged classification accuracy over 10,000 tasks.
4.1.3 IMPLEMENTATION DETAILS
For feature extractor, we use the WideResNet (Zagoruyko & Komodakis 2016) trained following previous work (Mangla al. (2020). For each dataset, we train the feature extractor with base classes and test the performance using novel classes. Note that the feature representation is ex- tracted from the penultimate layer (with a ReLU activation function) from the feature extractor, thus the values are all non-negative so that the inputs to Tukey's Ladder of Powers transformation in Equation and transfer them to calibrate novel class distribution for each dataset. We use the LR and SVM implementation of scikit-learn (Pedregosa et al. (201T) with the default settings. We use the same hyperparameter value for all datasets except for a. Specifically, the number of generated features
3 are valid. At the distribution calibration stage, we compute the base class statistics

---

Published as a conference paper at ICLR 2021
Support Set
Sampling 100 featuree per class
Sampling features per class
Query Set
[s]
[b]
(d)
Figure 2: t-SNE visualization of our distribution estimation. Different colors represent different classes. **' represents support features, 'x' in figure (d) represents query Teatures, 'A in figure (b)(c) represents generated features.
wi minilmageNNe
minilmageNee
0.70
Test Accuracy (5way--1e) 1sho0
3.6
0.66
Test Accuracy
0.63
0.00
WiTakey transfomation
0.56
20
50
100
Values of power X in Tukey transformation
Number of generated features per class
Figure 3: Left: Accuracy when increasing the power in Tukey's transformation when training with (red) or without (blue) the generated features. Right: Accuracy when increasing the number of generated features with the features transformed by Tukey's transformation (red) and without Tukey's 's transformation (blue).
is 750; k 2 and À 0.5. is 0.21, 0.21 and 0.3 for minilmageNet, tieredImageNet and CUB, respectively.
4.2 COMPARISION TO STATE-OF-THB-ART
Table 2 and Table 3 presents the 5way shot and 5way5shot classification results of our method on minilmageNet, tieredImageNet and CUB. We compare our method with the three groups of the few- shot learning method, optimization- briimizationnbasedd metric-based, and generation-based. Our method can be built on top of any classifier, and we use two popular and simple classifiers, namely SVM and LR to prove the effectiveness of our method. Simple linear classifiers equipped with our method perform better than the oate-off-theart few-shot classification method and achieve the best perfor- mance on I-shot and 5-shot settings of minilmageNNtt, lieredlmageNet and CUB. The performance method by 10% for
of our distribution calibration surpasses the state-of-the-art generation- the 5way 1shot setting, which proves that our method can handle extremely low-shot classification tasks better. Compared to other generation- -aasd methods, which require the design of generative model with extra training costs the learnable parameters, simple machine learning classifier with DC is much more simple, effective and flexible and can be equipped with any feature extractors and classifier model structures. Specifically, we show three variants, i.e, Maximum likelihood with DC, SVM with DC, Logistic Regression with DC in Table and Table A simple maximum likelihood classifier based on the calibrated distribution outperform previous baselines and training a SVM classifier or Logistic Regression classifier using the samples from the calibrated distribution can further improve the performance.
4.3 VISUALIZATION OF GENERATED SAMPLES
We show what the calibrated distribution looks like by visualizing the generated features sampled from the distribution. In Figure we show the t-SNE representation (van der Maaten & Hinton

---

Published as a conference paper at ICLR 2021
Table 4: Ablation study minilmageNee Sway Ishot and 5way5shot showing accuracy (% with 95% confi- dence intervals.
minilmageNet
Tukey transformation
Training with generated features
5way
5way5shot
56.37 0.68
79.03 + 0.51
X
64.30 + 0.53
81.33 + 0.35
63.70 0.38
82.26 + 0.73
68.57 + 0.55
82.88 + 0.42
Table 5: Swaylshot classification accuracy (%) miniImageNet with different backbones.
Backbones
without DC
with DC
conv4 (Chen et al.
2019a)
12.11=0.7
54.62
0.64 (1 12.51)
conv6 Chen et al.
2019a)
46.07 0.26
57.14 0.45
11.07)
resnet
(Chen et al. 20T9a
52.32 :: 0.82
61.50
0.47 (1 9.180)
WRN28
Mangla al. 2020)
54.53 0.56
64.38
0.63 9.850)
WRN28 Rotation Loss Maaglla et al. 2020
56.37 = 0.68
68.57 = 0.55
(1 12.20)
2008 of the original support set (a), the generated features (b,c) as well as the query set (d). Based on the calibrated distribution, the sampled features form Gaussian distribution and more samples (c) can have a more comprehensive representation of the distribution. Due to the limited number of examples in the support set, only in this case, the samples from the query set usually greater area and are a mismatch with the support set. This mismatch can be fixed to some extent by the generated features, i.e., the generated features (c) can overlap areas of the query set. Thus, training with these generated features can alleviate the mismatch between the distribution estimated only from the few-shot samples and the ground truth distribution.
4.4 APPLICABILITY OF DISTRIBUTION CALIBRATION
Applying distribution calibration different backbones
Our distribution calibration strategy agnostic to backbones feature extractors. Table 5 shows the consistent performance boost when applying distribution calibration different feature extrac- tors, i.e four convolutional layers (conv4), six convolutional layers (conv6), resnet18, WRN28 and wRN28 trained with rotation loss. Distribution calibration achieves around 10% accuracy improve- ment compared to the backbones trained with different baselines.
Applying distribution calibration other baselines
variety of works can benefit from training with the features generated by our distribution calibra- tion strategy. We apply distribution calibration strategy algorithms, Baseline (Chen et al. 2019a) and Baseline++ (Chen et al. (0119)). Table 6 shows that distribution calibration brings over 10% of accuracy improvement on both.
simple few-shot classification
4.5
EFFECTS OF FEATURE TRANSFORMATION AND TRAINING WITH GENERATED FEATURES
Ablation Study
Table Table[shooe shows the performance when our model is trained without Tukey's Ladder of Powers trans- formation for the features as in Equation and when it is trained without the generated features as clear that there is a severe decline in performance of over 10% if both are not used in the 5way Ishot setting. The ablation of either one results in a performance drop of around 5% in the 5way Ishot setting.
in Equation 7
Choices of Power for Tukey's Ladder of Powers Transformation
The left side of Figure 3 shows the 5way Ishot accuracy when choosing different powers for the Tukey's 'ss transformation Equation 3 when training the classifier with the generated features (red) and without (blue). Note that when the power equals 1, the transformation keeps the original feature representations. There consistent general tendency for training with and without the
8

---

Published as a conference paper at ICLR 2021
Table 6: Swaylshot classification accuracy (%) minilmageNet with different baselines using distribution calibration.
Method
without DC
with DC
Baseline (Chen et al. 2019a)
42.11 = 0.71
54.62 + 0.64 (1 12.51) 48.24 = 0.75 61.24 + 0.37 (1 13.00)
Baseline++ (Chen al. 2019a)
generated features and in both cases, we found X 0.5 is the optimum choice. With the Tukey's transformation, the distribution of query set features in target tasks become more aligned to the calibrated Gaussian distribution, thus benefits the classifier which trained on features sampled from the calibrated distribution.
Number of generated features
The right side of Figure 3 analyzes whether more generated features results in consistent improve- ment in both cases, namely when the features of support and query set are transformed by Tukey's s transformation (red) and when they are not (blue). We found that when the number of generated features below 500, both cases can benefit from more generated features. However, when more features sampled, the performance of the classifier tested untransformed features begins to decline. By training with the generated samples, the simple logistic regression classifier has a 12% relative performance improvement in 1-shot classification setting.
4.6 OTHER PARAMEESR
We select the hyperparameters based on the performance of the validation set. The k base class statistics to calibrate the novel class distribution in Equation [5] is set 2. Figure 4 shows the effect of different values of k. The in Equation 6 is a constant added on each element of the estimated covariance matrix, which can determine the degree of dispersion of features sampled from the calibrated distributions. An appropriate value of a can ensure good decision boundary for the classifier. Different datasets have different statistics and an appropriate value of may vary for different datasets. Figure 5 explores the effect of tieredImageNet and CUB. We observe that each dataset, the performance the validation set and the novel (testing) set generally has the same tendency, which indicates that the variance is dataset-dependent and is not overfitting to a specific set.
all three datasets, i.e. minilmageNet,
0.70
mindingghhet Teding 2et minimaggheet Valibaiion 507
Accuracy (5way-1 shot)
0.69
Accuracy (5way- 1shot)
minmmggsut texing miimaggheet Valddiion
Testing
mereehaeee magedin
0.65 2
Number retrieved base class statistics K 10
6
0,65 01 615 02 021 622 023 554 025 c3 635 54
Values of a added on covariance matrix
Figure 4: The effect of different values
Figure The effect of different values of a.
CONCLUSION AND FUTURE WORKS
We propose simple but effective distribution calibration strategy for few-shot classification. With- out complex generative models, training loss and extra parameters to learn, a simple logistic regres- sion trained with features generated by our strategy outperforms the current state-of-the-art methods by ~ 5% on minimmgeNNet. The calibrated distribution is visualized and demonstrates an accurate estimation of the feature distribution. Future works will explore the applicability of distribution cal- ibration on more problem settings, such as multi-domain few-shot classification, and more methods, such metric-based meta-learning algorithms.

---

Published as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos J. Storkey. Assume, augment and learn: Unsupervised few-shot meta- learning via random labels and data augmentation. CoRR, 2019.
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. ICLR, 2019a.
Zitian Chen, Yanwei Fu, Yinda Zhang, Yu-Gang Jiang, Xiangyang Xue, and Leonid Sigal. Multi- level semantic feature augmentation for one-shot learning. TIP, 28(9):4594-4005, 2019b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Hang Gao, Zheng Shou, Alireza Zareian, Hanwang Zhang, and Shih-Fu Chang. Low-shot learning via covariance-preservinng adversarial augmentation networks. In NeurlPS, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. CoRR, 2017.
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In ECCV, 2020a.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In CVPR, June 2020b.
Lu Liu, Tianyi Zhou, Guodong Long. Jing Jiang. Lina Yao, and Chengqi Zhang. Prototype prop- agation networks (PPN) for weakly-supervised few-shot learning on category graph. In IJCAI, 2019a.
Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Learning to propagate for graph meta-learning In NeurIPS, 2019b.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble of epoch-wise empirical bayes for few- shot learning. ECCV, 2020c.
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In WACV, 2020.
Seong-Jin Park, Seungju Han, Ji-won Back, Insoo Kim, Juhwan Song, Hae Beom Lee, Jae-Joon Han, and Sung Ju Hwang. Meta variance transfer: Learning to augment from the others. In ICML, 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Tiexin Qin, Wenbin Li, Yinghuan Shi, and Yang Gao. Diversity helps: Unsupervised few-shot learning via distribution shift-based data augmentation. 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICIRR 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2021
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Representations by Back- proppaapggg Errors. Nature, 323:533-5344 1986.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. CoRR, abs//449.0055. 2014.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Ruslan Salakhutdinov, Joshua Tenenbaum, and Antonio Torralba. One-shot learning with a hicrar- chical nonparametric bayesian model. In ICML workshop, 2012.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Ciryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. In NeurlPS, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, 2017.
John W Tukey. Exploratory data analysis. Aedison-Wesles Series in Behavioral Science. Addison- Wesley, Reading, MA, 1977. URL htttps::/cctt cern. ctt.ecooooooooo0005555555
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NearIPS, 2016.
Yu-Xion Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. In NeurlPS, 2019.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report ENS-TR-2010-00 California Institute of Technology, 2010.
Yongqin Xian, Tobias Lorenz, Bernt Schicle, and Zeynep Akata. Feature generating networks for zero-shot learning. In CVPR, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020.
Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In ICCV, 2019.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to shw-shot learning. In NeurlPS, 2018.

---

Published as a conference paper at ICLR 2021
Base class distributio
Novel class distribution
Novel class distribution after Tukey's Transformation
Figure 6: We show the feature distribution base classes, and the feature distribution of novel classes before/after Tukey's Transformation.
Training data
minilmageNet 5way Ishot
Support set only
56.37 + 0.68
Support set feature from the nearest class Support set 5 features from the nearest class Support set + 10 features from the nearest class Support set 100 features from the nearest class
62.39 + 0.49
59.73 = 0.42
58.93 = 0.49
57.33 = 0.48
Support set 100 features sampled from calibrated distribution
68.53 = 0.32
Table 7: The comparison with nearest class feature augmentation.
A AUGMENTATION WITH NEAREST CLASS FEATURES
Instead of sampling from the calibrated distribution, est class to augment support set. Table 7 shows the comparison of training using samples from the calibrated distribution, the different number of retrieved features from the nearest class, and only using the support set. We found the retrieved features can improve the performance compared to only using the support set but can damage the performance when increasing the number of retrieved fea- tures, where the retrieved samples probably serve as noisy data for tasks targeting different classes.
simply retrieve examples from the near-
B DISTRIBUTION CALIBRATION WITHOUT NOVEL FEATURE
We calibrate the novel class mean by averaging the novel class mean and the retrieved base class means in Equation [ Table Aablegaa shows the distribution calibration without averaging novel feature, in which the calibrated mean is calculated as 4ı Esc M4
C THE EFFECTS OF TUKEY'S TRANSFORMATION
Figure 6 shows the distribution of base classes and 5 novel classes before/after Tukey's trans- formation. is observed that the base class distribution satisfies Gaussian assumption well (left) while the novel class distribution is more skew (middle). The novel class distribution after Tukey's transformation (right) is more aligned with the Gaussian-like base class distribution.
minilmageNet 5way shot
Distribution Calibration w/o novel feature ø Distribution Calibration w/ novel feature
59.38 + 0.73
68.57 + 0.55
Table The comparison between distribution calibration with without novel feature ă.
12

---

Published as a conference paper at ICLR 2021
Novel class
Top-1 base class similarity
Top-2 base class similarity
DC improvement
malamute
93%
85%
21.30%
golden retriever
85%
74%
18.37%
71 %
67%
9.77%
Table 9: Performance improvement with respect to the similarity level between query novel class and the most similar base classes.
D THE SIMILARITY LEVEL ANALYSIS
We found that the higher similarities between the retrieved base class distribution and the novel class ground- truth distribution, the higher the performance improvement our method will bring as shown Table 9 The results in the table are under 5-way-1 1-shot setting.
13


Reviews:
[{"review_id": "zuXW6WDf8ia", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a method to calibrate the underlying distribution of a few samples in the few-shot classification scenario. The idea is to estimate a feature distribution of a few samples of a novel class from base class distributions. The authors assume that every dimension in the feature vector follows a Gaussian distribution. Based on the observation that the mean and variance of the distribution with respect to each class are correlated to the semantic similarity of each class, base class distribution can be transferred to the novel class distribution. After distribution calibration, features can be generated from the calibrated distribution and the generated features are used to train classifiers. SVM and logistic regression classifier are used to verify the approach on the mini-imagenet and CUB datasets.\n\nPros\n-   The idea can be applied to any types of feature extractors or classifiers when the Gaussian distribution assumption holds.\n-   The proposed approach shows competitive performance on two benchmarks, mini-imagenet and CUB.\n-   Extensive ablation studies verify hyper-parameter settings and their sensitivities.\n\nCons\n- Many hyper-parameters are involved in the approach. ( Turkey’s transformation parameter lambda, top-k base distributions, dispersion parameter alpha, number of generated features)\n-   Some hyper-parameter setting should be different depending on the dataset. (dispersion parameter alpha)\n-   The paper uses the previous work (Mangla et al. 2020) as a baseline and applies their approach on top of it. While theoretically the approach can be applied to any types of feature extractors, only one baseline improvement is shown in the paper.\n\n\nRating\n\nI give marginally below the threshold. The paper shows good performance and also claims the approach can be applied on top of any classifiers or feature extractor. The general applicability and effectiveness are the strength of the paper. However, the proposed approach is only verified on one baseline approach (Mangla et al. 2020). More empirical data would strengthen the claim of the paper.\n\nQuestions\n\nThe approach outputs calibrated distribution, i.e., a mixture of Gaussian distributions. These distributions can be directly used to classify query samples by calculating the likelihood of the sample on each distribution. How does likelihood classification perform compared to the retrained classifiers (SVM or RL)?\nThe optimal values for top-k parameter and alpha parameter are different depending on the dataset. Are optimal values for lambda and the number of generated features same or different depending on the dataset? Did the authors use the same lambda and number of generated features for both mini-imagenet and CUB experiments?\nThe authors only applied the proposed method on one baseline approach. Is the approach effective for more variety of backbone networks and losses? \n\nFeedback\n\n- Table1 shows that there is a correlation between feature distributions and semantic similarities. It would be interesting to see how the distribution calibration performs on each class depending on the similarity level of top-k base distributions.\n- The empirical evidence is not sufficient to claim the approach applies to general network architectures and few-shot learning approaches. Experiments with more baseline approach to show the effectiveness of the idea is recommended.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DD7-TM9kZb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simple and intuitive data augmentation method for few shot image classification. The proposed method assumes the feature distribution of a single class to be gaussian. Based on this assumption, the proposed method samples features from a gaussian distribution that is constructed using the statistics of the similar base classes. Combined with logistic regression, the proposed method achieves strong performance on two standard few-shot image classification benchmarks. This submission also carries out an ablation study on several design choices, which I really appreciate.\n\nThe submission is well-written and clear. The proposed method is novel and can inspire future augmentation-based methods in few-shot image classification.\n\nI have a few more requests/ablations that I am curious to see. \n\n- Looking at Figure 4, the 1-shot accuracy with only 1 retrieved class is already very strong. Instead of sampling from the \"calibrated\" distribution, can we simply retrieve examples from the nearest class? Namely, find the nearest class, randomly sample some examples, and use their features to augment the novel classes. This ablation should make clear what additional value the sampling procedure adds.\n- In equation (6), how important is it to include the novel class feature into the mean? Can we simply do \\mu_prime = \\sum_{i \\in S_N} \\mu_i /  k?\n- This method makes an important assumption that the feature distribution is gaussian. How well does this assumption hold in practice? Can the authors provide some analysis of the feature distribution? Ideally both before and after Tukey’s Ladder of Powers transformation. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6oQZy4VR38", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper identifies the problem of biased distributions in few-shot learning and proposes to fix it. In few-shot learning, only a few samples per class are available; this makes estimating the class distribution difficult. The paper proposes a distribution calibration algorithm that makes use of the meta-train class distributions to calibrate the few-shot class distributions. Once calibrated, more samples are drawn from this distribution to learn a classifier that generalizes better. This approach does not require additional learnable parameters and can be (potentially) built on-top of any pre-trained feature extractor. Empirical results show that this approach achieves state-of-the-art results on Mini-ImageNet and CUB.\n\nPros:\n1. This paper identifies and tries to tackle an important problem in few-shot learning - estimation of the class distribution. Due to the limited number of samples, this problem is difficult, but important for few-shot learning. The proposed algorithm is simple and effective in tacking this problem.\n2. As opposed to other related works, the proposed algorithm does not have any learnable parameters. It makes use of the features obtained for the meta-train and few-shot samples.\n\nCons:\n1. One of the claims of the paper is that the proposed algorithm is pre-trained feature extractor agnostic. However, there are no experiments to validate this claim. Consider adding feature extractors trained in different ways.\n2. Some of the popular few-shot datasets were not included in the experimental section, namely Tiered-ImageNet [1] and Meta-Dataset [2]. The former is a larger dataset than the ones used, and the latter provides cross-domain results for the approach. The proposed algorithm assumes that statistics from the meta-train classes would transfer to the few-shot classes, which would be tested more thoroughly in the cross-domain setting.\n\nClarifications:\n1. In the feature space, it is intuitive that the means of similar classes would be correlated. Is there a justification for why this is true for the variances?\n2. Tukey's Ladder of Powers transformation is used only on the few-shot samples and not the meta-train samples. Is there a reason for that? If the pre-trained feature extractor is trained using a certain metric (cosine distances for example), I would imagine transforming all the features would be beneficial.\n3. The backbone used in the experiments is trained using a supervised and self-supervised loss. What are the results without the self-supervised loss?\n4. How many samples are drawn from the calibrated distribution for the numbers in Table 2?\n\nNotes:\n1. The performance of few-shot learning algorithms has been traditionally evaluated using the averaged accuracy over multiple tasks, but that is not the only way to do it. Look at [3] for details.\n\n[1] Mengye Ren et al. Meta-Learning for Semi-Supervised Few-Shot Classification.\n[2] Eleni Triantafillou et al. Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples.\n[3] Guneet S. Dhillon et al. A Baseline for Few-Shot Image Classification.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel and powerful data augmentation strategy for few-shot learning, producing convincing improvements over current approaches. The request by the reviewers to include additional ablations, more backbones, and an additional dataset have been satisfactorily  resolved, with the results remaining strong. The reviewers are all unanimous in their recommendation that the paper be accepted for publication.

Decision: Accept (Oral)