Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport
Authors: Dan dan Guo
Long Tian
He Zhao
Mingyuan Zhou
Hongyuan Zha
Venue: NeurIPS 2022 Accept (2022)

Abstract: Few-shot classification aims to learn a classifier to recognize unseen classes during training, where the learned model can easily become over-fitted based on the biased distribution formed by only a few training examples. A recent solution to this problem is calibrating the distribution of these few sample classes by transferring statistics from the base classes with sufficient examples, where how to decide the transfer weights from base classes to novel classes is the key. However, principled approaches for learning the transfer weights have not been carefully studied. To this end, we propose a novel distribution calibration method by learning the adaptive weight matrix between novel samples and base classes, which is built upon a hierarchical Optimal Transport (H-OT) framework. By minimizing the high-level OT distance between novel samples and base classes, we can view the learned transport plan as the adaptive weight information for transferring the statistics of base classes. The learning of the cost function between a base class and novel class in the high-level OT leads to the introduction of the low-level OT, which considers the weights of all the data samples in the base class. Experimental results on standard benchmarks demonstrate that our proposed plug-and-play model outperforms competing approaches and owns desired cross-domain generalization ability, indicating the effectiveness of the learned adaptive weights.

Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transport
Dandan Guo 1.2, Long Tian', He Zhao 4, Mingyuan Zhou', Hongyuan Z-a
'School of Data Science, The Chinese University of Hong Kong, Shenzhen
2 Institute of Robotics and Intelligent Manufacturing
Xidian University *CSIRO's Data61 'The University of Texas at Austin
Shenzhen Institute of Artificial Intelligence and Robotics for Society guodandan&cuhk edu. en t tianl ongexin@@eeooo meeexiiinneee.. edu.cn
he. zha@ ieee. org mingyuan. zhou@mccombs utexas edu zhahy@cuhk, edu en
Abstract
Few-shot classification aims learn classifier recognize unseen classes during training, where the lcarned model can easily become over-fitted based on the biased distribution formed by only a few training examples. A recent solution to this problem is calibrating the distribution of these few sample classes by transferring statistics from the base classes with sufficient examples, where how decide the transfer weights from base classes to novel classes is the key. However, principled approaches for learning the transfer weights have not been carefully studied. To this end, we propose a novel distribution calibration method by learning the adaptive weight matrix between novel samples and base classes, which built upon a hierarchical Optimal Transport (H-OT) framework. By minimizing the high-level OT distance between novel samples and base classes, we can view the Iearned transport plan the adaptive weight information for transferring the statistics of base classes. The learning of the cost function between a base class and novel class in the high-level OT leads to the introduction of the low- level OT, which considers the weights of all the data samples in the base class. Experiments standard benchmarks demonstrate that proposed plug-and- play model outperforms competing approaches and owns desired generalization ability, proving the effectiveness of the learned adaptive weights.
Introduction
Deep learning models have become the regular ingredients for numerous computer vision tasks
such as image classification 2 and achieve state-of-the-art performance. However, the strong performance of deep neural networks typically relies on abundant labeled instances for training [3]. Considering the high cost of collecting and annotating a large amount of data, a major research effort is being dedicated trending research subject in the low data regime, few-shot classification aims to learn model on the data from the base classes, so that the model can generalize well on the tasks sampled from the novel classes. Several lines of works have been proposed such as those based on meta- meta-learning paradigms 3 II and those directly predicting the weights of the classifiers for novel classes
fields such as transfer learning [4] and domain adaptation (5). As
Recently, methods based on distribution calibration have gained increasing attention. As representative example, Yang al. [4] calibrate the feature distribution of the few-sample classes by transferring the statistics from the base classes and then utilize the sampled data to train classifier
12 13
II you interested in work, please HandanGuo1993/AAeee Distribution- Cestrbution-Calibrationn for Sow-Shot-Leamingg with Hierarchical Optimal-TTrasport
code, which is available at httpps//github.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

for novel classes. A unique advantage of distribution calibration methods over others is that they build on top of off-the-shede pretrained feature extractors and do not finetune/re-te train the feature extractor.
The key of distribution calibration methods is to select the corresponding base classes and transfer their statistics for the labeled samples in novel task. Existing approaches in this line usually do so with heuristic or less adaptive solutions. Specifically, Yang et al. [1] use the average features of the samples as the representation of a base class and select the top-k (e.g., k 2) closest base classes based on the Euclidean distance between the features of a novel sample and a base class. Despite the effectiveness of Yang et al. 14 it questionable whether the Euclidean distance is the proper metric to measure the closeness between a base class and a novel sample since viewing novel sample and a base class as points in the same space may not be the best solution. Moreover, it is less sound to characterize a base class only by the unweighted average over all its samples, when measuring its closeness with the novel sample. Representing a base class in this way would completely ignore the fact that each sample of a base class may contribute to the classification boundary differently. Finally, it may also be less effective to treat each of the top-k base classes equally as their contributions can also be different, not to mention the omission of the other base classes.
To this end, this work develops more adaptive distribution calibration method leveraging optimal transport (OT), which is a powerful tool for measuring the cost in transporting the mass in one distribution to match another given a specific point-to-point cost function. First, we formulate a distribution P over the base classes and a distribution Q over the labeled samples from the novel classes. With such formulation. how to transfer the statistics from the base classes to the novel samples can be viewed as the OT problem between two distributions, denoted as the high-level OT. By solving the high-level OT, the learned transport plan can be used as the similarity or closeness between novel samples and base classes. Since the high-level OT requires specifying the cost function between one base class and one novel sample, we further introduce a low-level OT problem to learn this cost automatically, where we formulate base class way, the similarity between a novel sample and a base class is no longer representing a base class by the unweighted average over all its samples and then using the Euclidean distance. In our method, the weights of the samples are considered in a principled way. In summary, the statistics of base classes can be better transferred to the novel samples for providing more effective way to measure the similarity between them. Notably, even in the challenging cross-domain few-shot learning. our H-OT can still effectively transfer the statistics from the source domain to the target domain.
distribution over its samples. In this
We can refer to this adaptive distribution calibration method as a novel hierarchical OT method (H-OT) for few-shot learning, which is applicable to a range of semi-supervised and supervised tasks, such as few-shot classification [9] and domain adaptation [5]. Our contributions are summarized as follows: (I) We develop new distribution calibration method for few-shot learning, which can be built on top of an arbitrary pre-trained feature extractor for being implemented over the feature-level. without further costly fine-tuning (2) We formulate the task of transferring statistics from base classes to novel classes in distribution calibration as the H-OT problem and tackle the task with a principled solution. (3) We apply our method to few-shot classification and also explore the cross- noss-domain generalization ability. Experiments on standardized benchmarks demonstrate that introducing the H-OT into distribution calibration methods learn adaptive weight matrix, paving a new way to transfer the statistics of base classes to novel samples.
2 Background
2.1 Optimal Transport Theory
Optimal Transport (OT) is a powerful tool for the comparison of probability distributions, which has been widely used in various machine learning problems, such as generative models [15]. text
analysis [66] 17), adversarial robustness [18]. and imbalanced classification [19] Here we limit
our discussion to OT for discrete distributions and refer the reader to Peyré and Cuturi [20] for more details. Denote p- = Σ a,5r, and q = Ey= bydg, as two n and m dimensional discrete probability distributions, respectively. In this case, a E A and b E AT", where A denotes the probability simplex of IR m. The OT distance between p and q is defined as
OT(p, q) = min (T,C), TEll(a))
(I)

---

where (.-)) denotes the Frobenius dot- dot-product; C E R220 is the transport cost function with element
C₁ C(z,, Uy): : T E Rax" denotes the doubly stochastic transport probability matrix such that [(p, = (T) Σ bb: Σ3 T. = =a,). meaning that T has to be one of the joint distribution of p and q. As directly optimising Equation (1) can be time-consuming for large-scale problems, the entropic regularization, = Eij Ti In Tni, u.. is introduced in Cuturi [21], resulting in the widely-used Sinkhorn algorithm for discrete OT problems with reduced complexity.
2.2 Few-Shot Classification
Following a typical few-shot learning problem, we divide the whole dataset with labeled examples into a base dataset Dase with B base classes and a novel dataset D novel with No novel classes, each with a disjoint set of classes. To build a commonly-used
V-way-K-shi task 18 [4]. we randomly sample N classes from Naul novel classes, and in each class, we only pick K (e.g.. or 5) samples
for the support set S = ((1 ] NNK to train or fine-tune the model and sample q instances for the N
query set Q= (z 4:) NiNNN K+1 to evaluate the model. By averaging the accuracy on the query set of multiple tasks from the novel dataset, we can evaluate the performance of model.
2.3 Distribution Calibration for Few-Shot Classification
Distribution calibration [4] uses the statistics of base classes to estimate the statistics of novel samples in the support set and generate more samples. Specifically, for the 6th base class, its samples are assumed to be generated from a Gaussian distribution, whose mean and covariance matrix are:
fAb
J6 -
E Σ Jo _ E
",, (c,- mb) (2, mb)"
(2)
where b BJ. 2 E TRV is the V-dimensional feature of sample j extracted from the pre-trained feature encoder, Jb the number of samples in class b, and {ub, ] are the statistics of base class b.
The samples of novel class are also assumed to be generated from a Gaussian distribution with mean μí and covariance E' As the novel class only has one or few labeled samples, it is hard to accurately estimate μı' and E' Thus, the key idea is to transfer the statistics of the base classes to calibrate the novel class's distribution. Once the distribution of the novel class is calibrated, we can generate more samples from it, which are useful for training good classifier. As result, how to effectively transfer the statistics from the base classes is critical to the success of distribution calibration basee methods for few-shot learning. Accordingly, Free-Lunch [14] designs heuristic approach that calibrates the Gaussian parameters of novel distribution with one data sample x::
P (B. + i
Σ' Eietoph(3.) Σ; +
k +
where: 1) ē is the transformed data by Tukey's Ladder of Powers transformation (TLPT) 22 i.e., a if X of and ã log(a) (x) À = 0, for reducing the skewness of distributions and make distributions more Gaussian- like. 2) S. = { I|4- ã [1,
distance set defined by the Euclidean distance between the transformed feature ã of novel sample in support set and the mean μ4, of the base class b. 3) tSp)(S,) the operation that selects the top-k closest base classes from the set S., 4) determines the degree of dispersion of features sampled from the calibrated distribution.
Although effective, Free-Lunch represents a base class by the unweighted average over all its samples when computing its closeness with a novel sample, which ignores the fact that each sample of a base class may contribute the classification boundary differentl.. In addition. the Euclidean distance in the feature space may not well capture the relations between base class and a novel sample. Moreover, each selected top-k base class has an equal weight (i.e., 1/k), which may not reflect the different contributions of the base classes and omit useful information in unselected base classes.
3
Our Proposed Model
3.1
Overall Method
In this work, we propose a novel adaptive distribution calibration framework, a holistic method for few-shot classification. Compared to the novel classes, which only have a limited number of labeled
3

---

samples, the base classes typically have sufficient amount of data, allowing their statistics to be estimated more accurately. Due to the correlation between novel and base classes, it reasonable to use the statistics of base classes to revise the distribution of the novel sample. Therefore, the key how to transfer the statistics from the base classes to a novel class to achieve the best calibration results, which is the focus of this paper. Here, we develop the H-OT to learn the Transport Plan matrix between base classes and novel samples, where each element of the transport plan measures the importance of each base class for each novel sample and more relevant classes usually have a larger transport probability. Computing the high-level OT requires the specification of the cost function between one base class and one novel class, which leads to the introduction of low-level OT problem. By viewing the learned transport plan as the adaptive weight matrix, we provide an elegant and principled way to transfer the statistics from the base classes to novel classes.
3.2 Hierarchical OT for Few-Shot Learning
Moving beyond the Free-Lunch method [14], which uses the Euclidean distance between novel sample and base class to decide their similarity and endow the chosen base classes with the equal importance, we aim to capture the correlations between the base class and novel samples at multiple levels and transfer the related statistics from base classes to novel samples. We learn the similarity by minimizing a high-level OT distance between base and novel classes and build the cost function used high-level OT by further introducing a low-level OT distance. To formulate the task as the high-level OT problem, we model P' as following discrete uniform distribution over B base classes:
B
1 6n..
1
P=
b=1 B
where Rs represents the base class b in the V V-dimmensional feature space, which will be introduced later. Taking the --way-l-shot task as the example, where each novel class has one labeled sample a, we represent Q as a discrete uniform distribution over N novel classes from support set:
Q = Σ=>m n=1 N S0
Zn E R Vx1
(5)
where H, is the transformed feature from æ following Yang et al. [4] and detailed below Equation 3 The OT distance between P and Q is thus defined as OT Q) = minTeT (T, C). We adopt regularised OT distance with an entropic constraint [21] and express the optimisation problem as:
OT, (P, def.
B,N b,n Tin - > b,m -Tim In Tim:
B.N
(6)
where é >>> 0, C
IR BxN is the transport cost matrix, and Cin. indicates the cost between base "2
B class b and novel sample n. Importantly, the transport probability matrix T E Bx should satisfy II())Q) = ΣN Ton B E. Tum x } with element Ton = T(Rs, æ,... which denotes the transport probability between the bth base class and the rith novel sample and is an upper-bounded positive metric. Therefore, Thn provides natural way to weight the importance of each base class which can be used as the class similarity matrix when calibrating the novel distribution. Hence, the transport plan is the main thing that we want to learn from the data.
To compute the OT in Equation 6 we need define the cost function C, which is the main parameter defining the transport distance between probability distributions and thus plays the paramount role in learning the optimal transport plan. In terms of the transport cost matrix, a naive method is to specify the C with Euclidean distance or cosine similarity between the feature space of the novel sample and mean of the features from the base classes, such as Can = cocos (ăn. Ms). However, these manually chosen cost functions might have the limited ability to measure the transport cost between a base class and novel sample. Besides, representing the base class only with the average of all features in class b might ignore the contributions of different samples for this class. Hence the optimal transport plan for these cost functions might be inaccurate. To this end, we further introduce a low-level OT optimization problem to automatically learn the transport cost function C in (6). Specifically, further treat each base class as an empirical distribution RG over the features within this class:
Jo E/e 1
2/ - RV.
4

---

where P is the weight of data æ to base class b and captures the importance of this sample and will be described in short order. Specifically, we train classifier parameterized by ø with the samples in the base classes, which predicts which base class sample is in. The predicted probability of sample j belonging to the base class b is denoted by sy and then ..'..... .',, is obtained by normalizing the
vector ........ sty_] with the Softmax function. We further define the low-level OT distance between each distribution Rd and distributions Q with an entropic constraint as
OT. Q) def. End
N.Js
DJ Mjn
>> N,J
MM In 1
(8)
n.
my E J&NN should satisfy II Q)
where the transport probability matrix M {E/ Mjne' - & = The element DJ, of cost function D E B is the
distance between the n-th novel sample and the j-th sample from base class b, where we naturally the distance between their features, i.., क, and z/.. Empirically, we find that the cosine similarity cos( x²)) works well practice, although other choices are possible.
Minimizing the low-level OT loss in Equation (8) can learn an optimal transport probability matrix M for E B, where M tells us the transport weight between the nith novel class and jth sample in b base class. Back to the cost function Con for the high-level OT, instead of using the manually chose cost functions, we further adopt the learned total transport cost between novel sample H,, and all samples from base class b in the low-level OT to represent the cost function Con in high-level OT:
Jo -m M"
Chn =
where Cin is fed into 66 for learning the transport plan between base classes and novel samples. Defined in this way Con is adaptive cost between base class b and novel sample n in support set, taking full advantage of all samples in base class. With the proposed low-level OT distance, the distance between novel sample and base class is no longer representing a base class by the unweighted average over all its samples and then using the Euclidean distance. In our method, the weights of the samples are considered in a principled way.
3.3 Calibrating Distribution and Training Classifier
Once we obtain the transport plan matrix by minimizing the OT problem in (6. we can compute the statistics of novel samples by following Yang et al. (14). For the n-th transformed feature S, in the support set, we calibrate its mean and covariance as follows:
N E bEn Tin/4s + En
Σn N N Tim EB + a,
1/
(10)
B ++1
B
where μa and Eg are the statisties of base class by respectively; Ton provides an adaptive way to weight the importance of base class b for novel sample m: N is used to scale Thn since Eoen Ton 1/N; a is a hyper-parameter explained in Equation
For W-way-K-shot task with K > 1, the N in aforementioned Equations 6 and 8 should be replaced with N K, and the distribution calibration Equation (10) should be undertaken K times for each novel class. Notably, this way can potentially achieve more diverse and accurate distribution estimation for avoiding the bias provided by one specific sample. Thus, for class y E N in novel task, we denote the calibrated distribution with set of statistics as
Sy (1/11 Exa) ..... (1'K: Exx)
(11)
Based on set Sy for novel class y. we can sample from the calibrated Gaussian distributions to generate set of feature vectors with label y: Dy {xx,v) v) ~ NN, E), V(n, Σ) S*}. Given the transformed support set Š with TLPT and the generated features Dy, we only need train the classifier for a task TT (under, L-way-K-shol by minimizing the cross-entropy loss:
Σ
log Pr(y 2x;
(12)
(I.g) (H.G)
where YT is the set of classes for the task T and the classifier is parameterized by 0. We describe our proposed framework in Algorithm
5

---

Algorithm 1 Workflow about our adaptive distribution calibration on few-shot learning.
Require: Datasets Dusse D Dovel , pretrained feature extractor ʃ()), classifier Φ, classifier 0. hyper-
parameters
Extract the features of D base and D novel with f()); compute {us; ) by (2)
Train a classifier & with Dbase to compute p/: Represent base class b as Rs with for t in number of tasks do
Randomly choose N-way-K-shon task; Learn M with (8) and compute cost Cbn with 9
Learn the transport plan matrix T with (6);
Calibrate the statistics with (TO) and generate the samples from TT:
Train the N-way classifier 0 with (12) and test the query set classifier 0.
end for
4 Related Work
Recently, many efforts have been devoted to exploring the idea of learning to learn meta-learning to alleviate the few-shot challenge. One approach to this problem is the optimization- based meta- learning algorithms 6 71 which aim to learn single set of model parameters that can be adapted to individual tasks with a small number of gradient update steps. Some simple but effective algorithms based metric learning are also developed, with the goal of "learning to compare." " For example, MatchingNet [ and ProtoNet [9] learned to classify samples by comparing the distance to the representatives of each class. Another line of algorithms is compensate for the insufficient samples by learning to augment. Most methods augment the training set using Generative Adversarial Networks (GANs) 23 autoencoders
24 to generate samples 22 or features 28 29
Recently, Free-Lunch [1]] is proposed to calibrate the distribution of the novel samples using the statistics from base classes and augment the data from the calibrated distribution. Although distribution calibration belongs to the scope of data augmentation, it does not need to build any complex generative models or fine-tune the backbone. Therefore, distribution calibration can be considered as a new promising line, where both Free-Lunch and our model fall into. Different from Free-Lunch which adopts heuristic method choose base classes for novel samples, develop novel hierarchical OT to learn the cost function and similarity (transport plan) between base classes and novel samples, providing general and adaptive distribution calibration framework for few-shot learning. A recent metric-based work of connecting few-shot learning with OT is DeepEMD (30), which decomposes an image into set of local features and uses the optimal matching cost between two images to represent their similarity. Besides, the authors of (31] introduce a prototype- oriiented OT (POT) framework for set-structured data and apply it to metric-based few-shot classification. However, we directly minimize the OT between base classes and novel samples to learn the transport plan, which serves as the adaptive weight matrix in distribution calibration. Although both using OT for few-shot learning, DeepEMD, POT and ours are totally different methods. The hierarchical topic transport distance (HOTT) [32 is developed to measure the distance between documents, where documents are modeled as distributions over topics (solved by a high-level OT), and topics are further modeled as distributions over words (solved by a low-level OT). Different from HOTT which captures the semantic difference between documents by leveraging OT, topic modeling, and word embeddings, we develop the H-OT to learn the adaptive weight matrix for improving distribution calibration in few-shot learning, task distinct from document representation.
5 Experiments
5.1 Experimental Setup
Datasets We evaluate our proposed method on several standard few-shot classification datasets with different levels of gramularity, including minilmageNet [33], tieredImageNet [34], CUB [35], and CIFAR-FS (6). Here, minilmageNet, tieredImageNet, and CIFAR-FS have broad range classes including various animals and objects while CUB is more fine-grained dataset that includes various species of birds. The minilmage which is randomly chosen from IISVVR--12 dataset (37]. contains 100 diverse classes with 600 samples per class. The image size is 84 x 84 x 3. Following

---

Table I: Classification accuracy (%) on minilmageNet. tieredImagenet and CUB with 95% confidence intervals. o indicates the results reported by authors and is our implementation of Free-Lunch with their released code.
minilmageNet
tieredImagenet 5 5way
CUB 5way5shot
Methods
Backbome
Swaylshot
Sshot
Swaylshot
Sway shot
E3BM
ResNer25
64.3 n/a
81.0 + w/a
70.0+ n/a
85.0+
LEO
WRN28
61.76 0.08
77.59 4: 0.12
66.33 #: 0.05
81.44 :: 0.09
MAML by
ResNet18
49.611
65.72
69.96 1101
02.701.0.65
Baselinc++
ResNet18
5188 + 0.77 75.68
=: 0.63
67.02=0.990
03.58.0..11
Nosiiive-Cosine
ResNet12
63.85
81.57
0.56
72.66=0.83
89,40=+034
DecpEMD
30
ResNet12
65.91
0.82
82.41
+: 0.56
71.16 0.87
86.03 +-0.58
75.65
88.69-+-0.50
MatchNet g
by
ResNet12
63.08
0.80
75.99 1 0.60
68.50 0.92
80.60 1. 0.71
71.87
85.08.10.57
MatchNet g by []
ResNet34
68.32 :: 0.66
84.66:10.55
MatchiNot +POT by 31
ResNet34
68.51 = 0.64
85,50=+0.60
ProtoNet
by
ResNer12
60.37 0.83
78.02 +
65.65=092
83.40+00.65
66.09=0092
82.50-+0.58
ProtaNet
[ by 31
ResNet34
73.99 1. 0.64
87.33.10..48
ProtoNet by
ResNet34
75.15 0.63
86.34-t0.46
Free-Tainch'
[14
ResNet12
64.73++
0.44
81.15:00.2
71.40+0.31
85.56++0.40
74.13=+0880
87.91++0.45
Our HI-T
ResNet12
65.63.10.32
10.43
73.7110.26
87.46.10.35
76.28
89.87.10.36
LunnchL [
WRN26+RTloss
68.57
0.55
82.88
0.42
75.10 A 0.32"
88.42.1 0.40
79.56.10.87
90.67.10.35
Our H-OT
WRN2E+RTlous
69.04 0.29 84.36 + 0.41
75.91=0.35
89.33=2048
81.23:+0.35
91.45::0.38
the previous work [33] we split the dataset into 64 base classes, 16 validation classes, and 20 novel classes. The tageed is a larger subset of ILSVRC-12 dataset [37]. It contains 608 classes sampled from a hierarchical category structure, where each class belongs to one of 34 higher-level categories from the high-level nodes in the ImageNet. The average number of images in each class is 1281. We adopt 351, 97, and 160 classes for training, validation, and test, respectively. The CUB is fine-grained few-shot classification benchmark, which contains 200 different classes of birds with a total of 11,788 images, and resize them as 84 84 x 3. We follow previous works [ and split the dataset into 100 base classes, 50 validation classes, and 50 novel classes. The CIFAR-FS R-FS dataset is recently proposed few-shot image classification benchmark, consisting of all 100 classes from CIFAR-100 [8 The classes are randomly split into 64, 16, and 20 for meta-training, meta-validation, and meta-testing, respectively. Each class contains 600 images of size 32 32
Implementation Details Since Free-Lunch is the SOTA distribution calibration method, we mainly compare with Free-Lunch. For minilmageNet and CUB, we directly use pre-extracted features from Free-Lunch's code [1]. which are produced by WideResNet trained from scratch on the base classes by following Mangla et al. [39] to optimize classification loss and a self-supervised rotation loss jointly, denoted as WRN28+RTLoss. Free-lunch did not share the features on tieredImageNett and CIFAR-FS, thus have taken the pre-trained WRN28+RTlos backbone of Mangla et al. 39] to extract the features, where we use this same backbone to implement Free-lunch with its official code and we have explored and selected an appropriate for frncc. To confirm the validity of our proposed method on the commonly- trained from scratch by minimizing standard classification loss (without the rotation loss). To make sure the log transform in TLPT is valid to its inputs, we need to guarantee the non-negativity of features. Therefore, we extract the features of given samples from the penultimate layer (with ReLU activation function) of the feature extractor. To demonstrate the effectiveness of our method, we consider the simple Logistic Regression (LR) classifier for 0 and 6 in Algorithm [ where the LR implementation of scikit-learn [40] with the default settings is adopted. We use the same hyper- parameter value for all datasets except for X. Specifically, the number of generated features is 750, the in Sinkhorn algorithm is 0.01, the in (10) is 0.21; and À is 0.5. 1, 1 and 0.8 for minilmageNer, tiered CUB, and CIFAR-FS, respectively, selected by a grid search using the validation set. The maximum iteration number in Sinkhorn algorithm is set as 200. The reported results are the averaged classification accuracy over 10,000 tasks.
backbone, we also take the ResNet12 as the example, which
5.2 Evaluation with the Standard Setting
We now conduct experiments on the most common setting in few-shot classification, 1-shot and 5-shot classification, where the results of different models on minilmageNet, tieredImagenet, CUB and CIFAR-FS are shown in Tables I and 2 To validate the effectiveness of our proposed model, we compare with three main groups the few-shot learning method, including optimization-basee metric-based. and distribution salibration-basee There are several observations. First, we can find
7

---

Table Sway Isho and 5way5shot classification accuracy (%) on CIFAR-FS with 95% confidence intervals. * means the results cited from 44 and + denotes our implementation of Free-Lunch with their official code.
CIFAR-FS
Methods
Backbone
Swaylshot
5way5shot
MAML"
G
32-32-32-32
58.9 1.
71.5 1.0
ProtoNet
64-64-64-64
5.7
0.6
RelaNet"
45
64-96-12-2256
55.0 1.0
69.3 19.3:000 0.8
R2D2"
46
96-192-384-512
65.3
7.14
Shot-Free
47
ResNet12
69.2 w/a
84.7
MctaOpt"
48
ResNet12
0.76
0.5
RFS
44
ResNet12
0.8
86,9
Free 1 auch
14
ResNet12
73.8
86.0 0.4
Our H-OT
ResNet12
1..
8..1
Free-Lumch
WRN28+RTIoss
74.9
86.24 0.5
Our H-OT
WRN28+KTloss
75.4 0.3
87.5 0.3
that our H-OT equipped with WRN28+RTLoss and simple linear classifier outperforms all previous competing methods on 5-way- 1-shot and 5-way-5-shot settings, which illustrates the superiority of the proposed framework. Second, our H-OT equipped with ResNet12 better than other baselines except for the 5way I task minilmageNet, where in this case our H-OT is still competitive with DeepEMD 30). This observation proves the effectiveness distribution calibration and transferring the statistics from base classes novel classes using OT. Third, the comparison between our model and the strong bascline of distribution calibration-baat the validity of incorporating the adaptive weight matrix when transferring the statistics from the base classes to the novel classes. Although both methods utilize distribution calibration to perform few-shot classification, our model can more effectively transfer the knowledge of the base classes to estimate the distribution of novel samples more accurately with the help of hierarchical OT (H-OT). Fourth, in terms of 5-way 1 1-shot tasks all datasets, the performance gain from model over Free-Lunch on the CUB larger than that of other datasets. However, for 5-way 5-shot tasks on all the CUB is
methods, i.c., Free-Lunch [I4), confirms
datasets, the improvement gain from our model over Free-Lunch that on other datasets. The reason behind this might be that CUB a fine-grained image classification dataset, which makes more effective for statistics transfer compared to other datasets under the 1 1-shot setting. Furthermore, with the development of label examples in the novel classes (5-shot), our method can perform more effective knowledge transfer from base classes novel samples in the coarse-grasd goasee-rraied image classification dataset. The results reveal that the granularity of the dataset is an important factor in the current few-shot classification setting. Our model avoids the costly fine-tuning for backbone and it only takes time at the testing stage. We defer the details on comparison of computational cost to Appendix Aland summary of test results to Appendix B
significant as
5.3 Evaluation with the Cross-domain Setting
To explore the cross- cross-domain generalization ability of H-OT for the few-shot classification task, we consider practical evalu- ation setting following Chen et al. [3], where there exists do- main shift between base and novel classes (e.g., sampling base classes from a coarse-grained dataset and novel classes from a fine-grained dataset). We design two cross-domain scenarios: minilmageNet - CUB, CIFAR-FS + CUB, where CUB is the target domain while both minilmageNet and CIFAR-FS are the source domains. Compared with collecting images from a general class, collecting images from fine-grained classes might be more difficult, so these two scenarios are very meaningful reality. We consider the comparison with Free-Lunch, which is closely related to ours. As shown in Fig. [ both H-OT and Free-Lunch achieve acceptable performance these two sce- narios, proving the effectiveness of the distribution calibration framework in the cross-domain case. Besides, H-OT shows its clear advante over Free-Lunch. It indicates that H-OT can
Herarch Free-Lunch
Accuracy (Sway
82 20
77
mnn-ScB CPAR->CUS Different Scenarfos
Figure 1: 5way Ishot accuracy un- der the cross domain scenarios, i.e. minilmageNet - CUB and CIFAR- + CUB. CUB indicates that both base and novel classes are from CUB.

---

19.3
Transport plasspottaaaannnn daain
0.0139
13.0
0.0159
58.5
$8.0
0.033
0.030
0,320
67.0
Herarchical OT
Free-Lunch
drncket Ean
200
Figure 3: Top-5 closest base classes for novel sample from the "plain" class on CIFAR-FS based on our learned transport plan.
Base classes
Number
Figure 2: Accuracy minilmageNet for Swaylshot task with varying number of generated features.
better utilize the statistics from the base classes of the source domain with the adaptive weight matrix and transfer them the novel classes of the target domain. Moreover, H-OT under the CIFAR-FS + CUB (blue bar in the 3rd column) scenario still outperforms Free-Lunch under CUB (orange bar in the 1st column). These results show that H-OT is less affected by domain shift between the base and novel classes and possesses the desired cross-domain generalization ability.
Table 3: Ablation study with different cost functions (5way shhot.
Table Isayl perfor-
Models
miniImageeN Net
CUB
CIFAR-FS
tieredImageeet
mance on minilmageNet with
lunch 14
68.57 + 0.55
79.56
0.87
14.94 : 0.64
75,10=0:32
diffcrent backbones.
High-level OT+
68.79 1 0.31
80.29 1 0.25
75,09 1 0.33
75.31
High-level 1an-leeelOOF-Bur Wecewhhee
68.80 + 0.32
80.33+ 0.27
78.13+ 0.31
75.35-+0.40
High-lovel
0.31
Comé 57.34 0.45
5001
High-level Oeecos-weighted
68.78 1: 0.33
81.021 0.30
75,10 1 0.34
75.55:0.41
ReaNati
64.41 20.3
65.22: 0.37
Our H1-TT
69.04 0.29
81,23 :: 0.35
75.42
0.32
75.91
Reseett
WRNDS-RTHe
68.37 08.37-00.5
01.50-10005
0004 0.24
6261 = 0.41
5.4 Ablation Study and Qualitative Analysis
Impact of adaptive cost from low-level OT Recall that we formulate our general goal as a high-level OT problem and we propose low-level OT for computing the cost function for the high-level one in an adaptive way. The low-level OT captures the distance between a base class and a novel class. Here consider commonly-usee cost functions replace the adaptive cost, including Euclidean distance and cosine similarity between the feature space of the novel classes and the mean of the features from the base classes, denoted as High-level +T+ and High-level OT+cos. Besides, to exclude the possibility that the gain of our H-OT comes only from the effect of introducing weighted mean in the low-level OT, we also consider use the weighted mean with ( implement High-level OT, denoted as High-level OT+Euc+weighted and High-level OT+cos++eightes As summarized in Table 66, our proposed H-OT achieves better performance than these degraded variants, which confirms the validity of adaptive cost from low-level OT in few-shot classification. And introducing the weighted mean into the cost function of high-level OT does not improve the performance. indicates that the performance gain comes mainly from the low-level OT rather than the effect of weighted mean. The reason behind this might be that our proposed low-level OT considers all samples within a base class when deciding the cost between novel samples and this base class, instead of using mean of the features from the base class. Besides, even with the given cost functions, our proposed method still outperforms the competing baseline (i.e., Free-I Free-Lunch), indicating the usefulness of the transport plan (i.c., adaptive matrix) learned from high-level OT.
Different backbones Since our framework is agnostic with the choice of backbones, following Yang al. 14) we further employ other commonly used backbones. such as networks with four convolutional layers (Conv4 or Convó), Resnet10 and Resnet18, which are trained from scratch following 3.
the classification performance (5-way-1-shot) comparison between our H-OT and strong baseline (Free-Lunch) on minilmageNet with different backbones. We find that H-OT always outperforms its baseline given the same backbone. This result reveals the superiority of learning an adaptive weight matrix to select statistics of base classes on different backbones. As H-OT agnostic the feature extractor and does not require costly fine-tuning, can also conveniently work with more advanced feature extractors to get better performance.

---

Number of generated features Fig. 2 shows the variation of the classification performance of distribution calibration models with the increasing number of generated features, where we consider our H-OT (red) and Free-Lunch (green) in 5-way-1-sho classification setting for minilmageNet. We can see that increasing the number generated samples results in consistent improvement in both cases. However, given the same number of generated samples, H-OT outperforms its baseline, which only selects the top k base classes for each novel sample using Euclidean distance [14]. Interestingly, our model achieves around 68.5 accuracy with only 50 generated samples, while Free-Lunch takes 750 generated samples to achieve a similar result. Hence, the adaptive weight matrix learned by H-OT can contribute to the distribution calibration strategy for sampling more representative features.
Learned transport plan To explore whether the learned transport plan can capture the correlations between the base classes and one given sample from the novel class ("plain"), we show a list of base classes with decreasing ranks in Fig. [] The top-5 base classes are highly related to the given novel sample about their semantic meanings. classes are usually taken on the plain, such as "elephant" and "bear". It indicates that the transport plan can be successfully optimized by our H-OT and provides an elegant way to measure the closeness between base classes and novel samples. Different from Free-Lunch that only selects top-k base classes (I=2), our model takes full advantage of all base classes for learning adaptive weight for each base class. Additional quantitative results and qualitative are deferred to the Appendix B
reasonable since the images from wild animal-related
6 Conclusion
This paper introduces a novel hierarchical OT to improve the existing distribution calibration-basee algorithm for few-shot learning. We first develop low-level OT problem learn the cost between novel samples and base class, which takes into account different weights of the samples in the base class. Moreover, we further design a high-level OT for measuring the distance between novel samples and all base classes, which uses the cost learned from the low-level one to define the cost function rather than manually chosen cost functions. The proposed H-OT can capture the correlations between the novel samples and base classes with two-level hierarchy. When transferring the statistics from base classes to novel samples, we view the learned optimal transport plan between them as the adaptive weight matrix, providing an effective way weigh each base class for novel sample. Extensive experiments have been conducted, showing that our proposed framework achieves competing performance commonly few-shot learning problems and cross-domain scenarios.
7 Negative Societal Impacts
We develop a simple and effective distribution calibration approach to the few-shot learning, which has the potential to encourage researchers to derive new and better methods for few-shot learning or meta-learning. Our work may indirectly lead to a negative outcome if there is sufficiently malicious or ill-informed choice of a few-shot classification task.
Acknowledgements. This work is partially supported by grant from the Shenzhen Science and Technology Program (JCY/202103241200 1032) and Shenzhen Institute of Artificial Intelligence and Robotics for Society.
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016
[2] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015, URL http:://tax////ie http://aaxxiv.orr/ab//// 10w.p/bbb/1111116600
[3] Wei- Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification, In International Conference on Learning Representations, 2019.
10

---

[4] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering. 22(10))1345-1355 2009.
[5] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[6] Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135 PMLR, 2017.
[7] Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[8] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3630-3638, 2016.
[9] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087, 2017.
[10] Bharath Hariharan and Ross B. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Itll, October 22-29, 2017, pages 3037-3046. IEEE Computer Society, 2017.
[11] Yu-Xiong Wang, Ross B. Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7278-7288 7786. Computer Vision Foundation IEEE Computer Society, 2018.
[12] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 4367-4375. Computer Vision Foundation / IEEE Computer Society, 2018.
[13] Hang Qi, Matthew Brown, and David G. Lowe. Low-shot learning with imprinted weights. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5822-588 5830. Computer Vision Foundation / IEEE Computer Society, 2018
[14] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. In International Conference on Learning Representations, 2021.
[15) Viet Huynh, Dinh Q Phung, and He Zhao. Optimal transport for deep generative models: State of the art and research challenges. In IJCAL, pages 4450-4457, 2021.
[16] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. In International Conference Learning Representations, 2021.
[17] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In International Conference on Learning Representations, 2022.
[18] Anh Tuan Bui, Trung Le, Quan Hung Tran, He Zhao, and Dinh Phung. A unified Wasserstein distributional robustness framework for adversarial training. In International Conference on Learning Representations, 2022.
[19] Dandan Guo, Zhuo Li, Meixi Zheng, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Learning to re-weight examples with optimal transport for imbalanced classification. arXiv preprint arXiv:2208.0m 02951, 2022.
[20] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 11(5-6):355-677, 2019..

---

[21] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems. 26:2292-2300. 2013.
[22] John W Tukey et al. Exploratory data analysis, volume 2. Reading, Mass., 1977.
[23]
Ian Goodfellow, Jean Douget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. volume 27, 2014.
[24]
David E. Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by propaapoogg crrors. nature, (2386608) 1986.
[25] Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan:
An adversarial approach to few-shot learning. NeurlPS, 2:8, 2018.
[26] Hang Gao, Zheng Shou, Alireza Zareian, Hanwang Zhang, and Shih-Fu Chang. Low-shot learning via eevariance-preserviin adversarial augmentation networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 983-993, 2018.
[27] Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogério Schmidt Feris, Raja Giryes, and Alexander M. Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2850-2860, 2018.
[28] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5542-5551, 2018
[29] Jian Zhang, Chenglong Zhao, Bingbing Ni, Minghao Xu, and Xiaokang Yang. Variational few-shot learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- -Nvember 2, 2019, pages 1685-1694 IEEE, 2019.
[30] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12203-12213, 2020.
[31] Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. Learning prototype-oriented representations for meta-learning. In International Conference on Learn- ing Representations, 2022.
[32] Mikhail Yurochkin, Sebastian Claici, Edward Chien, Farzaneh Mirzazadeh, and Justin M. Solomon. Hierarchical optimal transport for document representation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1599-1609, 2019.
[33] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
[34] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenen- baum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In 6th International Conference on Learming Representations, ICLR 2018, Van- couver; BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httLl//tts///oeneiieeeeee//oooo wtt.://wwwweoooouummmdosssseeee CZ
[35] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[36] Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv:1805.081355 2018.
12

---

[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., (15())221-2555) 2015.
[38] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar himl, 5:4, 2010.
[39] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2218 2227, 2020.
[40] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
[41] Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble epoch-wise empirical bayes for few-shot learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI, volume 12361 of Lecture Notes in Computer Science, pages 404-421. Springer, 2020.
[42] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[43] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part IV, volume 12349 of Lecture Notes in Computer Science, pages 438 453.-455.555 Springer, 2020.
[44] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 266-282. Springer, 2020.
[45] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Tomr and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018
[46] Luca Bertinetto,, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv preprint arXiv: 803.08136, 2018
[47] Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 331-339, 2019.
[48] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[49] Lénaïc Chizat, Pierre Roussillon, Flavien Léger, François- -ravier Xavier Vialard, and Gabriel Peyré. Faster wasserstein distance estimation with the sinkhorn divergence. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[50] Jason Altschuler, Jonathan Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017,
December 4-9, 2017, Long Beach, CA, USA, pages 1964-1974, 2017.
13

---

[51] Pavel E. Dvurechensky, Alexander V. Gasnikov, and Alexey Kroshnin. Computational optimal transport: Complexity by accelerated gradient descent better than by sinkhorn's 's algorithm. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüssson, Stockholm, Sweden, July 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1366-1375. PMLR, 2018.
[52] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
14

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix A.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix C.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will upload code into the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/]]
15


Reviews:
[{"review_id": "oONf1FinLu4", "reviewer": "Reviewer_aD8j", "summary": "This paper is on the topic of distribution calibration methods for few-shot learning, which seek to align the novel class feature distribution to that of the base classes. Methodological and experimental comparisons are primarily with a prior distribution calibration method (Free Lunch [14]) used the average features of the two closest base classes in Euclidean space. The authors instead propose using optimal transport to compare novel classes with all base classes. Additionally, the cost function to compare a base class with a novel sample is also learned as an optimal transport problem. Experiments are done on 5-way 1-shot and 5-way 5-shot miniImageNet, TieredImageNet, CUB, and CIFAR-FS, in comparison with Free Lunch and other popular few-shot methods. Overall, results are impressive, establishing a new state-of-the-art.", "questions": "1. Line 263: Why must the features be nonnegative? \n2. How does the number of base classes affect this method? I presume that a higher number of classes scales the computational cost of this OT-based approach significantly, in a way that previous approaches (e.g. ProtoNet) do not.\n3. Free Lunch [14] found that increasing the number of top-k base classes decreased performance, while the proposed method is somewhat akin to k=B and yet sees improvement. Is the difference due to the low-level OT method?\n", "limitations": "Discussion of limitations absent. What is the added computation cost of the method? When does OT tend to fail? How limiting is the Gaussian assumption of features?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed usage of optimal transport to align base and novel distributions is very sensible, as are the improvements over the previous distribution alignment method Free Lunch. It’s almost surprising that such an approach hasn’t been attempted before.\n2. Background concepts (e.g. few-shot learning, optimal transport) are introduced well for unfamiliar readers, though some of the writing could use some improvement (see below), and the authors should take care to avoid text that is too similar to prior works (see below). \n3. The empirical results are impressive. The main results are on 5-way 1-shot and 5-way 5-shot miniImageNet, TieredImageNet, CUB, and CIFAR-FS, which are the common FSL benchmarks in recent works. While all comparisons aren’t on the same backbone, the backbones used are listed in Table 1. Regardless, I believe these are the highest accuracies I’ve seen on these few-shot benchmarks, and the improvements over the most relevant baseline are consistent. Additional experiments in cross-domain settings and ablation studies lend further support to the efficacy of the method.\n\nWeaknesses:\n1. There isn’t a discussion on computational cost of this method. Optimal transport has a reputation for being expensive, so such a discussion would be very helpful, particularly given the two-level OT used here. How long does this method take to compute the transport plan? \n2. Similar to the above point, the computational cost of the optimal transport plan scales with the number of classes involved. While this may be manageable for the small size of common FSL benchmarks, I suspect that datasets with larger numbers of classes may prove more expensive.\n3. This work seems heavily based on the “Free Lunch” paper [14], not just in methodology, but also the paper itself. Some similarities are perhaps natural, as this work takes a similar strategy and builds off of [14]. However, Section 2.2 is mostly a paraphrase of [14]’s Section 3.1, there are strong similarities in certain lines of the Related Works, and several of the tables have the same formatting. There are also some odd variable substitutions from those of [14] that seem unnecessary (e.g. topw instead of the more common top-k, \\beta for Tukey’s Ladder of Powers transformation instead of the more common \\lambda).\n4. Writing: While generally understandable, the writing isn’t always idiomatically or grammatically correct. I strongly recommend another thorough round of edits. A non-exhaustive list of examples appear under “Miscellaneous” below.\n\nMiscellaneous:\n- Line 21: [1] is a strange choice as the only cited reference for image classification.\n- Line 30: “introduce to calibrate”\n- Line 38: “Although with reasonably good performance”\n- Line 51 (and elsewhere): “labelled” vs “labeled” <= Both spellings are correct, but generally one should stick to a single version.\n- Line 75: “Owing a rich theory” <= “Due to its rich theory”? Though the “rich theory” isn’t the reason why people use OT for comparing probability distributions.\n- Line 87: “We divide whole dataset” <= missing “the”\n- Line 89/98: B is being used to denote both the number of base classes and the set of base classes.\n- Line 99: “to generate from” => “to be generated from”\n- Line 167: While I see why using only the mean may be inaccurate, I fail to see why it’d be “severely biased”.\n- Line 175: “each distributions”\n- Line 176: “should subject to”\n- Line 214: incorrect usage of “Besides”\n- Algorithm 1: “t” should be in math mode. “test the query set on classifier \\theta” => “test classifier \\theta on the query set”\n- Line 238, 269: missing Oxford comma.\n- Datasets shouldn’t be preceded with the word “the” unless followed by the word “dataset”\n- Table 1: Why aren’t the best results on CUB bolded?\n- Line 269: “CIFAR-FAS”\n- Line 341: Starting a sentence with “And”\n- [2] and [5] in the references are the same paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XXomK2oQSZ", "reviewer": "Reviewer_Rd2f", "summary": "This paper presents a calibration-based few-shot learning method to address the problem of the biased model toward novel samples. The main tool to that end is optimal transport (OT), and thereby this work attempts to overcome the limitations of its preliminary work [14] by replacing Euclidean distance with a proposed distance based on OT that can measure the distance between two distributions in a way of obtaining matching cost of one distribution to another. In order to get such a matching cost matrix, the paper suggests to learn those costs between base classes to novel samples by a low-level OT optimization problem, where the goal is to capture the importance of a sample with respect to its class. The experimental results show that the proposed method performs better than the existing few-shot learning methods including Free-Lunch [14]. ", "questions": "1. It would be better if any specific example is given to illustrate the problem of Euclidean distance is truly addressed by the H-OT distance.\n2. The main strength of this work seems to lie in the effectiveness of cross-domain settings, and hence this part can be more emphasized throughout the paper.\n3. In Sections 3.2-3.3, notations are too complicated and descriptions are not very effective. This part can hopefully be revised in a much simpler manner.", "limitations": "Yes. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(Strengths)\n1. [14] is a seminal work with clear novelty, and this work adequately finds room for improvement and indeed improves the performance of [14] by designing a more statistically sophisticated distance metric. \n2. Experiments are well designed and conducted, and the proposed H-OT method seems to work well in most of the settings. \n3. In particular, H-OT is highly effective in cross-domain settings, compared to Free-Lunch.\n\n(Weaknesses)\n1. There is no experiment to support the limitations of Free-Lunch as explicitly claimed by the authors. Even though it is observed that the overall performance has been improved, it is not quite clear whether such a performance gain indeed comes from addressing the limitations of Free-Lunch. \n2. The presentation can be further improved as the current notations are too complicated and some consecutive sentences are not smoothly linked to each other. \n3. In the overall performance comparison (Table 1), the performance gain of H-OT is somewhat marginal particularly in 1shot settings. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QGWqg6ii_0P", "reviewer": "Reviewer_pJGr", "summary": "This work studies few-shot learning and proposes a optimal-transport-based weighting scheme to leverage examples from base classes for updating the classifiers of the learning of novel classes. Built on the framework of previous work [14], this work develops a optimal-transport-based weighting scheme instead of euclidean distance, and the resulting weights per base class implicitly considers similarity of individual examples within that base class to examples in the novel classes. Evaluation shows improved performance over [14] and competitive performance on standard few-shot image classification benchmarks across multiple backbones. Extensive ablation studies are conducted to verify superiority to [14].", "questions": "See weakness.", "limitations": "Limitations are sufficiently addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. Extensive experiments across multiple backbones, cross-domain results and ablation studies. Qualitative results on optimal transport weights helps understanding of the approach.\n2. Code provided for reproducing experiment results.\n\nWeakness\n\n1. The improvement over [14] in performance is relatively small with respect to how \"principled\" (L.59, L.191) the approach is. It would be nice to get an understanding on \n- How much room of improvement is available from distribution calibration, to put the improvements that has been extracted by the proposed approach into perspective. For example, if there's oracle base class weighting, how good the performance can be on the query set? \n- Are we moving in the right direction? For example, do the class weights (e.g. Appendix Figure 4) match the oracle weights? From [14] to this paper, are weights extracted looking more like oracle weights? \n\n2. Would be nice to demonstrate how the per-example weights Mjn catches differences among examples in the same class. For example, baby is similar to sofa. Does Mjn tell apart sofa images with a baby and sofa images without a baby? How do the transport probabilities generalize in cross-domain settings?\n\n======\n\nThe authors' response has adequately addressed my concerns. I have updated the score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DYNmUlSqaHv", "reviewer": "Reviewer_brZM", "summary": "This paper proposes a novel few-shot classification algorithm based on distribution calibration. The proposed work is mainly based on Free-Lunch (Yang et al., 2021), where a feature extractor is first trained using base classes and the features of few-shot classes are augmented by the additional features drawn from Gaussian distributions. Here, the mean and covariance of the Gaussians are constructed by adequately adapting the statistics computed from the base classes, and Yang et al., 2021 proposed a simple heuristic approach for that. This paper suggests using a more theoretically grounded approach for this procedure, where the transfer of statistics from base classes is designed to be more sensitive to the importance of individual samples included in base classes or novel classes. Specifically, the paper introduces a hierarchical optimal transport algorithm where the relevance between the base and novel classes is estimated via solving hierarchical optimal transport problems and using the estimated transition matrices for constructing parameters for Gaussians to generate novel features. ", "questions": "- How does the running time of the proposed algorithm (especially the time for solving the hierarchical OT problem) scale with the number of data ($N$ or $K$)? Does hierarchical OT incur a serious computational bottleneck?\n- Can you notice a notable difference in the learned transport plan when there is a significant shift between base and novel classes? For instance, in the case of the cross-domain setting?\n- Regarding the hierarchical optimal transport, I could find a relevant paper (Yurochkin et al., 2019) where the goal is to measure the distance between documents via hierarchical optimal transport; to better capture the semantic difference between documents, the paper proposes to first solve the lower-level topic space and use the learned topic transport for solving the higher-level document space optimal transport. I guess it is worth citing this paper and discussing its relevance. ", "limitations": "As I mentioned before, most of the algorithm is based on Free-Lunch (Yang et al., 2021) and the only different part is the construction of Gaussian parameters through advanced weighted sums where the weights are computed from hierarchical optimal transport. Still, I think the contribution is quite valuable, and the algorithm demonstrated excellent performance.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The paper is well-written and easy to follow.\n- The hierarchical optimal transport approach seems reasonable and sound.\n- The experimental results are extensive and convincing. The paper also provides an ablation study and helpful visualizations.\n\nWeakness\n- The algorithm can be considered somewhat incremental because most of the procedure is based on (Yang et al., 2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept