Realistic evaluation of transductive few-shot learning
Authors: Olivier Veilleux
Malik Boudiaf
Pablo Piantanida
Ismail Ben Ayed
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-shot classification
Class imbalance
Transductive learning

Abstract: Transductive inference is widely used in few-shot learning, as it leverages the statistics of the unlabeled query set of a few-shot task, typically yielding substantially better performances than its inductive counterpart.  The current few-shot benchmarks use perfectly class-balanced tasks at inference. We argue that such an artificial regularity is unrealistic, as it assumes that the marginal label probability of the testing samples is known and fixed to the uniform distribution. In fact, in realistic scenarios, the unlabeled query sets come with arbitrary and unknown label marginals.   We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference,  removing the class-balance artefact. Specifically, we model the marginal probabilities of the classes as Dirichlet-distributed random variables, which yields a principled and realistic sampling within the simplex.  This leverages the current few-shot benchmarks, building testing tasks with arbitrary class distributions. We evaluate experimentally state-of-the-art transductive methods over 3 widely used data sets, and observe, surprisingly, substantial performance drops, even below inductive methods in some cases. Furthermore, we propose a generalization of the mutual-information loss, based on α-divergences, which can handle effectively class-distribution variations. Empirically, we show that our transductive α-divergence optimization outperforms state-of-the-art methods across several data sets, models and few-shot settings.

Realistic Evaluation of Transductive Few-Shot Learning
Olivier Veilleux ÉTS Montreal
Malik Boudiaf ÉTS Montreal
Pablo Piantanida L2S, CentraleSupélec CNRS Université Paris-Saclay
Ismail Ben Ayed ÉTS Montreal
Abstract
Transductive inference is widely used in few-shot learning, as it leverages the statis- ties the unlabeled query set of a few-shot task, typically yielding substantially bet- ter performances than its inductive counterpart. The current few-shot benchmarks use perfectly class-balanced tasks at inference. We argue that such an artificial reg- ularity is unrealistic, it assumes that the marginal label probability of the testing samples is known and fixed to the uniform distribution. In fact, in realistic scenar- ios, the unlabeled query sets come with arbitrary and unknown label marginals. We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks inference, removing the class-balance artefact. Specifically, random
we model the marginal probabilities of the classes as variables, which yields principled and realistic sampling within the simplex. This leverages the current few-shot benchmarks, building testing tasks with arbitrary class distributions. We evaluate experimentally state-of-the-as transductive meth- ods over 3 widely used data sets, and observe, surprisingly, substantial performance drops, even below inductive methods in some cases. Furthermore, we propose generalization of the mutual-information loss, based on a-divergences, which can handle effectively distriiistinotoon variations. Empirically, we show that our transductive a-divergence optimization outperforms sate-of-the--rt methods across several data sets, models and few-shot settings. Our code publicly available at //gt://// com httpa//ooooo/eeeeeeee//aaaaaaaaa Tra.sstuctttrrnnuuiiiiieeee Fee___eeett
Introduction
Deep learning models are widely dominating the field. However, their outstanding performances are often built upon training on scale labeled data sets, and the models are seriously challenged when dealing with novel classes that were not seen during training. Few-shot learning 2 3] tackles this challenge, and has recently triggered substantial interest within the community. In standard few-shot settings, model is initially trained on large-scale data containing labeled examples from set of base classes. Then, supervision for new set classes, which are different from those seen in the base training, is restricted to just one few labeled samples per class. Model generalization evaluated over few-shot tasks. Each task includes query set containing unlabeled samples for evaluation, and is supervised by a support set containing few labeled samples per new class.
The recent few-shor classification literature is abundant and widely dominated by convoluted meta- learning and upisodic-trainin strategies. To simulate generalization challenges at test times, such strategies build sequences of artificially balanced few-shot tasks (or episodes) during base training, each containing both query and support samples. Widely adopted methods within this paradigm include: Prototypical networks 4 which optimizes the log-posteriors of the query points within each base-training episode; Matching networks
3 which expresses the predictions of query points
Equaa contributions, corresponding authors: 2olivier.veilleuxx., malik. boudiaf. 1) @ ens wwe.ssccsmmtit etsmtl.ca
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

as linear functions of the support labels, while deploying episodic training and memory architectures; MAML (Model-Agnostic Meta-Learning) [5]. which encourages a model be "easy" to fine-tune; and the meta-learner in [6) which prescribes optimization as a model for few-shot learning. These popular methods have recently triggered a large body of few-shot learning literature, for instance, 12 [3]
77
T0
to list a few.
Recently, a large body of works investigated transductive inference for few-shot tasks, e.g.,
12
18 19 20 21 [2 23 among many others, showing substantial improvements in performances over inductive inference Also, as discussed in [24], most meta- veta-learning approches rely critically on transductive batch normalization (TBN) to achieve competitive performances, for instance, the methods in [5] [25] 26). among others. Adopted initially in the widely used MAML [5]. TBN performs normalization using the statistics of the query set of a given few-shot task, and yields significant increases in performances [24]. Therefore, due to the popularity of MAMI,, several meta-learning techniques have used TBN. The transductive setting is appealing for few-shot learning, and the outstanding performances observed recently resonate well with a well- well-known fact in classical transductive inference [27 28 29] On small labeled data sets, transductive inference outperforms its inductive counterpart. In few-shot learning, transductive inference has access to exactly the same training and testing data as its inductive counterpart P The difference is that it classifies all the unlabeled query samples of each single few-shot task jointly, rather than one sample at a time.
15
16
17
The current few-shot benchmarks use perfectly testing. all the classes have exactly the same number of samples, i.c., the marginal probability of the classes is assumed to be known and fixed to the uniform distribution across all tasks. This may not reflect realistic scenarios, in which testing tasks might come with arbitrary class proportions. For instance, the unlabeled query set of a task could be highly imbalanced. In fact, using perfectly balanced query sets for benchmarking the models assumes exact knowledge of the marginal distri- butions of the true labels of the testing points, but such labels are unknown. This is, undeniably, an unrealistic assumption and an important limitation of the current few-shot classification benchmarks and datasets. Furthermore, this suggests that the recent progress in performances might be, in part, due to class- balancing priors (or biases) that are encoded in state-of-the-ar transductive models. Such priors might be implicit, e.g., through carefully designed prisodic-training schemes and specialized architectures, or explicit, c.g., in the design of transductive loss functions and constraints. For instance, the best performing methods in [3 31
tasks at inference: For each task used
use explicit label- terms constraints, which strongly enforce perfect class balance within the query set of each task. In practice, those class-balance priors and assumptions may limit the applicability of the existing few-shot benchmarks and methods. In fact, our experiments show that, over few-shot tasks with random class balance, the performances of state-of-the-art methods may decrease by margins. This motivates re-considering the existing benchmarks and re-thinking the relevance of class-balance biases in state-of-the-art methods.
Contributions We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference. Specifically, we relax the assumption of perfectly balanced query sets and model the marginal probabilities of the classes as devise a principled procedure for sampling simplex vectors from the Dirichlet distribution, which is widely used in Bayesian statistics for modeling categorical events. This leverages the current few-shot benchmarks by generating testing tasks with arbitrary class distributions, thereby reflecting realistic scenarios. We evaluate experimentally -fate-of-the-art transductive few-shot methods over 3 widely used datasets, and observe that the performances decrease by important margins, albeit at various degrees, when dealing with arbitrary class distributions. In some cases, the performances drop even below the inductive baselines, which are not affected by class-distribution variations (as they do not use the query-set statistics). Furthermore, we propose generalization of the transductive mutual- information loss, based on a-divergences, which can handle effectively class-distribution variations. Empirically, we show that transductive a-divergence optimization outperforms state-of-the-ar of-t few-shot methods across different data sets, models and few-shot settings.
random variables. We
'The best- performing of few-shot methods in the transductive-inferrnce setting have achieved performances that are up to 10% higher than their inductive counterparts; see (23], for instance. rransductiv-iiffrrnnce ' 'aah single shw-shot task is treated independently of the other tasks in the transductive inference setting.
Hence, the setting does not additional unlabeled data, unlike supervisdd few-shot learning (30).
2

---

2 Standard few-shot settings
Base training Assume that we have access to a fully labelled base dataset Dhose { yi yilien where 2; E Rhsse are data points in an input space Xaass yi {0, 13 Domasl the one-hot labels, and Mbsse the set of base classes. Base training learns feature extractor fø : X + Z. with Φ its learnable parameters and 2 a (lower-dimensiona) feature space. The vast majority of the literature adopts episodic training at this stage, which consists in formatting Dbase as a series of tasks (=episodes) in order to mimic the testing stage, and train a meta-learner to produce. through a differentiable process, predictions for the query set. However, it has been repeatedly demonstrated over the last couple years that a standard supervised training followed by standard transfer learning strategies Therefore, we adopt
Marre
actually outperforms most meta-learning based approaches 13 33
20
23
a standard cross-entropy training in this work.
Testing The model is evaluated on a set of few-shot tasks, each formed with samples from Drest = fax yi Neaat where y; E {0, 1)( such that Vaose n Vtest = 0. Each task is composed of a labelled support Set = { 34 TiETs and an unlabelled query set Q = { both containing instances only from K distinct classes randomly sampled from Vest, with K < [Dessel- Leveraging
feature extractor fø pre- mre-trained on the base data, the objective is to learn, for each few-shot task,
classifier fw: Z - Ak. with W the learnale parameters and AK {y [0, / /Ex 3/ = 1} the (K - -imippee.. To simplify the equations for the rest of the paper, we use the following
notations for the posterior predictions of cach E Is U lg and for the class marginals within Q:
Pix (foee)) = P(Y k|X = xz W, Φ) and Pk
Σ TeTe
Pc. = P(Yo k: O)
11
where X and Y are the random variables associated with the raw features and labels, respectively; Xe and Ye means restriction of the random variable to set Q. The end goal is to predict the classes of the unlabeled samples in Q for each few-shot task, independently of the other tasks. A large body of works followed a caadductive-redicctin among many others. Transductive inference performs joint prediction for all the unlabeled query samples of each single few-shot task, thereby leveraging the query-set statistics. On the current benchmarks, tranductive inference often outperforms substantially its inductive counterpart (i.e., classifying one sample at a time for a given task). Note that our method is agnostic to the specific choice of classifie sw, whose parameters are learned at inference. In the experimental evaluations of our method, similarly to [23], we used Pik X exp(-İ [w,- z; 1113), with W (Wi..... wK).
setting, e.g., [1] 14 12
15 16 17 18
19 20
22 23),
(m:)) is temperature parameter and basiingaining parameters & are fixed
[
Perfectly balanced vs imbalanced tasks In standard K-way few-shot settings, the support and query sets of each task T are formed using the following procedure: (i) Randomly sample K classes }ry C Vessti (ii) For each class & E Vr, randomly sample nz support examples, such that m [8)/ K; and (iii) For each class l E Vr, randomly sample n query examples, such that n = /2/ K. Such a setting undeniably artificial as we assume and Q have the same perfectly balanced class distribution. Several recent works (35] 36, 38) studied class imbalance exclusively on the support set S. This makes sense as, in realistic scenarios, some classes might have more labelled samples than others. However, even these works rely on the assumption that query set Q is perfectly balanced. We argue that such an assumption is not realistic, as one typically has even less control over the class distribution of Q than it has over that of S. For the labeled support S, the class distribution is at least fully known and standard strategies from imbalanced supervised learning could be applied [38). This does not hold for Q, for which we need to make class predictions at testing time and whose class distribution is unknown. In fact, generating perfectly balanced tasks at test times for benchmarking the models assumes that one has access to the unknown class distributions of the query points, which requires access to their unknown labels. More importantly, artificial balancing of Q is implicitly or explicitly encoded in several transductive methods, which use the query set statistics to make class predictions, as will be discussed in section 4
'6 is either fixed, e.g., 23 or fine-tuned during inference, e.g.,
There is. however, evidence in the 34 33 while reducing the inference time.
15
literature that freezing o yields better performances [23 32
3

---

a005,,03 0.3))
e =b S,
in (5A) 20
ligh decdiy
Low dealit
Figure 1: Dirichlet density function for = 3, with different choices of parameter vector a.
3
Dirichlet-distributedd class marginals for few-shot query sets
Standard few-shot settings assume that Pk. the proportion of the query samples belonging to class within a few-shot task, is deterministic (fixed) and known priori: pk: n / = 1/K, for all k and all few-shot tasks. We propose relax this unrealistic assumption, and distribution to model the proportions (or marginal probabilities) of the classes few-shot query sets random variables. Dirichlet distributions categorical events 5| The domain of the Dirichlet distribution is the set of K K-dimensional discrete distributions, i.e., the of vectors in (K -)simplex AK = (p [0. 1] Σx Ps 1}. Let Ps denotes a random variable associated with class probability Pk. and P the random simplex vector given by P (P...... Pk). We assume that P follows Dirichlet distribution with parameter vector Dir(a). The Dirichlet distribution has the following density function:
the Dirichlet
widely used in Bayesian statisties to model --way
= (aI...., ................................ ax) ) E F ap: B(a) I
for p (P1...., Pk) Ak. with B denoting the multivariate Beta function, which could be expressed with the Gamma function 6
K
[T_,, r(me)
B(a)
r(2
Figure I illustrates the Dirichlet density for K 3, with 2- 2-simplex support represented with an equilateral triangle, whose vertices are probability vectors (1, 0, (0, 1, 0) and (0, 1). We show the density for = al k. with 1 the K K-dimensional vector whose all components are equal to 1 and concentration parameter equal 0.5, 2, 5 and 50. Note that the limiting case a + too corresponds to the standard settings with perfectly balanced tasks, where only uniform distribution, middle of the simplex, could occur as marginal distribution the classes.
i.e. the point
The following result, well-known in the literature of random variate generation (39), suggests that one could generate samples from the multivariate Dirichlet distribution via simple and standard univariate Gamma generators.
Theorem 3.1. (139 p. 594]) Let .I....,NN Nx k be K independent Gamma-distriburee random vari- ables with parameters og: N₂ ~ Gamma(ax) ) Le., the probability density of Ng. is univariate Gamme with shape parameter og- Let PA Ek N₂
No k ===1 1,...,, K. Then, P = PPP (P1..... .P) is Dirichlet
distributed: P Dir(a), with a = -(0000000 (01,...,ax) ax).
K, could be found in (39] p. 594. This result prescribes the following simple procedure for sampling random simplex vectors (pi,... PK) from the multivariate Dirichlet distribution with parameters (01,..... )x): First,
proof based on the Jacobian of random variable transformations Ps ΣX N₂
No.
draw K independent random samples (RI,.... "0l,...,n.K] from Gamma distributions, with each ny drawn from univariate density Souma (n:: ax); To do so, one could use standard random generators for the univariate Gamma density; see Chapter
[39]. Then, we set Pk ΣK; This enables generate randomly ni the number of samples of class k within query
set @, as follows: ng is the closest integer to pal Q such that Ee n = [
'Note that the Dirichlet distribution is the conjugate prior of the categorical and multinomial distributions.
'The Gamma function is given by: ((a) = So to exp(-t)dt for > O. Note that [(a) = (a - 11)
when a is a strictly positive integer.
Univariate Gamma density given by: forms (rc; ax) n expl E R.
[(as)

---

4 On the class-balance bias of the best-performing few-shot methods
As briefly evoked in |ecciinn 2 the strict balancing of the classes in both S and Q represents strong inductive bias, which few-shot methods can either meta-learn during training or leverage at inference. In this section, we explicitly show how such a class-balance prior is encoded the two best-performing transductive methods in the literature [23 31], one based on mutual- nuttullinforrmtion
maximization 23 and the other on optimal transport 31
Class-balance bias of optimal transport Recently, the transductive method in [1. referred to as PT-MAP, achieved the best performances reported in the literature on several popular benchmarks, to the best of our knowledge. However, the method explicitly embeds a class-balance prior. Formally, the objectíve is find, for each few-shot task, optimal mapping matrix M E R which could be viewed as a joint probability distribution over Xe x Ye. At inference, a hard constraint M E {M M r, L M c} for some r and Sinkhorn- Knopp algorithm. In other words, the columns and rows of M are constrained to sum to pre-defined vectors r E IR and c E IR Setting e = l as done in [31 ensures that M defines a valid joint distribution, but also crucially encodes the strong prior that all the classes within the query sets are equally likely. Such a hard constraint is detrimental to the performance in more realistic scenarios where the class distributions of the query sets could be arbitrary, and not necessarily uniform. Unsurprisingly, PT-MAP undergoes substantial performance drop in the realistic scenario with Dirichlet-distribute class proportions, with a consistent decrease in accuracy between 18 and 20 % on all benchmarks, in the 5-ways case.
enforced through the use of the
Class- balance bias of transductive nutual-information maximization Let us now have closer look at the the transductive loss minimized in [23 for given few-shot task reads:
maximization in (23]. Following the notations introduced in insecciion 2
1
CTM CE I(Xo: CE
Pis log(pa)
ps log Pk:
11
H(YalXa))
-H(Yo)
- -H(Ye|Xo) + XH(Y)) is a weighted mutual information between the query samples and their unknown labels (the mutual information corresponds to X 1), and CE - Eies Σk=1 Bik log(pis) is a supervised cross-entropy term defined over the support samples. Let us now focus our attention on the label-marginal entropy term, H(Ya). As mentioned in [23]. this term is of significant importance as it prevents trivial, single-class solutions stemming from minimizing only conditional entropy H H())lXo). However, we argue that this term also encourages class-balanced solutions. In fact, we can write it as an explicit KL divergence, which penalizes deviation of the label marginals within query set from the uniform distribution:
where I(Xe; Yo) =
K
H(YY))
= Σ 1=1
De log (Pk) = log( K) - Dxr. (p)
Therefore, minimizing marginal entropy H(Y)) is equivalent to minimizing the KL. divergence between the predicted marginal distribution p- (pi.....7 and uniform distribution uą * k. This KL penalty could harm the performances whenever the class distribution of the few-shot task no longer uniform. In line with this analysis, and unsurprisingly, we observe in section 6 that the original model in (33 also undergoes a dramatic performance drop, up to 20%. While naively removing this aurginal-entrope term leads even worse performances, we observe that simply down- wwightigg it, i.c., decreasing À in Eq. (II. can drastically alleviate the problem, in contrast to the case of optimal transport where the class-bale balance constraint is enforced in a hard manner.
Generalizing mutual information with a-divergences
In this section, we propose non-trivial, but simple generalization of the mutual-information loss in (T) based on a-divergences, which can tolerate more effectively dlas-diistiiuution variations. We identified in secciiononn 4 class-balance bias encoded in the marginal Shannon entropy term. Ideally,
5

---

d=1
d=2
gw?
a-11
Hg(p)
0.1-
02
-0.1
-0.2
0.
0.
P
p
Figure 2: (Left) a-entropy function of p o(). (Right) Gradient of a-entropy w.r. the logit I E R as function of p = (1). Best viewed in color.
we would like to extend this Ghannon-enttope term in a way that allows for more flexibility: Our purpose is control how far the predicted dubel-marging distribution, p. could depart from the uniform distribution without being heavily penalized.
5.1
Background
We argue that such flexibility could be controlled through the of a-divergences 40
42
43 44), which generalize the well-known and widely used KL. divergence. a-divergences form a whole family of divergences, which encompasses Tsallis and Renyi a-divergences, among others. In this work, we focus on Tsallis's [40 43
formulation of a-divergence. Let us first introduce
the generalized logarithm [4]: loga
(x--a 1). Using the latter, Tsallis a-divergence
naturally extends KL. For two discrete distributions = (pk k= and (gx (gx))=1=
we have:
(pll))
Σ
Phe loga
1- Σ
(3)
Pk
k=
Note that the Shannon entropy in Eq. (2) elegantly generalizes to Tsallis a-entropy:
1
Ha(p) (p) (K) Kl-a Da (plluk) a Σ
Pk
The derivation of Eq. is provided in appendix. Also, lim loga (x) = log (z)) which implies:
K
lim (p(p))) DxL (p)lg) and
lim H. = H(()) a-1
Σ
Pa. log (px)
Note that a-divergence (pl q) inherits the nice properties of the KI. divergence, including not limited to convexity with respect to both and and strict positivity (p(()) 2 0 with equality if = q. Furthermore, beyond its link to the forward KL divergence PKL. (pllg) a-divergence smoothly connects several well-known divergences, including the reverse KL divergence Dxi. (glpp) (a 0),
the Hellinger (a 0.5) and the Pearson Chi-square (a = 2) distances
5.2 Analysis of the gradients
As observed from Eq. 4 a-entropy is, just like Shannon Entropy, intrinsically biased toward the uniform distribution. Therefore, we still have not properly answered the question: why would a-entropy be better suited to imbalanced situations? We argue the that learning dynamics subtly but crucially differ. To illustrate this point, let us consider simple toy logistic-regression example. Let I E IR denotes a logit, and (l) the corresponding probability, where stands for the usual sigmoid function. The resulting probability distribution simply reads p = {p. p). Figure we plot both the a-entropy Ha (left) and its gradients 07. (right) as functions of The advantage of a-divergence now becomes clearer: as a increases, H. (p) accepts more and more deviation from the uniform distribution (p 0.5 on Figure 2). while still providing burrier preventing trivial solutions (i.e., p = or p = 1, which corresponds all the samples predicted
1). Intuitively, such behavior makes a-entropy with better suited to class imbalance than Shannon entropy.

---

5.3 Proposed formulation
In light of the previous discussions, we advocate a new a-mutual information loss, a simple but very effective extension of the Shannon mutual information in Eq. !
K
Ia = -
Pik
Pk
(5)
iETo Km
with Na the a-entropy as defined in Eq. (4) Note that our generalization in Eq. 5 has no link to the
a-mutual information derived in 45 Finally, our loss for transductive few-shot inference reads:
TT CE Yx)o):Yo)
6 Experiments
In this section, we thoroughly evaluate the most recent few-shot transductive methods using our imbalanced setting. Except for SIB [16] and LR-ICI [17 all the methods have been reproduced in our common framework. All the experiments have been executed on a single GTX 1080 Ti GPU.
Datasets We use three standard benchmarks for few-shot classification: mini- Imagenet 46). tiered- Imagenet [30] and Caltech- UCSD Birds 200 (CUB) [47. The mimi-emmgennt benchmark is a subset of the ILSVRC-12 dataset 46). composed of 60,000 color images of size 84 x 84 pixels []] It includes 100 classes, each having 600 images. In all experiments, we used the standard split of 64
base-training, 16 validation and 20 test classes 6 33 The tiered- Imagentt benchmark is larger subset of ILSVRC- 12, with 608 classes and 779, 165 color images of size 84 x 84 pixels. We used standard split of 351 base-training. 97 validation and 160 test classes. The Caltech-UCS UCCD Birds 200 (CUB) benchmark also contains images of size 84 84 pixels, with 200 classes. For CUB, we used standard split of 100 base-training. 50 validation and 50 test classes, as in 32). It is important to note that for all the splits and data-sets, the base-training, validation and test classes are all different.
Task sampling We evaluate all the methods the 1-shot 5-way, 5-shot 5-way, 10-shot 5-way and 20-shot 5-way scenarios, with the classes of the query sets randomly distributed following Dirichlet's distribution, as described in section 3 Note that the total amount of query samples [ remains fixed to 75. All the methods are evaluated by the average accuracy used different Dirichlet's s concentration parameter a for validation and testing. The validation-task generation is based task generation uses a = 2- 1 k to reflect the fact that extremely imbalanced tasks (i.e., only one class is present in the task) unlikely to happen in practical scenarios; see Figure 1 for visualization.
10,000 tasks, following [33. We
random sampling within the simplex (i.e Dirichlet with
1 x). Testing-
slyper-parameters Unless identified as directly linked to a class-balance bias, all the hyper- parameters kept similar the ones prescribed in the original papers of the reproduced methods. For instance, the marginal entropy in TIM (23) was identified in insscciion a|as penalty that encourages class balance. Therefore, the weight controlling the relative importance of this term tuned. For all methods, hyper- parameter tuning is performed on the validation set of each dataset, using the validation sampling described in the previous paragraph.
Base-training procedure All non ooi-edico methods use the same feature extractors, which are trained using the same procedure as in 23 20 via a standard cross-entropy minimization on the base classes with label smoothing. The feature extractors are trained for 90 epochs, using learning rate initialized 0.1 and divided by at epochs 45 and 66. We use batch size of 256 for ResNet- 18 and of 128 for WRN28- 10. During training, color jitter, random croping and random horizontal flipping augmentations are applied. For episodic/meta -earninn- methods, given that each requires a specific training, we use the pre-trained models provided with the GitHub repository of each method.
6.1 Main results
The main results are reported Table As baselines, we also report the performances of state-of- the-art inductive methods that do not use the statistics of the query at adaptation and are, therefore,
7

---

Table I: Comparisons state-of-the-art methods in our realistic setting on mini-ImageNet, tiered- ImageNet and CUB. Query sets are sampled following Dirichlet distribution with = 2 1 Accuracy averaged over 10,000 tasks. red arrow (L) indicates performance drop between the setting and testing procedure, and a blue arrow (1) an improvement. Arrows are not displayed for the inductive methods as, for these, there is no significant change performance between both settings (expected). " signifies the result was computationally intractable to obtain.
mini-ImageNet
Method
Network
1-shot
5-shot
10-shot
20-shol
Protonet (NEURIPS'17 (
53.4
74.2
79.2
82.4
Induct.
Baseline (1CLR 12
56.0
78.9
83.2
85.9
RN-18
Baseline++ (ICLR 19
60.4
79.7
83.8
86.3
Simpleshot (ARXiV 33
63.0
80.1
84.0
86.1
MAML (ICML'17 (5)
(4
(4
66.2 (L.5)
64.2
Versa (ICLR"19 (25
(..2)
(1.9.)
65.6 (..36)
67.3 (4.40)
Transduct.
Entropy-min (ICLR
16
(L..)
(.33)
77.2 (..0)
79.3 (L.
LR+ICI (CVPR 2020
(.11)
(.7)
(2)77
(4
PT-MAP (ARXIV 331
RN-18
60.1
67.1 18.2)
68.8 18.0)
(L
LaplacianShot (ICMI)20
(4.47)
81.6 (L.05)
84.1 0.)2)
86.0 0.5)
BD-CSPN (ECCV'20 21
(124
80.201 1.8)
82.9 1.4)
(L
TIM (NeuxIPS'2N 20 [3
(4
79.8 (114)
82.3 (.8)
(4
G-TIM (ours)
67.4
82.5
85.9
87.9
Induct.
Baseline (ICLR'19
52
62.2
81.9
85.5
87.9
Baseline++ (ICLR'19
19
WRN
64.5
82.1
85.7
87.9
Simpleshot (ARXIV 38
66.2
82.4
85.6
87.4
Entropy-mii min (ICLR'20
60.4 (1)5)
76.2
PТ-MAP
60.6 (4. 18.3)
66.8 (1 19.8)
68.5 (1. 19.3)
69.9 (L 19.0)
Transduct.
(ICLR'20 [+]
64.7 5.3)
72.5 6.7)
73.6
74.2 8.7) 8.7)
LaplacianShol (ICML/20 200
WRN
68.1 (1488)
0.6)
85.9 ( 0.4)
(1 0.6)
TIM (NBuRIPS'20 [x]
69.8 4.8)
81.6 4.3)
84,2 3.9)
85.9 3.7)
BD-CSPN (BCCV"20 21
70.4 (.1)
82.3(1
84.5 1.4)
85.7 1.1)
Q-TIM (ours)
69.8
84.8
87.9
89.7
tiered- ImageNt
1-shot
5-shot
10-shot
20-shot
Induct.
Baseline (ICLR119
63.5
83.8
87.3
89.0
Baseline++ (ICLR'19
33
RN-18
68.0
84,2
87.4
89.2
Simpleshot (AuXiv
3
69.6
84.7
87.5
89.1
Emnropy-min (ICLR'20
(1)
75.5 7.67
78.0 7.9)
79.8
(..7
Transduct.
PT-MAP (ARXIV
64.1 (1 18.8)
70.0 (1 18.8)
71.9 (1 17.8)
73.4 17.1)
LaplacianShot (ICML/20
72.3 (1-4.8)
85.7 0.5)
87.9 (L.90).
BD-CSPN (BCCV'20 A1
RN-18
74.1 2.2)
84.8 (1 1.4)
86.7 1.11
87.9 (L)
TIM (NeuxIPS'20 231
74.1 4.55
84.1 3.6)
3.33)
87.4 (L..))
LR+ICI (CVPR'20
74.6 (1 6.2)
85.1 (18)
88.0 (L
90.2 (1
0- a-TIM (ours)
74.4
86.6
89.3
90.9
Induct.
Baseline R'19 k7
64.6
84,9
88.2
89.9
Baseline++ (ICLR'14
54
WRN
68.7
85.4
88.4
Simpleshot (ARXIv 31
70.7
85.9
88.7
90.1
Entropy-min -mi (ICLR
[5]
62.9 (4
(L
Transduct.
PT-MAP (ARXIV A11)
65.1 (1 19.5)
71.0 (L 19.0)
72.5 (L 18.3)
74.0 (L 17.7)
LaplacianShot (ICML/20
73.5 (.3)
86.8 (L5)
88.6 (L 0.4)
89.6 (.2 0.2)
ID-CSPN (BCCV'20
WRN
75.4 2.3)
85.9 1.5)
87.8 1.0)
89.1 0.6)
TIM (NEURIPS'20 (3
75.8 4.5)
85.4 (1
87.3 3.2)
88.7 (19
a-TIM (ours)
76.0
87.8
90.4
91.9

---

Table 2: Comparaisons of tfate-of-the-an methods in our realistic setting on CUB. Query sets are sampled following a Dirichlet distribution with a = 2 1 Accuracy is averaged over 10,000 tasks. red arrow (L) indicates performance drop between the artificially-balance
setting and our testing procedure, and a blue arrow (t) an improvement. Arrows are not displayed for the inductive methods as, for these, there is no significant change performance between both settings (expected). '-' signifies the result was computationally intractable to obtain.
CUB
1-shot
5-shot
10-shot
20-shot
Transduct. Induct.
Baseline (ICLR'19 BX
64,6
86,9
90.6
92.7
Baseline++ (ICLR'19 52
RN-18
69,4
87.5
91.0
93.2
Simpleshot (ARXIV
70.6
87.5
90.6
92.2
PT-MAP (ARXIV [I
65.1 (L 20.4)
71.3 20.00
73.0 19.2)
72.2 (L 18.9)
Entropy- min (ICLR 20 (55)
67.5 5.3)
82.9 6.0)
(L
86.8 5.7)
LaplacianShot (ICML/20 20
73.7 5.2)
87.7 (1 1.1)
89.8 (L77)
90.6 (L 0.5)
RN-18
BD-CSPN (FCCV'20 (1)
74.5 3.4)
87.1 (1 1.8)
89.3 1.3)
90.3 (.11)
TIM (NauRIPS'20 (33)
74.8 5.5)
86.9
(1
91.7 (1 2.8)
a-TIM (ours)
75.7
89.8
92.3
94.6
unaffected by class imbalance. In the 1-shot scenario, all the transductive methods, without exception, undergo a significant drop in performances as compared to the balanced setting. Even though the best-perfrming transductive methods still outperforms the inductive ones, we observe that more than half of the transductive methods evaluated perform overall worse than inductive baselines in our realistic setting. Such a surprising finding highlights that the standard benchmarks, initially developed for the inductive setting, are not well suited to evaluate transductive methods. In particular, when evaluated with our protocol, the current state-of-the-art holder PT-MAP averages more than 18% performance drop across datasets and backbones, Entropy-Min around 7%, and TIM around 4%. Our proposed a-TIM method outperforms transductive methods across almost all task formats, datasets and backbones, and is the only method that consistently inductive baselines in fair setting. While stronger inductive baselines have been proposed in the literature 48), we show in the supplementary material that a- a-TIM keeps a consistent relative improvement when evaluated under the same setting.
6.2 Ablation studies
In-depth comparison of TIM and c-TIM While not included in the main Table 1 keeping the same hyper-parameterss parameters for TIM as prescribed in the original paper [23] would result in a drastic drop of about 18% across the benchmarks. As briefly mentioned in section 4 and implemented for tuning [23] in Table I] adjusting marginal-entropy weight À in Eq. (1) strongly helps in imbalanced scenarios, reducing the drop from 18% to only 4%. However,
sub-optimal in comparison to using a-divergences, where the only argue that such a strategy is flexibility of the
controlling the term becomes a. First, seen from Table our a-TIM achieves consistently better performances with the same budget of the standard TIM. In fact, in higher-shots scenarios (5 or higher), the performances of a-TIM substantially better that the standard mutual information (i.e. TIM). Even more crucially, we show in Figure that a-TIM does not fail drastically when is chosen sub-optimally, as opposed the case of weighting parameter À for the TIM formulation. We argue that such robustness makes of a-divergences particularly interesting choice for more practical applications, where such tuning might be intractable. Our results points to the high potential of a-divergences as loss functions for leveraging unlabelled data, beyond the few-shot scenario, e.g., in superviid or unsupervised domain adaptation problems.
optimization as
Varying imbalance severity While our main experiments used a fixed value a = 2-1 we wonder whether our conclusions generalize to different levels of imbalance. Controlling for Dirichlet's parameter naturally allows us to vary the imbalance severity. In|Figure 4] we display the results obtained by varying a, while keeping the same hyper-parameters obtained through our validation protocol. Generally, most methods follow the expected trend: as a decreases and tasks become more severely imbalanced, performances decrease, with sharpest losses for TIM 2X and PT-MAP [T]. In fact, past certain imbalance severity, the inductive baseline in 33 becomes more competitive than

---

TIM (va))
TEM (bok)
T (ma)
TIM (bea))
I-shot
L-shoo
5-shot
6-shot
20
55
mini- mimi--mageeNa et
Aveuracy
75
00
70
70
56
50
A
Qz
90
91
55-
Accurass
CUB
600
50-
83
60
75
79
00
6
2
70
A
Figure 3: Validation and Test accuracy versus X for TIM 23 and for our proposed a-TIM, using our protocol. Results are obtained with a RN-18. Best viewed in color.
most transductive methods. Interestingly, both LaplacianShot 20 and our proposed a-TIM are able to cope with extreme imbalance, while still conserving good performances on balanced tasks.
ITM
PUSHAP
LB+K3
Dolatt Ancuracy
35
11
Figure 4: 5-shot test accuracy of transductive methods versus imbalance level (lower a corresponds to more severe imbalance, as depicted in Figure I).
On the use of transductive BN In the case of imbalanced query sets, we note that transductive batch normalization (e.g as done in the popular MAML (49)) hurts the performances. This aligns with recent observations from the concurrent work [50], where a shift in the marginal label distribution is clearly identified as a failure case of statistic alignment via batch normalization.
Conclusion
We make the unfortunate observation that recent transductive few-shot methods claiming large gains over inductive ones may perform worse when evaluated with our realistic setting. The artificial balance of the query sets in the vanilla setting makes it easy for transductive methods to implicitly encode this strong prior at meta- -raintrainin stage, or even explicitly at inference. When rendering such a property obsolete at test-time, the current porfforonnnng method becomes noncompetitive, and all the transductive methods undergo performance drops. Future works could study the mixed effect of imbalance on both support and query sets. We hope that our observations encourage the community to rethink the current transductive literature, and build upon our work to provide fairer grounds of comparison between inductive and transductive methods.
10

---

Acknowledgments
This project was supported by the Natural Sciences and Engineering Research Council of Canada (Discovery Grant RGPIN 2019-05954). This project has received funding from the European Union's s Horizon 2020 research and innovation programme under the Marie Sklodowska grant agreement N+792464.
References
[1] L. Fei-Fei, R. Fergus. and P. Perona, "One-shot learning of object categories," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 594-5944699 394-611, 611, 2006.
[2] E. Miller, N. Matsakis, and P. Viola, "Learrnng from one example through shared densitics on transforms,"
in IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2000.
[3] O. Vinyals, C. Blundell, T. P. Lillicrap, K. Kavukcuoglu, and D. Wierstra, "Matching networks for one shot learning," " in Neural Information Processing Systems (NeurIPSS, 2016.
[4] J. Snell, K. Swersky, and R. Zemel, "Prototypical networks for few-shot learning," in Neural Information Processing Systems (NeurIPS), 2017.
[5] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," International Conference on Machine Learning (ICML), 2017.
S. Ravi and H. Larochelle, "Optimization as model for few-shot learning." in International Conference on Learning Representations (ICLR), 2017.
[7] F. Sung, Y. Yang. L. Zhang. T. Xiang. P. H. Tomr. and T. M. Hospedales, "Learning to compare: Relation network for few-shot learning," in /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[8] B. Oreshkin, P. R. López, and A. Lacoste, "Tadam: Task dependent adaptive metric for improved few-shot learning." in Neural Information Processing Systems (NeurlPS), 2018.
[9] N. Mishra, M. Rohaninejad, Chen, and P. Abbeel, "A simple neural attentive meta-learner" Interna- tional Conference on Learning Representations (ICIR), 2018.
[10] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, "Meta-learning with latent embedding optimization." International Conference on Learning Representations (ICLR), 2019.
[11] L. Yanbin, J. Lee, M. Park, S. Kim, E. Yang, S. Hwang, and Y. Yang. "Learning propagate labels: Transductive propagation network few-shot learning." " International Conference on Learning Repre- sentations (ICLR). 2019.
[12] R. Hou, H. Chang, M. Bingpeng, S. Shan, and X. Chen, "Cross attention network for few-shot classification," in Neural Information Processing Systems (NeurIPS), 2019.
[13] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, "Few-shot learning via embedding adaptation with set-to-set functions," in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[14] L. Qiao, Y. Shi, I., Y. Wang, Huang, and Y. Tian, "Transductive episodic-wise adaptive metric for few-shot learning," in JEEE/CVF International Conference on Computer Vision (ICCV),
[15] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto, "A baseline for few-shot image classification." in International Conference on Learning Representations (ICLR), 2020. 2020.
[16] S. X. Hu, P. G. Moreno, Y. Xiao, X. Shen, G. Obozinski, N. D. Lawrence, and A. Damianou, "Empirical bayes transductive meta-learning with synthetic gradients," in International Conference on Learning Representations (ICLR), 2020.
[17] Y. Wang, C. Xu, C. Liu, L. Zhang, and Y. Fu, "Instance credibility inference for few-shot learning," in IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[18] Y. Guo and N.-M. Cheung, "Attentive weights generation for few shot learning via information maximiza- tion," in JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[19] I.. Yang, L., 7. Zhang, X. 7hou, F., Thou, and Y. Liu, "Dpgn: Distribution propagation graph network for few-shot learning"" " JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[20] I. M. Ziko, Dolz, E. Granger, and I. Ben Ayed, "Laplacian regularized few-shot lew-shotearrning." learning." in International Conference on Machine Learning (ICML), 2020.
[21] J. Liu, L. Song, and Y. Qin. "Prototype rectification for few-shot learning." in European Conference on Computer Vision (ECCV), 2020.

---

[22] Y. Liu, B. Schiele, and Q. Sun, "An ensemble of epoch-wise empirical bayes for few-shot learning." in European Conference on Computer Vision (ECCV). 2020.
[23] M. Boudiaf, I. M. Ziko, J. Rony, J. Dolz, P. Piantanida, and I. Ben Ayed, "Transductive information
maximization for few-shot learning," in Neural Information Processing Systems (NeurlPS), 2020.
[24] J. Bronskill, J. Gordon, J. Requeima, S. Nowozin, and R. E. Tumne, "Tasknorm: Rethinking batch normalization for meta-learning: in International Conference on Machine Learning (ICML), 2020.
[25] J. Gordon, J. Bronskill, M. Bauer, S. Nowozin, and R. Turner, "Meta-learning probabilistic inference for prediction," in International Conference on Learning Representations (ICLR), 2019.
[26] X. Zhen, H. Sun, Du, Xu, Y. Yin, L. Shao, and Snock, "Learning to learn kernels with variational random features," in International Conference on Machine Learning (ICML), 2020.
[27] V. N. Vapnik, "An overview of statistical learning theory." IEEE Transactions on Neural Networks (TNN), vol. 10, no. 5. pp. 988-999, 1999.
[28] T. Joachims. "Transductive inference for text classification using support vector machines," in International Conference on Machine Learning (ICML), 1999.
[29] Z. Dengyong, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, "Learning with local and global consistency," in Neural Information Processing Syxtems (NeurIPS). 2004.
[30] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, "Meta-learning for semi-supervised few-shot classification," in International Conference on Learning Representationss (ICLR), 2018.
[31] Y. Hu. V. Gripon, and S. Pateux, "Leveraging the feature distribution in transfer-based few-shot learning." arXiv preprint:2006.66 03806, 2020.
[32] W.YY. Chen, Y.-.. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, "A closer look at few-shot classification," " in International Conference on Learning Representations (ICLR), 2019.
[33] Y. Wang. W.-L. Chao, K. Q. Weinberger, and L. van der Maaten, "Simpleshot: Revisiting nearest-neighbor classification for sew- learning," arXiv preprint:19] 19t1:mn::9910046633. 2019.
[34] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, "Rethinking few-shot image classification: a good embedding is all you need?" in European Conference on Computer Vision (ECCV). 2020.
[35] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evei, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol et al., "Meta-dataset: A dataset of datasets for learning to learn from few examples," arXiv preprint: 1909603096 2019.
[36] H. B. Lee, H. Lee, D. Na, S. Kim, M. Park, E. Yang. and S. J. Hwang, "Learning to balance; Bayesian meta- leta-learning for imbalanced and out out-of-disse oi-diisribution tasks," arXiv preprint: 19...19911111111 2019.
[37] X. Chen, H. Dai, Y. Li, X. Gao, and L.. Song. "Learming stop while learning to predict," International Conference on Machine Learning (ICMI.),00)) 2020.
[38] M. Ochal, M. Patacchiola, A. Storkey, J. Vazquez, and S. Wang. "Few-shot learning with class imbalance," arXiv preprint:2101.02555 2021.
[39] L. Devroye, Non-Uniform Random Variate Generation. Springer, 1986.
[40] H. Chernoff et al., "A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations." The Annals of Mathematical Statistics, vol. 23, no. 4, pp. 493 597, 1952.
[41] S.-L. Amari, "a-divergence is unique, belonging to both f-divergence and bregman divergence classes," "
IEEE Transactions on Information Theory, vol. 55, no. 11. pp. 4925-4931, 2000.
[42] J. Havrda and F. Charvát, "Quantification method of classification processes concept of structural a-entropy," Kybernetika, vol. 3, no. 1, pp. 30-35, 1967.
[43] C. Tsallis, "Possible generalization of boltzmann-gibbs statistics," Journal of statistical physics, vol. 52, no. 1, pp. 479-487, 1988
[44] A. Cichocki and S.-I. Amari, "Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities," Entrpy, vol. 12, no. pp- 1532-1568, 2010.
[45] S. Arimoto, "Information measures and capacity of order a for discrete memoryless channels." Topics in information theory, 1977.
[46] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bern- stein, A. C. Berg, and L. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision (IJC)), vol. 115, no. 3, pp. 21 211-252, 2015.
[47] C. Wah. S. Branson, P. Welinder, P. Perona, and S. Belongie, "The caltech-uesd 2irds-200-2011 dataset." 2011.
12

---

[48] C. Zhang. Y. Cai, G. Lin, and C. Shen. "Deepend: Few-shot image classification with differentiable earth mover's distance and structured classifiers," " in Proceedings of the IEEE/CVF conference on computer vision and puttern recognition (CVPR). 2020.
[49] C. Finn, K. Xu, and S. Levine, "Probabilistic model-agnostic meta-learning"" in Advances in Neural Information Processing Systems (NeurIPS), 2018.
[50] C. Burns and J. Steinhardt, "Limitations of post-hoc feature alignment for robustness." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
13


Reviews:
[{"review_id": "sCInkZq1Mhl", "reviewer": "Reviewer_3EcU", "summary": "This paper studies transductive evaluation in few-shot learning, which allows the model to use statistics of all the unlabeled query set examples when making class predictions. Models evaluated in transductive fashion typically attain higher metrics by using this extra information. Most work on transductive evaluation, however, assumes that the query set is evenly distributed across the possible classes. This is of course an unrealistic assumption for the real world and this paper specifically studies this aspect: (1) They evaluate in a more realistic transductive evaluation scenario where the class counts within the query set are sampled according to a Dirichlet distribution and show that most transductive evaluation-based methods perform worse in this setup; (2) They propose a modification of an existing model for transductive evaluation to work better in this scenario where the class counts can be varied in the query set.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the paper considers an under-studied but important aspect of transductive evaluation that to my knowledge hasn't been considered in previous work. Additionally the authors propose a modification to an existing transductive evaluation method [1] to deal with varied class makeup in the query set. The extension is simple but well-motivated as there was careful analysis to show why it would be expected to work better.\n\nQuality: the submission seems very technically sound. The proposal for their extension is backed up by strong reasoning and the experiments seem to be conducted well.\n\nClarity: the paper was very well-written. It motivates the problem and why it is important well. The setup of the new sampling scheme for query set class distribution was also described well. Lastly, the description of previous methods for transductive shot-learning and the proposed extension were also stated clearly and were easy to follow.\n\nSignificance: the paper presents some interesting results: (1) It shows how most of the transductive evaluation papers are benefitting from the uniform class distribution assumption in the query set. In fact, the authors observe that more than half of the transductive methods studied perform worse than inductive baselines in this more realistic setting. This definitely brings to question some of the value of this type of work as they are taking advantage of an artificial scenario. (2) The proposed transductive method has strong performance, as its the leading method for mini-ImageNet and tiered-ImageNet in most cases and sometimes gives benefit up to 2-3% compared to the leading transductive method.\n\n[1] Boudiaf et al. Transductive information maximization for few-shot learning. NeurIPS 2020.\f", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Though not discussed, the societal impact of this work is probably similar to that of any other few-shot method.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KNyYcScXZGh", "reviewer": "Reviewer_KiFQ", "summary": "The researchers use transductive inference and introduce the effect of arbitrary class distributions within the query sets of few-shot tasks, rather than using class-balanced tasks, by removing the class-balance artefact. They do this by modeling the marginal probabilities of the\nclasses as Dirichlet-distributed random variables, rather than from a known and fixed uniform distribution.\nThey assess their model by comparing transductive methods over 3 datasets, and show their experiment setting’s performance drops compared to inductive methods. They also propose a generalization of the mutual-information loss based on alpha-divergences, which is an extension of the Shannon mutual information and tolerates class-distribution variations more effectively.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The quality of the paper was fairly good. It is significant as it builds upon the current few-shots learning landscape, which utilizes the transductive and inductive methods. They propose the Dirichlet distribution to sample query sets as it creates a realistic setting, instead of using the current artificially balanced uniform distribution setting,\n\nThe paper illustrates that transductive few-shot methods claiming significant gains over inductive tasks may perform worse when evaluated with their realistic setting, based on their results. The paper demonstrates the conclusion with thorough experiments. \n\nThe paper also proposes alpha TIM formulation, with better generalizability n when training a few-shot scenario and sheds light on the use of alpha-divergences in such applications. The experiments also demonstrate that the alpha TIM consistently performs better with the same budget of hyper-parameter optimization as the standard TIM.\n\nThe hardware requirement is low, requiring only a single 1080 Ti GPU.\n\nFurthermore, in terms of the clarity of the paper, it flowed well. The paper should be reproducible.\nTypo on the caption of Table 1: \"Comparaisons\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed the limitations of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "EUH3weaFXbc", "reviewer": "Reviewer_W251", "summary": "The paper studies transductive few-shot learning in the more realistic setting with arbitrary class distributions within the query sets. In the proposed setting, authors use Dirichlet distribution to model the marginal probabilities of the classes in the query sets as random variables and generate random samples within the simplex. Focusing on two recent transductive methods PT-MAP and TIM, they first show that class-balance prior is encoded in these two methods and then extend TIM by generalizing mutual-information loss to \\alpha divergences to more effectively deal with arbitrary class distributions. The experiments validate proposed \\alpha-divergence approach on three standard benchmark datasets (mini-ImageNet, tiered-ImageNet and CUB) and show that the method outperforms inductive and transductive few-shot learning methods in the proposed setting.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper is well-motivated, clearly written and deals with a more realistic evaluation of the transductive few-shot learning methods.\n- Proposed alpha-divergence approach is a simple and effective extension of TIM for dealing with the arbitrary class distributions of the query\n- Experimental results show improvements over existing methods\n\nCons:\n- The paper focuses on only two transductive methods PT-MAP and TIM to show that the class-balance prior is encoded in these methods. What about other transductive methods? For example, from the experiments, BD-CSPN does not seem to be so sensitive (especially under assumption that some performance drop is expected). Why is the paper focused only on transfer learning and not on meta-learning approaches?  The current study is not comprehensive.\n- Experimental results are not convincing. First, there is a lack of stronger baselines in Table 1. Authors claim to outperform state-of-the-art inductive methods but do not compare to recent approaches such as MetaOptNet [1], CTM [2] and DeepEMD [3]. The difference between inductive and transductive methods in this setting seems small and with stronger baselines claims might change. \n- Why is Dirichlet parameter a set to 2? How does the performance change under different imbalance distributions? Where is the point where existing methods start performing better? Extremely, how does the method compare under a standard balanced scenario? More experiments are needed to answer these questions.\n- In Table 1, how is the \\alpha parameter tuned? Which values of \\alpha are reported in the table? Similarly for TIM, how is \\lambda tuned and which values are reported?\n\n[1] Meta-Learning with Differentiable Convex Optimization. CVPR ‘19\n\n[2] Finding Task-Relevant Features for Few-Shot Learning by Category Traversal. CVPR ‘19\n\n[3] DeepEMD: Differentiable Earth Mover’s Distance for Few-Shot Learning. CVPR ‘20\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work are not mentioned. For example, is it better to use existing methods with balanced class distributions (see cons)? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5yyshzF2Ch", "reviewer": "Reviewer_maGM", "summary": "This paper evaluates several transductive few-shot learning methods [15,28,20,21,23,17] on class-imbalanced FSL tasks. The class distribution in the query set follows a Dirichlet distribution. The support set is balanced. This work proposes a novel method based on $\\alpha$-divergences addressing the imbalance problem. The novel method $\\alpha$-TIM achieves competitive performance compared to several recent baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\\+ means strength, \\- means weakness\n\n**Originality**\n\n\\+ *Novel setting.* This paper presents a novel, more realistic setting for the evaluation of transductive learning methods. Unlike previous work on few-shot imbalance [32, 33, 34, 35], this paper focuses on the imbalance present in the query set.\n\n\\+ *Novel algorithm.* This paper presents an augmentation of the TIM algorithm [23] that combines with $\\alpha$-divergences [37, 38, 39, 40, 41]. This algorithm achieves state-of-the-art performances compared to several recent state-of-the-art transductive few-shot methods [15,28,20,21,23,17]. Unlike this previous work, the authors apply a unique mechanism to deal with class imbalance.\n\n**Quality**\n\n\\+ *The paper compares a range of recent methods, datasets and backbones.* Several recent transductive [15,28,20,21,23,17] and non-transductive [4, 29, 30] algorithms are evaluated in this novel setting. The datasets included Mini-ImageNet, tiered-ImageNet, and CUB. And backbones included WRN and ResNet-18. \n\n\\- *Evaluation on only one distribution is limiting.* The authors use one alpha value for the Dirichlet distribution (lines 257-264) to model query set imbalance during evaluation. However, it is not clear why this distribution was picked over others. The class imbalance community commonly a range of distributions, for example, step imbalance and linear imbalance (Buda et al., 2018), long-tail imbalance (Wang et al., 2018), and in few-shot previous work sampled uniformly at random [32,33,34,35], or a combination of these distributions [35]. It might be appropriate to evaluate several distributions or vary the $a$ in the Dirichlet distribution. \n\n\\- *The optimal $\\alpha$ in $\\alpha$-TIM might depend on the Dirichlet distribution and the shot setting.* Figure 3 shows that in the 1-shot setting, the optimal $\\alpha$ value for the test dataset is around the value of the $\\alpha=2$ similar to that concentration parameter used in the Dirichlet distribution ($a=2$). It could be interesting to show the correlation between $\\alpha$ and $a$ in a graph. The authors could also examine a wider $\\alpha$ range as the values for the 5-shot settings sometimes do not appear optimal. What are the optimal $\\alpha$'s for higher shot settings (e.g. 10-shot, 20-shot)? Is there a correlation between the $\\alpha$ and shots? \n\n\\- *Missing values in Table 1.* What is $alpha$-TIM performance on the balanced setting? For completeness and clarity, for methods [4, 29, 30], it might be worth adding a symbol (similar to the red/blue arrow) to indicate no change in performance. Also, there is no performance for [15] in 10-shot and 20-shot settings with mini-ImageNet and tiered ImageNet. Was that intentional? \n\n**Clarity**\n\n\\+/\\- *The paper is generally well organised but some clarifications and adjustments would help.* Some adjustments would make this work easier to follow. For instance, the bullet points for the contributions section. The paper should emphasise earlier that imbalance is studied only at the query set level, and support sets are assumed to be balanced. Some sentences could be improved, split or condensed, e.g. lines 81-83. \n\n\\- *Clarification for eq. (4).* Why do the authors multiply the $\\mathcal{D}_\\alpha (p||u_K)$ by the term $K^{1-\\alpha}$ in eq. (4)? The term does not appear in the RHS of eq. (2). In addition, it might be worth adding a derivation in the appendix. \n\n\\- *Clarification about control and imbalance of the query set (Q).* The authors claim (in lines 121-122) that recent studies [32, 33, 34, 35] \"rely on\" keeping the query set balanced. However, I can't entirely agree, and I would argue that for *non-transductive* few-shot classification methods such as those studied in [32, 33, 34, 35], the distribution of Q does not matter during meta-testing. During the meta-training (assuming full access to the base dataset), Q can be controlled arbitrarily. Keeping a balanced Q is usually done for convenience but this does not restrict \"non-transductive\" methods. However, I do agree that for *transductive* methods assuming that the query set is balanced and accessible all at once is unrealistic, and will directly affect performance (as evidenced in this paper) so studying this area seems important. \n\n\\- *No $\\lambda$ in mutual information tradeoff?*. Eq. (5) does not contain the $\\lambda$ in eq. (1). What is the reasoning behind this? What is the performance when $\\lambda$ is included and tuned with $\\alpha$?\n\n**Significance**\n\n\\+ *The paper presents a more realistic evaluation of transductive methods.* This analysis shows that several relevant and recent transductive few-shot methods underperform under query set imbalance. These results would be both interesting and valuable to the community. \n\n\\+ *The paper presents a novel and interesting approach to the problem.* The paper presents an interesting take on the class imbalance problem through the $\\alpha$-divergences. Making the method $\\alpha$-TIM significantly different from TIM and having the most significant improvements in the higher shot settings. \n\n\\- *However, the paper feels like work in progress and could be improved significantly* (as outlined here). I am inclined towards a borderline reject at the moment, but I look forward to hearing back from the authors and seeing how this work progresses in the future. \n\n**Some spelling mistakes**\n\n\\- line 133, \"priori\" --> \"a prior\"\n\n\\- line 283, \"outperforms\" --> \"outperform\"\n\n\\- line 304, \"results points\" --> \"result points\" / \"results point\"\n\n**References**\n\nBuda et al., 2018, \"A systematic study of the class imbalance problem in convolutional neural networks.\" Neural Networks 106 (2018): 249-259.\n\nWang et al., 2017, \"Learning to Model the Tail.\" NeurIPS 2017\n\n\n_____\n### POST-REBUTTAL\n_____\nThe authors have answered all of my questions above expectations through additional experiments and clear explanations. I will now recommend this paper for the conference with a score of 7 and increased confidence (conditioned on the inclusion of the additional experiments in the main paper / supplementary materials).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations were listed, but the paper would benefit from a separate limitation section. ", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission investigates the effect of a non-uniform marginal label distribution on transductive approaches to few-shot classification. When the marginal label distribution of test episodes is sampled from a Dirichlet distribution, the paper finds that state-of-the-art transductive approaches suffer a substantial performance drop. The submission also introduces a generalization of the mutual information loss which better handles class imbalance and is shown to outperform competing approaches in that setting.

Reviewers found that the paper is well-written and sheds light on an underexplored and important aspect of evaluating transductive approaches to few-shot classification. Overall they found that the proposed generalization of the mutual information loss is technically sound and convincingly backed up by experiments.

Some reviewers expressed concerns about the experimental design (the value of alpha and the number of values tried, its optimal value as a function of the shot setting, the choice of baselines to compare against) which were addressed to their satisfaction by the authors.

Ultimately, all reviewers agree that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)