Realistic Evaluation of Transductive Few-Shot Learning
Year: 2022

Abstract:
Transductive inference is widely used in few-shot learning, as it leverages the statistics of the unlabeled query set of a few-shot task, typically yielding substantially better performances than its inductive counterpart. The current few-shot benchmarks use perfectly class-balanced tasks at inference. We argue that such an artificial regularity is unrealistic, as it assumes that the marginal label probability of the testing samples is known and fixed to the uniform distribution. In fact, in realistic scenarios, the unlabeled query sets come with arbitrary and unknown label marginals. We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference, removing the class-balance artefact. Specifically, we model the marginal probabilities of the classes as Dirichlet-distributed random variables, which yields a principled and realistic sampling within the simplex. This leverages the current few-shot benchmarks, building testing tasks with arbitrary class distributions. We evaluate experimentally state-of-the-art transductive methods over 3 widely used data sets, and observe, surprisingly, substantial performance drops, even below inductive methods in some cases. Furthermore, we propose a generalization of the mutual-information loss, based on α-divergences, which can handle effectively class-distribution variations. Empirically, we show that our transductive α-divergence optimization outperforms state-of-the-art methods across several data sets, models and few-shot settings. Our code is publicly available at https://github.com/oveilleux/Realistic_Transductive_Few_Shot.

heading: 1 Introduction
text: Deep learning models are widely dominating the field. However, their outstanding performances are often built upon training on large-scale labeled data sets, and the models are seriously challenged when dealing with novel classes that were not seen during training. Few-shot learning [1, 2, 3] tackles this challenge, and has recently triggered substantial interest within the community. In standard few-shot settings, a model is initially trained on large-scale data containing labeled examples from a set of base classes. Then, supervision for a new set of classes, which are different from those seen in the base training, is restricted to just one or a few labeled samples per class. Model generalization is evaluated over few-shot tasks. Each task includes a query set containing unlabeled samples for evaluation, and is supervised by a support set containing a few labeled samples per new class.
The recent few-shot classification literature is abundant and widely dominated by convoluted metalearning and episodic-training strategies. To simulate generalization challenges at test times, such strategies build sequences of artificially balanced few-shot tasks (or episodes) during base training, each containing both query and support samples. Widely adopted methods within this paradigm include: Prototypical networks [4], which optimizes the log-posteriors of the query points within each base-training episode; Matching networks [3], which expresses the predictions of query points ∗Equal contributions, corresponding authors: {olivier.veilleux.2, malik.boudiaf.1}@ens.etsmtl.ca
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
as linear functions of the support labels, while deploying episodic training and memory architectures; MAML (Model-Agnostic Meta-Learning) [5], which encourages a model to be “easy” to fine-tune; and the meta-learner in [6], which prescribes optimization as a model for few-shot learning. These popular methods have recently triggered a large body of few-shot learning literature, for instance, [7, 8, 9, 10, 11, 12, 13], to list a few.
Recently, a large body of works investigated transductive inference for few-shot tasks, e.g., [11, 14, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23], among many others, showing substantial improvements in performances over inductive inference2. Also, as discussed in [24], most meta-learning approches rely critically on transductive batch normalization (TBN) to achieve competitive performances, for instance, the methods in [5, 25, 26], among others. Adopted initially in the widely used MAML [5], TBN performs normalization using the statistics of the query set of a given few-shot task, and yields significant increases in performances [24]. Therefore, due to the popularity of MAML, several meta-learning techniques have used TBN. The transductive setting is appealing for few-shot learning, and the outstanding performances observed recently resonate well with a well-known fact in classical transductive inference [27, 28, 29]: On small labeled data sets, transductive inference outperforms its inductive counterpart. In few-shot learning, transductive inference has access to exactly the same training and testing data as its inductive counterpart3. The difference is that it classifies all the unlabeled query samples of each single few-shot task jointly, rather than one sample at a time.
The current few-shot benchmarks use perfectly class-balanced tasks at inference: For each task used at testing, all the classes have exactly the same number of samples, i.e., the marginal probability of the classes is assumed to be known and fixed to the uniform distribution across all tasks. This may not reflect realistic scenarios, in which testing tasks might come with arbitrary class proportions. For instance, the unlabeled query set of a task could be highly imbalanced. In fact, using perfectly balanced query sets for benchmarking the models assumes exact knowledge of the marginal distributions of the true labels of the testing points, but such labels are unknown. This is, undeniably, an unrealistic assumption and an important limitation of the current few-shot classification benchmarks and datasets. Furthermore, this suggests that the recent progress in performances might be, in part, due to class-balancing priors (or biases) that are encoded in state-of-the-art transductive models. Such priors might be implicit, e.g., through carefully designed episodic-training schemes and specialized architectures, or explicit, e.g., in the design of transductive loss functions and constraints. For instance, the best performing methods in [23, 31] use explicit label-marginal terms or constraints, which strongly enforce perfect class balance within the query set of each task. In practice, those class-balance priors and assumptions may limit the applicability of the existing few-shot benchmarks and methods. In fact, our experiments show that, over few-shot tasks with random class balance, the performances of state-of-the-art methods may decrease by margins. This motivates re-considering the existing benchmarks and re-thinking the relevance of class-balance biases in state-of-the-art methods.
Contributions We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference. Specifically, we relax the assumption of perfectly balanced query sets and model the marginal probabilities of the classes as Dirichlet-distributed random variables. We devise a principled procedure for sampling simplex vectors from the Dirichlet distribution, which is widely used in Bayesian statistics for modeling categorical events. This leverages the current few-shot benchmarks by generating testing tasks with arbitrary class distributions, thereby reflecting realistic scenarios. We evaluate experimentally state-of-the-art transductive few-shot methods over 3 widely used datasets, and observe that the performances decrease by important margins, albeit at various degrees, when dealing with arbitrary class distributions. In some cases, the performances drop even below the inductive baselines, which are not affected by class-distribution variations (as they do not use the query-set statistics). Furthermore, we propose a generalization of the transductive mutualinformation loss, based on α-divergences, which can handle effectively class-distribution variations. Empirically, we show that our transductive α-divergence optimization outperforms state-of-the-art few-shot methods across different data sets, models and few-shot settings.
2The best-performing state-of-the-art few-shot methods in the transductive-inference setting have achieved performances that are up to 10% higher than their inductive counterparts; see [23], for instance.
3Each single few-shot task is treated independently of the other tasks in the transductive-inference setting. Hence, the setting does not use additional unlabeled data, unlike semi-supervised few-shot learning [30].

heading: 2 Standard few-shot settings
text: Base training Assume that we have access to a fully labelled base dataset Dbase = {xi,yi}Nbasei=1 , where xi ∈ Xbase are data points in an input space Xbase, yi ∈ {0, 1}|Ybase| the one-hot labels, and Ybase the set of base classes. Base training learns a feature extractor fφ : X → Z , withφ its learnable parameters and Z a (lower-dimensional) feature space. The vast majority of the literature adopts episodic training at this stage, which consists in formatting Dbase as a series of tasks (=episodes) in order to mimic the testing stage, and train a meta-learner to produce, through a differentiable process, predictions for the query set. However, it has been repeatedly demonstrated over the last couple years that a standard supervised training followed by standard transfer learning strategies actually outperforms most meta-learning based approaches [32, 33, 34, 20, 23]. Therefore, we adopt a standard cross-entropy training in this work.
Testing The model is evaluated on a set of few-shot tasks, each formed with samples from Dtest = {xi,yi}Ntesti=1 , where yi ∈ {0, 1}|Ytest| such that Ybase ∩ Ytest = ∅. Each task is composed of a labelled support set S = {xi, yi}i∈IS and an unlabelled query setQ = {xi}i∈IQ , both containing instances only from K distinct classes randomly sampled from Ytest, with K < |Ytest|. Leveraging a feature extractor fφ pre-trained on the base data, the objective is to learn, for each few-shot task, a classifier fW : Z → ∆K , withW the learnable parameters and ∆K = {y ∈ [0, 1]K / ∑ k yk = 1} the (K − 1)-simplex. To simplify the equations for the rest of the paper, we use the following notations for the posterior predictions of each i ∈ IS ∪ IQ and for the class marginals within Q:
pik = fW (fφ(xi))k = P(Y = k|X = xi;W ,φ) and p̂k = 1 |IQ| ∑ i∈IQ pik = P(YQ = k;W ,φ),
where X and Y are the random variables associated with the raw features and labels, respectively; XQ and YQ means restriction of the random variable to setQ. The end goal is to predict the classes of the unlabeled samples in Q for each few-shot task, independently of the other tasks. A large body of works followed a transductive-prediction setting, e.g., [11, 14, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23], among many others. Transductive inference performs a joint prediction for all the unlabeled query samples of each single few-shot task, thereby leveraging the query-set statistics. On the current benchmarks, tranductive inference often outperforms substantially its inductive counterpart (i.e., classifying one sample at a time for a given task). Note that our method is agnostic to the specific choice of classifierfW , whose parameters are learned at inference. In the experimental evaluations of our method, similarly to [23], we used pik ∝ exp(− τ2 ‖wk − zi‖ 2 ), with W := (w1, . . . ,wK), zi = fφ(xi) ‖fφ(xi)‖2 , τ is a temperature parameter and base-training parameters φ are fixed4.
Perfectly balanced vs imbalanced tasks In standard K-way few-shot settings, the support and query sets of each task T are formed using the following procedure: (i) Randomly sample K classes YT ⊂ Ytest; (ii) For each class k ∈ YT , randomly sample nSk support examples, such that nSk = |S|/K; and (iii) For each class k ∈ YT , randomly sample n Q k query examples, such that nQk = |Q|/K. Such a setting is undeniably artificial as we assume S and Q have the same perfectly balanced class distribution. Several recent works [35, 36, 37, 38] studied class imbalance exclusively on the support set S. This makes sense as, in realistic scenarios, some classes might have more labelled samples than others. However, even these works rely on the assumption that query set Q is perfectly balanced. We argue that such an assumption is not realistic, as one typically has even less control over the class distribution of Q than it has over that of S . For the labeled support S , the class distribution is at least fully known and standard strategies from imbalanced supervised learning could be applied [38]. This does not hold forQ, for which we need to make class predictions at testing time and whose class distribution is unknown. In fact, generating perfectly balanced tasks at test times for benchmarking the models assumes that one has access to the unknown class distributions of the query points, which requires access to their unknown labels. More importantly, artificial balancing of Q is implicitly or explicitly encoded in several transductive methods, which use the query set statistics to make class predictions, as will be discussed in section 4.
4φ is either fixed, e.g., [23], or fine-tuned during inference, e.g., [15]. There is, however, evidence in the literature that freezing φ yields better performances [23, 32, 34, 33], while reducing the inference time.

heading: 3 Dirichlet-distributed class marginals for few-shot query sets
text: Standard few-shot settings assume that pk, the proportion of the query samples belonging to a class k within a few-shot task, is deterministic (fixed) and known priori: pk = nQk /|Q| = 1/K, for all k and all few-shot tasks. We propose to relax this unrealistic assumption, and to use the Dirichlet distribution to model the proportions (or marginal probabilities) of the classes in few-shot query sets as random variables. Dirichlet distributions are widely used in Bayesian statistics to model K-way categorical events5. The domain of the Dirichlet distribution is the set of K-dimensional discrete distributions, i.e., the set of vectors in (K − 1)-simplex ∆K = {p ∈ [0, 1]K | ∑ k pk = 1}. Let Pk denotes a random variable associated with class probability pk, and P the random simplex vector given by P = (P1, . . . , PK). We assume that P follows a Dirichlet distribution with parameter vector a = (a1, . . . , aK) ∈ RK : P ∼ Dir(a). The Dirichlet distribution has the following density function: fDir(p;a) = 1 B(a) ∏K k=1 p ak−1 k for p = (p1, . . . , pK) ∈ ∆K , with B denoting the multivariate Beta
function, which could be expressed with the Gamma function6: B(a) = ∏K k=1 Γ(ak)
Γ( ∑K k=1 αk) .
Figure 1 illustrates the Dirichlet density for K = 3, with a 2-simplex support represented with an equilateral triangle, whose vertices are probability vectors (1, 0, 0), (0, 1, 0) and (0, 0, 1). We show the density for a = a1K , with 1K the K-dimensional vector whose all components are equal to 1 and concentration parameter a equal to 0.5, 2, 5 and 50. Note that the limiting case a → +∞ corresponds to the standard settings with perfectly balanced tasks, where only uniform distribution, i.e., the point in the middle of the simplex, could occur as marginal distribution of the classes.
The following result, well-known in the literature of random variate generation [39], suggests that one could generate samples from the multivariate Dirichlet distribution via simple and standard univariate Gamma generators.
Theorem 3.1. ([39, p. 594]) Let N1, . . . , NK be K independent Gamma-distributed random variables with parameters ak: Nk ∼ Gamma(ak), i.e., the probability density ofNk is univariate Gamma7, with shape parameter ak. Let Pk = Nk∑K
k=1Nk , k = 1, . . . ,K. Then, P = (P1, . . . , PK) is Dirichlet
distributed: P ∼ Dir(a), with a = (a1, . . . , aK).
A proof based on the Jacobian of random-variable transformations Pk = Nk∑K k=1Nk
, k = 1, . . . ,K, could be found in [39], p. 594. This result prescribes the following simple procedure for sampling random simplex vectors (p1, . . . , pK) from the multivariate Dirichlet distribution with parameters a = (a1, . . . , aK): First, we draw K independent random samples (n1, . . . , nK) from Gamma distributions, with each nk drawn from univariate density fGamma(n; ak); To do so, one could use standard random generators for the univariate Gamma density; see Chapter 9 in [39]. Then, we set pk =
nk∑K k=1 nk . This enables to generate randomly nQk , the number of samples of class k within query
set Q, as follows: nQk is the closest integer to pk|Q| such that ∑ k n Q k = |Q|.
5Note that the Dirichlet distribution is the conjugate prior of the categorical and multinomial distributions. 6The Gamma function is given by: Γ(a) = ∫∞ 0 ta−1 exp(−t)dt for a > 0. Note that Γ(a) = (a − 1)!
when a is a strictly positive integer. 7Univariate Gamma density is given by: fGamma(n; ak) = n
ak−1 exp(−n) Γ(ak) , n ∈ R.

heading: 4 On the class-balance bias of the best-performing few-shot methods
text: As briefly evoked in section 2, the strict balancing of the classes in both S and Q represents a strong inductive bias, which few-shot methods can either meta-learn during training or leverage at inference. In this section, we explicitly show how such a class-balance prior is encoded in the two best-performing transductive methods in the literature [23, 31], one based on mutual-information maximization [23] and the other on optimal transport [31].
Class-balance bias of optimal transport Recently, the transductive method in [31], referred to as PT-MAP, achieved the best performances reported in the literature on several popular benchmarks, to the best of our knowledge. However, the method explicitly embeds a class-balance prior. Formally, the objective is to find, for each few-shot task, an optimal mapping matrix M ∈ R|Q|×K+ , which could be viewed as a joint probability distribution over XQ × YQ. At inference, a hard constraint M ∈ {M : M1K = r,1|Q|M = c} for some r and c is enforced through the use of the Sinkhorn-Knopp algorithm. In other words, the columns and rows of M are constrained to sum to pre-defined vectors r ∈ R|Q| and c ∈ RK . Setting c = 1K1K as done in [31] ensures that M defines a valid joint distribution, but also crucially encodes the strong prior that all the classes within the query sets are equally likely. Such a hard constraint is detrimental to the performance in more realistic scenarios where the class distributions of the query sets could be arbitrary, and not necessarily uniform. Unsurprisingly, PT-MAP undergoes a substantial performance drop in the realistic scenario with Dirichlet-distributed class proportions, with a consistent decrease in accuracy between 18 and 20 % on all benchmarks, in the 5-ways case.
Class-balance bias of transductive mutual-information maximization Let us now have a closer look at the mutual-information maximization in [23]. Following the notations introduced in section 2, the transductive loss minimized in [23] for a given few-shot task reads:
LTIM = CE− I(XQ;YQ) = CE− 1 |IQ| ∑ i∈Q K∑ k=1
pik log(pik)︸ ︷︷ ︸ H(YQ|XQ) +λ
K∑ k=1
p̂k log p̂k︸ ︷︷ ︸ −H(YQ) , (1)
where I(XQ;YQ) = −H(YQ|XQ) + λH(YQ) is a weighted mutual information between the query samples and their unknown labels (the mutual information corresponds to λ = 1), and CE := − 1|IS | ∑ i∈S ∑K k=1 yik log(pik) is a supervised cross-entropy term defined over the support samples. Let us now focus our attention on the label-marginal entropy term,H(YQ). As mentioned in [23], this term is of significant importance as it prevents trivial, single-class solutions stemming from minimizing only conditional entropyH(YQ|XQ). However, we argue that this term also encourages class-balanced solutions. In fact, we can write it as an explicit KL divergence, which penalizes deviation of the label marginals within a query set from the uniform distribution:
H(YQ) = − K∑ k=1 p̂k log (p̂k) = log(K)−DKL(p̂‖uK). (2)
Therefore, minimizing marginal entropy H(YQ) is equivalent to minimizing the KL divergence between the predicted marginal distribution p̂ = (p̂1, . . . , p̂K) and uniform distribution uK = 1K1K . This KL penalty could harm the performances whenever the class distribution of the few-shot task is no longer uniform. In line with this analysis, and unsurprisingly, we observe in section 6 that the original model in [23] also undergoes a dramatic performance drop, up to 20%. While naively removing this marginal-entropy term leads to even worse performances, we observe that simply down-weighting it, i.e., decreasing λ in Eq. (1), can drastically alleviate the problem, in contrast to the case of optimal transport where the class-balance constraint is enforced in a hard manner.

heading: 5 Generalizing mutual information with α-divergences
text: In this section, we propose a non-trivial, but simple generalization of the mutual-information loss in (1), based on α-divergences, which can tolerate more effectively class-distribution variations. We identified in section 4 a class-balance bias encoded in the marginal Shannon entropy term. Ideally,
we would like to extend this Shannon-entropy term in a way that allows for more flexibility: Our purpose is to control how far the predicted label-marginal distribution, p̂, could depart from the uniform distribution without being heavily penalized.

heading: 5.1 Background
text: We argue that such a flexibility could be controlled through the use of α-divergences [40, 41, 42, 43, 44], which generalize the well-known and widely used KL divergence. α-divergences form a whole family of divergences, which encompasses Tsallis and Renyi α-divergences, among others. In this work, we focus on Tsallis’s [40, 43] formulation of α-divergence. Let us first introduce the generalized logarithm [44]: logα(x) = 1 1−α ( x1−α − 1 ) . Using the latter, Tsallis α-divergence naturally extends KL. For two discrete distributions p = (pk)Kk=1 and q = (qk) K k=1, we have:
Dα(p‖q) = − K∑ k=1 pk logα ( qk pk ) = 1 1− α ( 1− K∑ k=1 pαk q 1−α k ) . (3)
Note that the Shannon entropy in Eq. (2) elegantly generalizes to Tsallis α-entropy:
Hα(p) = logα(K)−K1−α Dα(p‖uK) = 1
α− 1
( 1−
∑ k pαk
) . (4)
The derivation of Eq. (4) is provided in appendix. Also, limα→1 logα(x) = log(x), which implies:
lim α→1 Dα(p‖q) = DKL(p‖q) and lim α→1 Hα(p) = H(p) = − K∑ k=1 p̂k log (p̂k) .
Note that α-divergence Dα(p‖q) inherits the nice properties of the KL divergence, including but not limited to convexity with respect to both p and q and strict positivity Dα(p‖q) ≥ 0 with equality if p = q. Furthermore, beyond its link to the forward KL divergenceDKL(p‖q), α-divergence smoothly connects several well-known divergences, including the reverse KL divergence DKL(q‖p) (α→ 0), the Hellinger (α = 0.5) and the Pearson Chi-square (α = 2) distances [44].

heading: 5.2 Analysis of the gradients
text: As observed from Eq. (4), α-entropy is, just like Shannon Entropy, intrinsically biased toward the uniform distribution. Therefore, we still have not properly answered the question: why would α-entropy be better suited to imbalanced situations? We argue the that learning dynamics subtly but crucially differ. To illustrate this point, let us consider a simple toy logistic-regression example. Let l ∈ R denotes a logit, and p = σ(l) the corresponding probability, where σ stands for the usual sigmoid function. The resulting probability distribution simply reads p = {p, 1− p}. In Figure 2, we plot both the α-entropyHα (left) and its gradients ∂Hα/∂l (right) as functions of p. The advantage of α-divergence now becomes clearer: as α increases,Hα(p) accepts more and more deviation from the uniform distribution (p = 0.5 on Figure 2), while still providing a barrier preventing trivial solutions (i.e., p = 0 or p = 1, which corresponds to all the samples predicted as 0 or 1). Intuitively, such a behavior makes α-entropy with α > 1 better suited to class imbalance than Shannon entropy.

heading: 5.3 Proposed formulation
text: In light of the previous discussions, we advocate a new α-mutual information loss, a simple but very effective extension of the Shannon mutual information in Eq. (1):
Iα(XQ;YQ) = Hα(YQ)−Hα(YQ|XQ) = 1
α− 1  1 |IQ| ∑ i∈IQ K∑ k=1 pαik − K∑ k=1 p̂ αk  (5) withHα the α-entropy as defined in Eq. (4). Note that our generalization in Eq. (5) has no link to the α-mutual information derived in [45]. Finally, our loss for transductive few-shot inference reads:
Lα-TIM = CE− Iα(XQ;YQ). (6)

heading: 6 Experiments
text: In this section, we thoroughly evaluate the most recent few-shot transductive methods using our imbalanced setting. Except for SIB [16] and LR-ICI [17] all the methods have been reproduced in our common framework. All the experiments have been executed on a single GTX 1080 Ti GPU.
Datasets We use three standard benchmarks for few-shot classification: mini-Imagenet [46], tieredImagenet [30] and Caltech-UCSD Birds 200 (CUB) [47]. The mini-Imagenet benchmark is a subset of the ILSVRC-12 dataset [46], composed of 60,000 color images of size 84 x 84 pixels [3]. It includes 100 classes, each having 600 images. In all experiments, we used the standard split of 64 base-training, 16 validation and 20 test classes [6, 33]. The tiered-Imagenet benchmark is a larger subset of ILSVRC-12, with 608 classes and 779,165 color images of size 84× 84 pixels. We used a standard split of 351 base-training, 97 validation and 160 test classes. The Caltech-UCSD Birds 200 (CUB) benchmark also contains images of size 84× 84 pixels, with 200 classes. For CUB, we used a standard split of 100 base-training, 50 validation and 50 test classes, as in [32]. It is important to note that for all the splits and data-sets, the base-training, validation and test classes are all different.
Task sampling We evaluate all the methods in the 1-shot 5-way, 5-shot 5-way, 10-shot 5-way and 20-shot 5-way scenarios, with the classes of the query sets randomly distributed following Dirichlet’s distribution, as described in section 3. Note that the total amount of query samples |Q| remains fixed to 75. All the methods are evaluated by the average accuracy over 10,000 tasks, following [33]. We used different Dirichlet’s concentration parameter a for validation and testing. The validation-task generation is based on a random sampling within the simplex (i.e Dirichlet with a = 1K). Testingtask generation uses a = 2 ·1K to reflect the fact that extremely imbalanced tasks (i.e., only one class is present in the task) are unlikely to happen in practical scenarios; see Figure 1 for visualization.
Hyper-parameters Unless identified as directly linked to a class-balance bias, all the hyperparameters are kept similar to the ones prescribed in the original papers of the reproduced methods. For instance, the marginal entropy in TIM [23] was identified in section 4 as a penalty that encourages class balance. Therefore, the weight controlling the relative importance of this term is tuned. For all methods, hyper-parameter tuning is performed on the validation set of each dataset, using the validation sampling described in the previous paragraph.
Base-training procedure All non-episodic methods use the same feature extractors, which are trained using the same procedure as in [23, 20], via a standard cross-entropy minimization on the base classes with label smoothing. The feature extractors are trained for 90 epochs, using a learning rate initialized to 0.1 and divided by 10 at epochs 45 and 66. We use a batch size of 256 for ResNet-18 and of 128 for WRN28-10. During training, color jitter, random croping and random horizontal flipping augmentations are applied. For episodic/meta-learning methods, given that each requires a specific training, we use the pre-trained models provided with the GitHub repository of each method.

heading: 6.1 Main results
text: The main results are reported in Table 1. As baselines, we also report the performances of state-ofthe-art inductive methods that do not use the statistics of the query set at adaptation and are, therefore,
unaffected by class imbalance. In the 1-shot scenario, all the transductive methods, without exception, undergo a significant drop in performances as compared to the balanced setting. Even though the best-performing transductive methods still outperforms the inductive ones, we observe that more than half of the transductive methods evaluated perform overall worse than inductive baselines in our realistic setting. Such a surprising finding highlights that the standard benchmarks, initially developed for the inductive setting, are not well suited to evaluate transductive methods. In particular, when evaluated with our protocol, the current state-of-the-art holder PT-MAP averages more than 18% performance drop across datasets and backbones, Entropy-Min around 7%, and TIM around 4%. Our proposed α-TIM method outperforms transductive methods across almost all task formats, datasets and backbones, and is the only method that consistently inductive baselines in fair setting. While stronger inductive baselines have been proposed in the literature [48], we show in the supplementary material that α-TIM keeps a consistent relative improvement when evaluated under the same setting.

heading: 6.2 Ablation studies
text: In-depth comparison of TIM and α-TIM While not included in the main Table 1, keeping the same hyper-parameters for TIM as prescribed in the original paper [23] would result in a drastic drop of about 18% across the benchmarks. As briefly mentioned in section 4 and implemented for tuning [23] in Table 1, adjusting marginal-entropy weight λ in Eq. (1) strongly helps in imbalanced scenarios, reducing the drop from 18% to only 4%. However, we argue that such a strategy is sub-optimal in comparison to using α-divergences, where the only hyper-parameter controlling the flexibility of the marginal-distribution term becomes α. First, as seen from Table 1, our α-TIM achieves consistently better performances with the same budget of hyper-parameter optimization as the standard TIM. In fact, in higher-shots scenarios (5 or higher), the performances of α-TIM are substantially better that the standard mutual information (i.e. TIM). Even more crucially, we show in Figure 3 that α-TIM does not fail drastically when α is chosen sub-optimally, as opposed to the case of weighting parameter λ for the TIM formulation. We argue that such a robustness makes of α-divergences a particularly interesting choice for more practical applications, where such a tuning might be intractable. Our results points to the high potential of α-divergences as loss functions for leveraging unlabelled data, beyond the few-shot scenario, e.g., in semi-supervised or unsupervised domain adaptation problems.
Varying imbalance severity While our main experiments used a fixed value a = 2·1K , we wonder whether our conclusions generalize to different levels of imbalance. Controlling for Dirichlet’s parameter a naturally allows us to vary the imbalance severity. In Figure 4, we display the results obtained by varying a, while keeping the same hyper-parameters obtained through our validation protocol. Generally, most methods follow the expected trend: as a decreases and tasks become more severely imbalanced, performances decrease, with sharpest losses for TIM [23] and PT-MAP [31]. In fact, past a certain imbalance severity, the inductive baseline in [33] becomes more competitive than
most transductive methods. Interestingly, both LaplacianShot [20] and our proposed α-TIM are able to cope with extreme imbalance, while still conserving good performances on balanced tasks.
On the use of transductive BN In the case of imbalanced query sets, we note that transductive batch normalization (e.g as done in the popular MAML [49]) hurts the performances. This aligns with recent observations from the concurrent work in [50], where a shift in the marginal label distribution is clearly identified as a failure case of statistic alignment via batch normalization.
Conclusion
We make the unfortunate observation that recent transductive few-shot methods claiming large gains over inductive ones may perform worse when evaluated with our realistic setting. The artificial balance of the query sets in the vanilla setting makes it easy for transductive methods to implicitly encode this strong prior at meta-training stage, or even explicitly at inference. When rendering such a property obsolete at test-time, the current top-performing method becomes noncompetitive, and all the transductive methods undergo performance drops. Future works could study the mixed effect of imbalance on both support and query sets. We hope that our observations encourage the community to rethink the current transductive literature, and build upon our work to provide fairer grounds of comparison between inductive and transductive methods.

heading: Acknowledgments
text: This project was supported by the Natural Sciences and Engineering Research Council of Canada (Discovery Grant RGPIN 2019-05954). This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement №792464.

Review summaries:
This paper evaluates several transductive few-shot learning methods [15,28,20,21,23,17] on class-imbalanced FSL tasks. The class distribution in the query set follows a Dirichlet distribution. The support set is balanced. This work proposes a novel method based on $\alpha$-divergences addressing the imbalance problem. The novel method $\alpha$-TIM achieves competitive performance compared to several recent baselines.

The paper studies transductive few-shot learning in the more realistic setting with arbitrary class distributions within the query sets. In the proposed setting, authors use Dirichlet distribution to model the marginal probabilities of the classes in the query sets as random variables and generate random samples within the simplex. Focusing on two recent transductive methods PT-MAP and TIM, they first show that class-balance prior is encoded in these two methods and then extend TIM by generalizing mutual-information loss to \alpha divergences to more effectively deal with arbitrary class distributions. The experiments validate proposed \alpha-divergence approach on three standard benchmark datasets (mini-ImageNet, tiered-ImageNet and CUB) and show that the method outperforms inductive and transductive few-shot learning methods in the proposed setting.

This paper studies transductive evaluation in few-shot learning, which allows the model to use statistics of all the unlabeled query set examples when making class predictions. Models evaluated in transductive fashion typically attain higher metrics by using this extra information. Most work on transductive evaluation, however, assumes that the query set is evenly distributed across the possible classes. This is of course an unrealistic assumption for the real world and this paper specifically studies this aspect: (1) They evaluate in a more realistic transductive evaluation scenario where the class counts within the query set are sampled according to a Dirichlet distribution and show that most transductive evaluation-based methods perform worse in this setup; (2) They propose a modification of an existing model for transductive evaluation to work better in this scenario where the class counts can be varied in the query set.

The researchers use transductive inference and introduce the effect of arbitrary class distributions within the query sets of few-shot tasks, rather than using class-balanced tasks, by removing the class-balance artefact. They do this by modeling the marginal probabilities of the classes as Dirichlet-distributed random variables, rather than from a known and fixed uniform distribution. They assess their model by comparing transductive methods over 3 datasets, and show their experiment setting’s performance drops compared to inductive methods. They also propose a generalization of the mutual-information loss based on alpha-divergences, which is an extension of the Shannon mutual information and tolerates class-distribution variations more effectively.