Realistic evaluation of transductive few-shot learning

1 Introduction . Deep learning models are widely dominating the field . However , their outstanding performances are often built upon training on large-scale labeled data sets , and the models are seriously challenged when dealing with novel classes that were not seen during training . Few-shot learning [ 1 , 2 , 3 ] tackles this challenge , and has recently triggered substantial interest within the community . In standard few-shot settings , a model is initially trained on large-scale data containing labeled examples from a set of base classes . Then , supervision for a new set of classes , which are different from those seen in the base training , is restricted to just one or a few labeled samples per class . Model generalization is evaluated over few-shot tasks . Each task includes a query set containing unlabeled samples for evaluation , and is supervised by a support set containing a few labeled samples per new class . The recent few-shot classification literature is abundant and widely dominated by convoluted metalearning and episodic-training strategies . To simulate generalization challenges at test times , such strategies build sequences of artificially balanced few-shot tasks ( or episodes ) during base training , each containing both query and support samples . Widely adopted methods within this paradigm include : Prototypical networks [ 4 ] , which optimizes the log-posteriors of the query points within each base-training episode ; Matching networks [ 3 ] , which expresses the predictions of query points ∗Equal contributions , corresponding authors : { olivier.veilleux.2 , malik.boudiaf.1 } @ ens.etsmtl.ca 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as linear functions of the support labels , while deploying episodic training and memory architectures ; MAML ( Model-Agnostic Meta-Learning ) [ 5 ] , which encourages a model to be “ easy ” to fine-tune ; and the meta-learner in [ 6 ] , which prescribes optimization as a model for few-shot learning . These popular methods have recently triggered a large body of few-shot learning literature , for instance , [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ] , to list a few . Recently , a large body of works investigated transductive inference for few-shot tasks , e.g. , [ 11 , 14 , 12 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] , among many others , showing substantial improvements in performances over inductive inference2 . Also , as discussed in [ 24 ] , most meta-learning approches rely critically on transductive batch normalization ( TBN ) to achieve competitive performances , for instance , the methods in [ 5 , 25 , 26 ] , among others . Adopted initially in the widely used MAML [ 5 ] , TBN performs normalization using the statistics of the query set of a given few-shot task , and yields significant increases in performances [ 24 ] . Therefore , due to the popularity of MAML , several meta-learning techniques have used TBN . The transductive setting is appealing for few-shot learning , and the outstanding performances observed recently resonate well with a well-known fact in classical transductive inference [ 27 , 28 , 29 ] : On small labeled data sets , transductive inference outperforms its inductive counterpart . In few-shot learning , transductive inference has access to exactly the same training and testing data as its inductive counterpart3 . The difference is that it classifies all the unlabeled query samples of each single few-shot task jointly , rather than one sample at a time . The current few-shot benchmarks use perfectly class-balanced tasks at inference : For each task used at testing , all the classes have exactly the same number of samples , i.e. , the marginal probability of the classes is assumed to be known and fixed to the uniform distribution across all tasks . This may not reflect realistic scenarios , in which testing tasks might come with arbitrary class proportions . For instance , the unlabeled query set of a task could be highly imbalanced . In fact , using perfectly balanced query sets for benchmarking the models assumes exact knowledge of the marginal distributions of the true labels of the testing points , but such labels are unknown . This is , undeniably , an unrealistic assumption and an important limitation of the current few-shot classification benchmarks and datasets . Furthermore , this suggests that the recent progress in performances might be , in part , due to class-balancing priors ( or biases ) that are encoded in state-of-the-art transductive models . Such priors might be implicit , e.g. , through carefully designed episodic-training schemes and specialized architectures , or explicit , e.g. , in the design of transductive loss functions and constraints . For instance , the best performing methods in [ 23 , 31 ] use explicit label-marginal terms or constraints , which strongly enforce perfect class balance within the query set of each task . In practice , those class-balance priors and assumptions may limit the applicability of the existing few-shot benchmarks and methods . In fact , our experiments show that , over few-shot tasks with random class balance , the performances of state-of-the-art methods may decrease by margins . This motivates re-considering the existing benchmarks and re-thinking the relevance of class-balance biases in state-of-the-art methods . Contributions We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference . Specifically , we relax the assumption of perfectly balanced query sets and model the marginal probabilities of the classes as Dirichlet-distributed random variables . We devise a principled procedure for sampling simplex vectors from the Dirichlet distribution , which is widely used in Bayesian statistics for modeling categorical events . This leverages the current few-shot benchmarks by generating testing tasks with arbitrary class distributions , thereby reflecting realistic scenarios . We evaluate experimentally state-of-the-art transductive few-shot methods over 3 widely used datasets , and observe that the performances decrease by important margins , albeit at various degrees , when dealing with arbitrary class distributions . In some cases , the performances drop even below the inductive baselines , which are not affected by class-distribution variations ( as they do not use the query-set statistics ) . Furthermore , we propose a generalization of the transductive mutualinformation loss , based on α-divergences , which can handle effectively class-distribution variations . Empirically , we show that our transductive α-divergence optimization outperforms state-of-the-art few-shot methods across different data sets , models and few-shot settings . 2The best-performing state-of-the-art few-shot methods in the transductive-inference setting have achieved performances that are up to 10 % higher than their inductive counterparts ; see [ 23 ] , for instance . 3Each single few-shot task is treated independently of the other tasks in the transductive-inference setting . Hence , the setting does not use additional unlabeled data , unlike semi-supervised few-shot learning [ 30 ] . 2 Standard few-shot settings . Base training Assume that we have access to a fully labelled base dataset Dbase = { xi , yi } Nbasei=1 , where xi ∈ Xbase are data points in an input space Xbase , yi ∈ { 0 , 1 } |Ybase| the one-hot labels , and Ybase the set of base classes . Base training learns a feature extractor fφ : X → Z , withφ its learnable parameters and Z a ( lower-dimensional ) feature space . The vast majority of the literature adopts episodic training at this stage , which consists in formatting Dbase as a series of tasks ( =episodes ) in order to mimic the testing stage , and train a meta-learner to produce , through a differentiable process , predictions for the query set . However , it has been repeatedly demonstrated over the last couple years that a standard supervised training followed by standard transfer learning strategies actually outperforms most meta-learning based approaches [ 32 , 33 , 34 , 20 , 23 ] . Therefore , we adopt a standard cross-entropy training in this work . Testing The model is evaluated on a set of few-shot tasks , each formed with samples from Dtest = { xi , yi } Ntesti=1 , where yi ∈ { 0 , 1 } |Ytest| such that Ybase ∩ Ytest = ∅ . Each task is composed of a labelled support set S = { xi , yi } i∈IS and an unlabelled query setQ = { xi } i∈IQ , both containing instances only from K distinct classes randomly sampled from Ytest , with K < |Ytest| . Leveraging a feature extractor fφ pre-trained on the base data , the objective is to learn , for each few-shot task , a classifier fW : Z → ∆K , withW the learnable parameters and ∆K = { y ∈ [ 0 , 1 ] K / ∑ k yk = 1 } the ( K − 1 ) -simplex . To simplify the equations for the rest of the paper , we use the following notations for the posterior predictions of each i ∈ IS ∪ IQ and for the class marginals within Q : pik = fW ( fφ ( xi ) ) k = P ( Y = k|X = xi ; W , φ ) and p̂k = 1 |IQ| ∑ i∈IQ pik = P ( YQ = k ; W , φ ) , where X and Y are the random variables associated with the raw features and labels , respectively ; XQ and YQ means restriction of the random variable to setQ . The end goal is to predict the classes of the unlabeled samples in Q for each few-shot task , independently of the other tasks . A large body of works followed a transductive-prediction setting , e.g. , [ 11 , 14 , 12 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] , among many others . Transductive inference performs a joint prediction for all the unlabeled query samples of each single few-shot task , thereby leveraging the query-set statistics . On the current benchmarks , tranductive inference often outperforms substantially its inductive counterpart ( i.e. , classifying one sample at a time for a given task ) . Note that our method is agnostic to the specific choice of classifierfW , whose parameters are learned at inference . In the experimental evaluations of our method , similarly to [ 23 ] , we used pik ∝ exp ( − τ2 ‖wk − zi‖ 2 ) , with W : = ( w1 , . . . , wK ) , zi = fφ ( xi ) ‖fφ ( xi ) ‖2 , τ is a temperature parameter and base-training parameters φ are fixed4 . Perfectly balanced vs imbalanced tasks In standard K-way few-shot settings , the support and query sets of each task T are formed using the following procedure : ( i ) Randomly sample K classes YT ⊂ Ytest ; ( ii ) For each class k ∈ YT , randomly sample nSk support examples , such that nSk = |S|/K ; and ( iii ) For each class k ∈ YT , randomly sample n Q k query examples , such that nQk = |Q|/K . Such a setting is undeniably artificial as we assume S and Q have the same perfectly balanced class distribution . Several recent works [ 35 , 36 , 37 , 38 ] studied class imbalance exclusively on the support set S. This makes sense as , in realistic scenarios , some classes might have more labelled samples than others . However , even these works rely on the assumption that query set Q is perfectly balanced . We argue that such an assumption is not realistic , as one typically has even less control over the class distribution of Q than it has over that of S . For the labeled support S , the class distribution is at least fully known and standard strategies from imbalanced supervised learning could be applied [ 38 ] . This does not hold forQ , for which we need to make class predictions at testing time and whose class distribution is unknown . In fact , generating perfectly balanced tasks at test times for benchmarking the models assumes that one has access to the unknown class distributions of the query points , which requires access to their unknown labels . More importantly , artificial balancing of Q is implicitly or explicitly encoded in several transductive methods , which use the query set statistics to make class predictions , as will be discussed in section 4 . 4φ is either fixed , e.g. , [ 23 ] , or fine-tuned during inference , e.g. , [ 15 ] . There is , however , evidence in the literature that freezing φ yields better performances [ 23 , 32 , 34 , 33 ] , while reducing the inference time .

Summary: This paper studies transductive evaluation in few-shot learning, which allows the model to use statistics of all the unlabeled query set examples when making class predictions. Models evaluated in transductive fashion typically attain higher metrics by using this extra information. Most work on transductive evaluation, however, assumes that the query set is evenly distributed across the possible classes. This is of course an unrealistic assumption for the real world and this paper specifically studies this aspect: (1) They evaluate in a more realistic transductive evaluation scenario where the class counts within the query set are sampled according to a Dirichlet distribution and show that most transductive evaluation-based methods perform worse in this setup; (2) They propose a modification of an existing model for transductive evaluation to work better in this scenario where the class counts can be varied in the query set.