Realistic evaluation of transductive few-shot learning

1 Introduction . Deep learning models are widely dominating the field . However , their outstanding performances are often built upon training on large-scale labeled data sets , and the models are seriously challenged when dealing with novel classes that were not seen during training . Few-shot learning [ 1 , 2 , 3 ] tackles this challenge , and has recently triggered substantial interest within the community . In standard few-shot settings , a model is initially trained on large-scale data containing labeled examples from a set of base classes . Then , supervision for a new set of classes , which are different from those seen in the base training , is restricted to just one or a few labeled samples per class . Model generalization is evaluated over few-shot tasks . Each task includes a query set containing unlabeled samples for evaluation , and is supervised by a support set containing a few labeled samples per new class . The recent few-shot classification literature is abundant and widely dominated by convoluted metalearning and episodic-training strategies . To simulate generalization challenges at test times , such strategies build sequences of artificially balanced few-shot tasks ( or episodes ) during base training , each containing both query and support samples . Widely adopted methods within this paradigm include : Prototypical networks [ 4 ] , which optimizes the log-posteriors of the query points within each base-training episode ; Matching networks [ 3 ] , which expresses the predictions of query points ∗Equal contributions , corresponding authors : { olivier.veilleux.2 , malik.boudiaf.1 } @ ens.etsmtl.ca 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as linear functions of the support labels , while deploying episodic training and memory architectures ; MAML ( Model-Agnostic Meta-Learning ) [ 5 ] , which encourages a model to be “ easy ” to fine-tune ; and the meta-learner in [ 6 ] , which prescribes optimization as a model for few-shot learning . These popular methods have recently triggered a large body of few-shot learning literature , for instance , [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ] , to list a few . Recently , a large body of works investigated transductive inference for few-shot tasks , e.g. , [ 11 , 14 , 12 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] , among many others , showing substantial improvements in performances over inductive inference2 . Also , as discussed in [ 24 ] , most meta-learning approches rely critically on transductive batch normalization ( TBN ) to achieve competitive performances , for instance , the methods in [ 5 , 25 , 26 ] , among others . Adopted initially in the widely used MAML [ 5 ] , TBN performs normalization using the statistics of the query set of a given few-shot task , and yields significant increases in performances [ 24 ] . Therefore , due to the popularity of MAML , several meta-learning techniques have used TBN . The transductive setting is appealing for few-shot learning , and the outstanding performances observed recently resonate well with a well-known fact in classical transductive inference [ 27 , 28 , 29 ] : On small labeled data sets , transductive inference outperforms its inductive counterpart . In few-shot learning , transductive inference has access to exactly the same training and testing data as its inductive counterpart3 . The difference is that it classifies all the unlabeled query samples of each single few-shot task jointly , rather than one sample at a time . The current few-shot benchmarks use perfectly class-balanced tasks at inference : For each task used at testing , all the classes have exactly the same number of samples , i.e. , the marginal probability of the classes is assumed to be known and fixed to the uniform distribution across all tasks . This may not reflect realistic scenarios , in which testing tasks might come with arbitrary class proportions . For instance , the unlabeled query set of a task could be highly imbalanced . In fact , using perfectly balanced query sets for benchmarking the models assumes exact knowledge of the marginal distributions of the true labels of the testing points , but such labels are unknown . This is , undeniably , an unrealistic assumption and an important limitation of the current few-shot classification benchmarks and datasets . Furthermore , this suggests that the recent progress in performances might be , in part , due to class-balancing priors ( or biases ) that are encoded in state-of-the-art transductive models . Such priors might be implicit , e.g. , through carefully designed episodic-training schemes and specialized architectures , or explicit , e.g. , in the design of transductive loss functions and constraints . For instance , the best performing methods in [ 23 , 31 ] use explicit label-marginal terms or constraints , which strongly enforce perfect class balance within the query set of each task . In practice , those class-balance priors and assumptions may limit the applicability of the existing few-shot benchmarks and methods . In fact , our experiments show that , over few-shot tasks with random class balance , the performances of state-of-the-art methods may decrease by margins . This motivates re-considering the existing benchmarks and re-thinking the relevance of class-balance biases in state-of-the-art methods . Contributions We introduce and study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference . Specifically , we relax the assumption of perfectly balanced query sets and model the marginal probabilities of the classes as Dirichlet-distributed random variables . We devise a principled procedure for sampling simplex vectors from the Dirichlet distribution , which is widely used in Bayesian statistics for modeling categorical events . This leverages the current few-shot benchmarks by generating testing tasks with arbitrary class distributions , thereby reflecting realistic scenarios . We evaluate experimentally state-of-the-art transductive few-shot methods over 3 widely used datasets , and observe that the performances decrease by important margins , albeit at various degrees , when dealing with arbitrary class distributions . In some cases , the performances drop even below the inductive baselines , which are not affected by class-distribution variations ( as they do not use the query-set statistics ) . Furthermore , we propose a generalization of the transductive mutualinformation loss , based on α-divergences , which can handle effectively class-distribution variations . Empirically , we show that our transductive α-divergence optimization outperforms state-of-the-art few-shot methods across different data sets , models and few-shot settings . 2The best-performing state-of-the-art few-shot methods in the transductive-inference setting have achieved performances that are up to 10 % higher than their inductive counterparts ; see [ 23 ] , for instance . 3Each single few-shot task is treated independently of the other tasks in the transductive-inference setting . Hence , the setting does not use additional unlabeled data , unlike semi-supervised few-shot learning [ 30 ] . 2 Standard few-shot settings . Base training Assume that we have access to a fully labelled base dataset Dbase = { xi , yi } Nbasei=1 , where xi ∈ Xbase are data points in an input space Xbase , yi ∈ { 0 , 1 } |Ybase| the one-hot labels , and Ybase the set of base classes . Base training learns a feature extractor fφ : X → Z , withφ its learnable parameters and Z a ( lower-dimensional ) feature space . The vast majority of the literature adopts episodic training at this stage , which consists in formatting Dbase as a series of tasks ( =episodes ) in order to mimic the testing stage , and train a meta-learner to produce , through a differentiable process , predictions for the query set . However , it has been repeatedly demonstrated over the last couple years that a standard supervised training followed by standard transfer learning strategies actually outperforms most meta-learning based approaches [ 32 , 33 , 34 , 20 , 23 ] . Therefore , we adopt a standard cross-entropy training in this work . Testing The model is evaluated on a set of few-shot tasks , each formed with samples from Dtest = { xi , yi } Ntesti=1 , where yi ∈ { 0 , 1 } |Ytest| such that Ybase ∩ Ytest = ∅ . Each task is composed of a labelled support set S = { xi , yi } i∈IS and an unlabelled query setQ = { xi } i∈IQ , both containing instances only from K distinct classes randomly sampled from Ytest , with K < |Ytest| . Leveraging a feature extractor fφ pre-trained on the base data , the objective is to learn , for each few-shot task , a classifier fW : Z → ∆K , withW the learnable parameters and ∆K = { y ∈ [ 0 , 1 ] K / ∑ k yk = 1 } the ( K − 1 ) -simplex . To simplify the equations for the rest of the paper , we use the following notations for the posterior predictions of each i ∈ IS ∪ IQ and for the class marginals within Q : pik = fW ( fφ ( xi ) ) k = P ( Y = k|X = xi ; W , φ ) and p̂k = 1 |IQ| ∑ i∈IQ pik = P ( YQ = k ; W , φ ) , where X and Y are the random variables associated with the raw features and labels , respectively ; XQ and YQ means restriction of the random variable to setQ . The end goal is to predict the classes of the unlabeled samples in Q for each few-shot task , independently of the other tasks . A large body of works followed a transductive-prediction setting , e.g. , [ 11 , 14 , 12 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] , among many others . Transductive inference performs a joint prediction for all the unlabeled query samples of each single few-shot task , thereby leveraging the query-set statistics . On the current benchmarks , tranductive inference often outperforms substantially its inductive counterpart ( i.e. , classifying one sample at a time for a given task ) . Note that our method is agnostic to the specific choice of classifierfW , whose parameters are learned at inference . In the experimental evaluations of our method , similarly to [ 23 ] , we used pik ∝ exp ( − τ2 ‖wk − zi‖ 2 ) , with W : = ( w1 , . . . , wK ) , zi = fφ ( xi ) ‖fφ ( xi ) ‖2 , τ is a temperature parameter and base-training parameters φ are fixed4 . Perfectly balanced vs imbalanced tasks In standard K-way few-shot settings , the support and query sets of each task T are formed using the following procedure : ( i ) Randomly sample K classes YT ⊂ Ytest ; ( ii ) For each class k ∈ YT , randomly sample nSk support examples , such that nSk = |S|/K ; and ( iii ) For each class k ∈ YT , randomly sample n Q k query examples , such that nQk = |Q|/K . Such a setting is undeniably artificial as we assume S and Q have the same perfectly balanced class distribution . Several recent works [ 35 , 36 , 37 , 38 ] studied class imbalance exclusively on the support set S. This makes sense as , in realistic scenarios , some classes might have more labelled samples than others . However , even these works rely on the assumption that query set Q is perfectly balanced . We argue that such an assumption is not realistic , as one typically has even less control over the class distribution of Q than it has over that of S . For the labeled support S , the class distribution is at least fully known and standard strategies from imbalanced supervised learning could be applied [ 38 ] . This does not hold forQ , for which we need to make class predictions at testing time and whose class distribution is unknown . In fact , generating perfectly balanced tasks at test times for benchmarking the models assumes that one has access to the unknown class distributions of the query points , which requires access to their unknown labels . More importantly , artificial balancing of Q is implicitly or explicitly encoded in several transductive methods , which use the query set statistics to make class predictions , as will be discussed in section 4 . 4φ is either fixed , e.g. , [ 23 ] , or fine-tuned during inference , e.g. , [ 15 ] . There is , however , evidence in the literature that freezing φ yields better performances [ 23 , 32 , 34 , 33 ] , while reducing the inference time .

Summary: The researchers use transductive inference and introduce the effect of arbitrary class distributions within the query sets of few-shot tasks, rather than using class-balanced tasks, by removing the class-balance artefact. They do this by modeling the marginal probabilities of the classes as Dirichlet-distributed random variables, rather than from a known and fixed uniform distribution. They assess their model by comparing transductive methods over 3 datasets, and show their experiment setting’s performance drops compared to inductive methods. They also propose a generalization of the mutual-information loss based on alpha-divergences, which is an extension of the Shannon mutual information and tolerates class-distribution variations more effectively.