Towards Understanding the Cause of Error in Few-Shot Learning

1 INTRODUCTION . Learning novel concepts from few samples is one of the most important ability in human cognition system ( Chen et al . ( 2018 ) ; Dhillon et al . ( 2019 ) ; Wang et al . ( 2020 ) ) . By contrast , massive achievements of modern artificial intelligent systems are dependent upon lots of data and annotation which are hard to acquire in many scenarios . Blocked by the difficulty in obtaining large labeled datasets , community shows more interests in developing algorithms with high data-efficiency . It is so-called few-shot learning that learns to generalize well to new categories with scarce labeled samples ( Sung et al . ( 2018 ) ; Vinyals et al . ( 2016 ) ) . Existing methods deal with few-shot learning in the general framework of meta-learning where a base learner is developed and optimized across different episodes ( or tasks ) . Episodes are formed in a N-way K-shot fashion where K support samples per class are available for training . The overall objective is enabling the base learner to exploit on base classes and to transfer learnt knowledge to recognize novel classes with few support data . Since training and evaluation are performed on different tasks , the base learner holds different task-specific classifiers that depend on data sampling . In general , classification model has two components : feature extractor and classifier ( Simonyan & Zisserman ( 2015 ) ; He et al . ( 2016 ) ; Zagoruyko & Komodakis ( 2016 ) ) . Most approaches of few-shot learning exploit from according perspectives : learning a good embedding and finding a right base learner . Rethinking-FSC ( Tian et al . ( 2020 ) ) demonstrates that a good learned embedding space can be more effective than many sophisticated meta-learning algorithms . It argues for the performance on meta set where embeddings are learnt in supervised or self-supervised way . Goldblum et al . ( 2020 ) reveal the importance of feature clustering in few-shot learning . Since classifier performance is sample-dependent especially in one-shot scenario , variance of feature is expected to be small so as to retain good performance . It shows that classifier performance is not stable across different tasks . MetaOptNet ( Lee et al . ( 2019 ) ) and R2-D2 ( Bertinetto et al . ( 2018 ) ) explore training and optimization routines for linear classifier , enabling good few-shot performance through simple base learner . These literatures develop specific algorithms from the aspects of learning good representation or optimizing base learner . Most recent methods use linear classifier as base learner , so we also consider linear model in this paper . To our best knowledge , there has been little research focusing on how the two components ( aka feature representation and classifier ) respectively influence the performance on novel classes in FSL . In this paper , we introduce an upper bound of error rate in few-shot learning , indicating that the error comes from two aspects : 1 ) linear separability in the embedding space and 2 ) classifier discrepancy between task-specific and task-independent classifiers . The ideal classifier is viewed to be task-independent since its performance is not sample-dependent ( Goldblum et al . ( 2020 ) ) . To quantitively estimate each term , an experiment is performed where we use error rate of supervised classification tasks on novel classes to measure feature separability , and use disagreement of results obtained from different classifiers to measure discrepancy . It comes to an interesting observation that features learned through simple methods are sufficiently discriminative and the error mainly comes from classifier discrepancy . Based on our observation and theoretical analysis , we propose a simple method of reducing classifier discrepancy so as to boost few-shot performance . Experiments on three benchmarks are conducted to empirically verify our theory . Results on different datasets with various base learners show consistent improvements , supporting our finding and theory in few-shot learning . The main contributions of this paper are : 1 . The upper bound of error rate on novel classes is theoretically analyzed . From derived equations we figure out that the error in FSL is caused by linear separability in the feature space and discrepancy between task-specific and task-independent classifiers . 2 . Quantitative experiments are conducted to verify the theoretical analysis . Results show that the error is dominantly caused by classifier discrepancy . 3 . Based on the theoretical analysis and the experiment results , a constraint is proposed to reduce classifier discrepancy so as to decrease the upper bound of error rate in FSL . 4 . Further experiments on mini-ImageNet , tiered-ImageNet and CIFAR-FS confirm the effectiveness of the proposed method . It shows that decreasing classifier discrepancy can consistently achieve improvements in most cases . 2 RELATED WORK . Algorithms of Few-Shot Learning Prototypical Network ( Snell et al . ( 2017 ) ) is a classical algorithm for its simplicity and effectiveness , which performs few-shot classification by nearestprototype matching . Since class prototype is the mean of features , the linear separability in feature space has direct impact on classification results . Performance of following series of prototype based methods ( Allen et al . ( 2019 ) ; Liu et al . ( 2020 ) ) is also limited by feature separability . Different with these methods using nearest-neighbor classifier , Bertinetto et al . ( 2018 ) adopt ridge regression and logistic regression as base learner . Similarly , Lee et al . ( 2019 ) use classical linear classifier SVM in few-shot learning to learn representations . Simple linear classifier shows competitive performance and in this paper , we use linear classifier in measuring linear separability and classifier discrepancy . Theoretical Analysis of Few-Shot Learning Cao et al . ( 2019 ) introduce a bound for accuracy of Prototypical Network ( Snell et al . ( 2017 ) ) , demonstrating that the intrinsic dimension of the embedding function ’ s output space varies with the number of shots . They further propose a method to overcome the negative impact of mismatched shots in meta-train and meta-test stages . As in ( Liu et al . ( 2020 ) ) , they give a lower bound for accuracy cosine similarity based prototypical network . Two key factors : intra-class bias and cross-class bias are theoretically formulated . We also analyze theoretical bounds in few-shot learning . Theory in this paper does not focus on specific algorithm like Prototypical Network but on general scenarios , from the perspective of feature separability and classifier discrepancy . Theoretical Analysis of Domain Adaptation : Methods of domain adaptation ( Ben-David et al . ( 2007 ; 2010 ) ; Ganin & Lempitsky ( 2015 ) ) solve the problem of how to train a classifier on source domain and guarantee the classifier performs well on target domain . A classifier ’ s target error is bound by its source error and the divergence between the two domains in ( Ben-David et al . ( 2010 ) ) . They utilize H-divergence and H∆H-divergence to measure discrepancy between two domains . H∆H-divergence can be computed from finite unlabeled data , allowing us to directly estimate the error of a source-trained classifier on the target domain . Inspired by their work , we also use H∆Hdivergence to measure discrepancy between sets on novel classes and base classes . 3 BACKGROUND . 3.1 PROBLEM SETUP . The common setup of few-shot learning used in this paper is described below . A space of class is divided into two parts : base classes Cbase and novel classes Cnovel where Cbase ∩ Cnovel = ∅ . Dataset Dbase of base classes is used for model training and the model is evaluated on dataset Dnovel whose samples belong to unseen classes during training . The model is composed of a feature extractor F and a classifier h. In few-shot learning , we usually consider N -way K-shot Q-query tasks T . In task τi = ( Dsi , D q i , h ) , the support set D s i includes K data x ∈ Rd per class and its true label y ∈ { c1 , ... , cN } . The goal is to predict labels for query data in Dqi given Dsi . In this paper , we use error rate on novel classes to evaluate the few-shot performance of a trained model . The error rate is formulated as : novel = E [ τ ] = 1 M ×Q M∑ i Q∑ j 1 ( h ( F ( xi , j ) ) ! = yi , j ) ( 1 ) where M is the number of sampled tasks τi ∼ Tnovel . 1 ( · ) is indicator function . 3.2 DISTRIBUTION DIVERGENCE . We adopt following concepts to explore the cause of error in few-shot scenarios . Definition 1 Given a set D = { ( x1 , y1 ) , ... , ( xm , ym ) } where xi ∈ X and yi ∈ Y , for any mappings h1 , h2 ∈ X , disagreement is defined in Eqn . 2 to measure the difference of these two mappings . dis ( h1 , h2 ) = Px∼DX ( h1 ( x ) 6= h2 ( x ) ) ( 2 ) Definition 2 Given a domain X withD1 andD2 probability distributions over X , letH be a hypothesis class on X and denote by I ( h ) the set for which h ∈ H is the characteristic function ; that is , x ∈ I ( h ) ⇔ h ( x ) = 1 . H divergence between D1 and D2 is dH ( D1 , D2 ) = 2suph∈H|PrD1 [ I ( h ) ] − PrD2 [ I ( h ) ] | ( 3 ) Definition 3 For hypotheses h , h′ ∈ H , the symmetric difference hypothesis space H∆H is the set of hypotheses g ∈ H∆H ⇔ g ( x ) = h ( x ) ⊕h′ ( x ) where ⊕ is the XOR function . H∆H divergence over distributions is defined as following : dH∆H ( D1 , D2 ) = 2suph∈H|Prx∼D1 [ h ( x ) 6= h′ ( x ) ] − Prx∼D2 [ h ( x ) 6= h′ ( x ) ] | ( 4 ) 4 PROPOSED METHOD . 4.1 MEASURING CLASSIFICATION PERFORMANCE ON NOVEL CLASSES . A classification model generally consists of two parts : feature extractor and classifier . Hence , the model holds two expectations : 1 ) Extracted features are expected to be discriminative for classification ( Snell et al . ( 2017 ) ; Lee et al . ( 2019 ) ) . 2 ) Classifier is supposed to be stable concerning different tasks ( Cao et al . ( 2019 ) ; Liu et al . ( 2020 ) ) . Consider linear classifier in this paper , we investigate from linear separability and classifier stability to measure classification performance on novel classes . To quantitively estimate these two terms , we design experiments using a base model with ResNet-12 backbone and FC ( fully-connected ) layer classifier . The base model is first trained on base classes in supervised way . When testing on novel classes , we replace FC layer classifier with ProtoNet ( Snell et al . ( 2017 ) ) and ridge regression ( RR ) as in paper Ye et al . ( 2020 ) . The novel ( h ∗ ) ( i.e . error rate on Cnovel with classifier h∗ ) , is used to approximate and quantify feature separability on novel classes ( Eqn . 5 ) . novel ( h ∗ ) = 1 N ×Q∗ N×Q∗∑ i 1 ( ŷi 6= yi ) ( 5 ) In Eqn . 5 , Q∗ is the number of all samples of each class c ∈ { c1 , ... , cN } . ŷi is predicted label and yi is true label . h∗ is trained under supervised way from large set of samples and is used to approximate the expected ideal N -way classifier . On the other hand , we use disagreement defined in Eqn . 6 to measure classifier discrepancy . dis ( h , h∗ ) = 1 N ×Q N×Q∑ i 1 ( ŷi 6= ŷ∗i ) ( 6 ) where Q is the number of query samples . h is task-specific classifier that differentiates among tasks , decided by sampled support data . h∗ is task-independent concerning these N classes . Thus , dis ( h , h∗ ) indicates the discrepancy between the task-specific classifiers and the ideal classifier . Table 1 shows results on two benchmarks : mini-ImageNet and tiered-ImageNet . From Table 1 , we can see that novel ( h∗ ) is generally lower than dis ( h , h∗ ) in a large margin . For example , 1-shot dis ( h , h∗ ) on mini-ImageNet with RR is up to 38.34 % while novel ( h∗ ) is 6.72 % , which is five times lower . Furthermore , the obvious drop of dis ( h , h∗ ) from 1-shot to 5-shot indicates obvious raising of classifier discrepancy . An interesting conclusion can be drawn from this experiment that the error on novel classes is dominantly caused by classifier discrepancy in low-data regimes rather than linear separability . More details about this experiment are presented in the appendix .

Summary: This paper seeks to understand theoretically the current bottleneck in few-shot learning and address it with a new way of training the embedding. The authors find that the key issue in few-shot learning is not the separability of the novel classes but the discrepancy between classifiers trained on large datasets and few-shot classifiers. They then propose a way to reducee this discrepancy.