A Closer Look at Prototype Classifier for Few-shot Image Classification

1 INTRODUCTION . Few-shot learning is used to adapt quickly to new classes with low annotation cost . Meta-learning is a standard training procedure to tackle the few-shot learning problem and Prototypical Network ( Snell et al. , 2017 ) a.k.a ProtoNet is a widely used meta-learning algorithm for few-shot learning . In ProtoNet , we use a prototype classifier based on meta-learning to predict the classes of unobserved objects by constructing class-specific prototypes without adjusting the hyper-parameters during meta-testing . ProtoNet has the following advantages . ( 1 ) Since the nearest neighbor method is applied on query data and class prototypes during the meta-test phase , no hyper-parameters are required in meta-test phase . ( 2 ) Since the number of data for few-shot is small , the inference time is almost negligible . ( 3 ) The classifiers can quickly adapt to new environments because they do not have to be re-trained for the support set when new classes appear . The generalization bound of ProtoNet in relation to the number of shots in a support set has been studied ( Cao et al. , 2020 ) . The bound suggests that the performance of ProtoNet depends on the ratio of the between-class variance to the within-class variance of features of a support set extracted using the meta-trained model . There have been studies on training a new linear classifier on the features extracted using a pretrained model without meta-learning , which can perform comparably with the meta-learned models ( Chen et al. , 2019 ; Tian et al. , 2020 ) . We call this approach as linear-evaluation-based approach . In these studies , the models are trained with the standard classification problem , i.e. , models are trained with cross-entropy loss after linear projection from the embedding space to the class-probability space . The linear-evaluation-based approach has the following advantages over meta-learning . ( 1 ) Training converges faster than meta-learning . ( 2 ) Implementation is simpler . ( 3 ) Meta-learning decreases in performance if the number of shots does not match between meta-training and metatesting ( Cao et al. , 2020 ) ; however , the linear-evaluation-based approach do not need to take this into account . However , the linear-evaluation-based approach requires retraining a linear classifier every time a new class appears . In contrast , a prototype classifier can be applied to any trained feature extractor and does not require model learning in the testing phase . Therefore , a prototype classifier can be a a practical and useful first step for few-shot learning problems . In order to avoid meta-learning during the training phase and the linear evaluation during the testing phase , we focus on using a prototype classifier on the testing phase and training models in a standard classification manner . As we discuss in section 4 , we found that when we directly constructed prototypes from the feature vectors extracted using pretrained models and applied the nearest neighbor method as in the testing phase in ProtoNet , this does not perform as well as the linear-evaluation-based approach . We hypothesize that the reason is the difference between the loss function in ProtoNet and pretrained models . As described in section 3 , if we consider a prototype as a pseudo sample average of the features in each class , the loss function of ProtoNet can be considered having a regularizing effect that makes it closer to the sample average of the class . Since standard classification training computes cross-entropy loss with additional linear projection to make the features linearly separable , the loss function does not have such an effect and can cause large within-class variance . Figure 1 shows a scatter plot of the features extracted using a neural network with two dimension output trained on cifar-10 with ProtoNet ( 1a ) and cross-entropy loss with a linear projection layer ( 1b ) . This figure implies that the features extracted using a model trained in a standard classification manner distributes away from the origin and causes large within-class variance along the direction of the norm of class mean vectors , while that of ProtoNet is more centered to its class means . This phenomenon is also observed in face recognition literatures ( Wen et al. , 2016 ; Liu et al. , 2017 ; Wang et al. , 2018 ; Deng et al. , 2019 ) . We now focus on the theoretical analysis on a prototype classifier . A current study ( Cao et al. , 2020 ) analyzed an upper bound of risk by using a prototype classifier . The bound depends on the number of shots of a support set , between-class variance , and within-class variance . However , the bound requires the class-conditioned distribution of features to be Gaussian and to have the same covariance matrix among classes . In addition , since the bound does not depend on the norm of the feature vectors , it is not clear from the bound what feature-transformation method can lead to performance improvement . Thus , we derive a novel bound for a prototype classifier . Our contributions are threefold . 1 . We relax the assumption ; specifically , the bound does not require that the features distribute on any specific distribution , and each covariance matrix does not have to be the same among classes . 2 . We clarify the effect of the variance of the norm of the feature vectors on the performance of a prototype classifier . 3 . We investigate the effectiveness of reducing the variance of the norm empirically . 2 RELATED WORK . We summarize related work by describing a prototype classifier with meta-learning , linearevaluation-based approach without meta-learning , and theoretical analysis related to the few-shot learning problem . A prototype classifier with meta-learning On the basis of the hypothesis that features well distinguished in the training phase are also useful for classifying new classes , constructing one or multiple prototypes for classifying unseen examples is a widely used approach ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Pahde et al. , 2021 ; Ji et al. , 2021 ; Sung et al. , 2018 ; Allen et al. , 2019 ; Doersch et al. , 2020 ; Qi et al. , 2018 ) . Certain algorithms compute similarities between multiple prototypes and unseen examples by using their own modules , such as attention mechanism ( Vinyals et al. , 2016 ; Doersch et al. , 2020 ) , relation network ( Sung et al. , 2018 ) , reweighting mechanisms by taking into account between-class and within-class interaction ( Ji et al. , 2021 ) , and latent clusters ( Allen et al. , 2019 ) . Prototypes are also constructed in a multi-modal way ( Pahde et al. , 2021 ) . Another line of research is transforming the space of extracted features to a better distinguishable space ( Simon et al. , 2020 ; Yoon et al. , 2019 ; Das et al. , 2020 ; Das & Lee , 2020 ) or taking variance of fetures into account ( Bateni et al. , 2020 ) . Because of its convenience , the approach with a prototype classifier is adapted to other domain such as semantic segmentation ( Dong & Xing , 2018 ) , text classification ( Sun et al. , 2019 ) , and speech recognition ( Wang et al. , 2019a ) . Update-based meta-learning In contrast to the approach with a prototype classifier and metalearning , in update-based meta-learning approaches , model parameters are adjusted in the test phase so that a model can adapt to new classes . Model-agnostic meta-Learning ( MAML ) and its variants ( Finn et al. , 2017 ; 2018 ; Rajeswaran et al. , 2017 ) search for a good initialization parameters that adapt to new classes with a few labeled data and a few update steps of the parameters . Another approach involves learning an effective update rule for the parameters of a base-learner model through a sequence of training episodes ( Bertinetto et al. , 2019 ; Lee et al. , 2019 ) . Both approaches require additional learning of hyper-parameters and training time ; thus , they prevent quick adaptation to new classes . linear-evaluation-based approach without meta-learning Interestingly , recent studies have shown that training a new linear classifier with features extracted using a model trained with crossentropy loss on base-dataset performs comparably with meta-learning based methods ( Chen et al. , 2019 ) ( Wang et al. , 2019b ) . More effective method for training a new classifier in few-shot settings has been proposed ( Yang et al. , 2021 ; Phoo & Hariharan , 2021 ) , such as calibrating distribution generated by a support set ( Yang et al. , 2021 ) , self-supervised learning on query data ( Phoo & Hariharan , 2021 ) , and distilling knowledge to obtain better embeddings ( Tian et al. , 2020 ) . Liu et al . ( 2020 ) focuses on pretraining phase and found the negative margin in cross-entropy helps improving the performance in few-shot settings . However , similar to update-based meta-learning , these studies require additional hyper-parameters and training to apply to new classes ; thus if we tackle these points , we would have an alternative method that is easier and more convenient to use . Theoretical analysis of few-shot learning Even though much improvement has been empirically made in few-shot learning , theoretical analysis is scarce . In the context of meta-learning , Du et al . ( 2021 ) provided a risk bound on the meta-testing phase that is related to the number of meta-training data and meta-testing data . Lucas et al . ( 2021 ) derived information-theoretic lower-bounds on minimax rates of convergence for algorithms that are trained on data from multiple sources and tested on novel data . Cao et al . ( 2020 ) derived a new bound on a prototype classifier and theoretically demonstrated that the mismatch regarding the number of shots in a support set between meta-training and meta-testing degrades the performance of prototypical networks , which has been only experimentally observed . However , their bound depends on several assumptions : the class-conditional distributions of features are Gaussian and have the same covariance matrix among classes . In contrast , we derive a novel bound that does not depend on any specific distribution . 3 THEORETICAL ANALYSIS OF PROTOTYPE CLASSIFIER IN TERMS OF VARIANCE OF NORM OF FEATURE VECTORS . In this section , we first formulate our problem settings and point out the drawbacks of the current theoretical analysis on a prototype classifier . Next we provide our novel bound of a prototype classifier with the bound related to the variance of the norm of the feature vectors . Finally , we list several methods that can improve the performance of a prototype classifier based on our bound . 3.1 PROBLEM SETTING . Let Y be a space of a class , τ a probability distribution over Y , X a space of input data , D a probability distribution over X , Dy a probability distribution over X given a class y . We define D⊗nk and D⊗ny by D⊗ny = Πni=1Dy and D⊗nk ≡ Πni=1D ⊗k i , respectively . We sample N classes from τ to form the N -way classification problem . Denote by K a number of annotated data in each class and x ∈ X , y ∈ Y as input data and its class respectively . We define a set of support data of class c sampled from τ as Sc = { xi | ( xi , yi ) ∈ X × Y ∩ yi = c } Ki=1 and a set of support data in the N -way K-shot classification problem as S = ⋃N c=1 Sc . Suppose a feature extractor computes a function ϕ : X → RD , where D is the number of the embedding dimensions . ϕ ( Sc ) is defined by ϕ ( Sc ) = 1K ∑ x∈Sc ϕ ( x ) . Let Φ be a space of the extractor function ϕ. Denote by M : Φ × X × ( X × Y ) NK → RN a prototype classifier function that computes probabilities of input x belonging to class c as follows . M ( ϕ , x , S ) c = pM ( y = c|x , S , ϕ ) = exp ( −∥ϕ ( x ) − ϕ ( Sc ) ∥2 ) ∑N l=1 exp ( −∥ϕ ( x ) − ϕ ( Sl ) ∥2 ) , ( 1 ) where ∥v∥2 = ∑D d=1 ( v ( d ) ) 2 , and v ( d ) is the d-th dimension of vector v. The prediction of an input x , denoted by ŷ ∈ Y , is computed by taking argmax for M ( ϕ , x , S ) , i.e. , ŷ = argmaxM ( ϕ , x , S ) . We denote by Ez∼q ( z ) [ g ( z ) ] an operation to take the expectation of g ( z ) over z distributed as f ( z ) and we simply denote Ez∼q ( z ) [ g ( z ) ] as Ez [ g ( z ) ] when z is obviously distributed on q ( z ) . We define Varz∼q ( z ) [ g ( z ) ] as an operation to take variance of g ( z ) over z distributed as q ( z ) . With I denoting the indicator function , we define expected risk RM of a prototype classifier as RM ( ϕ ) = ES∼D⊗nkEc∼τEx∼Dc [ I [ argmaxM ( ϕ , x , S ) ̸= c ] . ( 2 ) For simplicity , we now discuss the binary classification setting . We show a case of multi-class classification in Appendix A.5 due to lack of space . Let c1 and c2 denote any pair of classes sampled from τ . We consider that a query data x belongs to class c1 and support sets S consist of class c1 ’ s support set and c2 ’ s support set . Then , equation 2 is written as follows . RM ( ϕ ) = ES∼D⊗2kEc1∼τEx∼Dc1 [ I [ argmaxM ( ϕ , x , Sc1 ∪ Sc2 ) ̸= c1 ] ] . ( 3 )

Summary: This paper analyzes how a prototype classifier works well without fine-tuning and meta-learning. The feature is trained with cross-entropy loss with a linear projection layer. In testing, the classifier is a Prototype classifier.  This paper derives a novel generalization bound for the prototypical network and shows that focusing on the variance of the norm of a feature vector can improve performance. The proposed upper bound is a modification of (Cao et al., 2020), and it does not require the features distributed on any specific distribution. Each covariance matrix does not have to be the same among classes.   The authors experimentally investigated several normalization methods (L2-N, V-N, LDA, EST, EST-L2-N) for minimizing the variance of the norm.