A Theoretical Analysis of the Number of Shots in Few-Shot Learning

1 INTRODUCTION . Human cognition has the impressive ability of grasping new concepts from exposure to a handful of examples ( Yger et al. , 2015 ) . In comparison , while modern deep learning methods achieve unprecedented performances with very deep neural-networks ( He et al. , 2016 ; Szegedy et al. , 2015 ) , they require extensive amounts of data to train , often ranging in the millions . Few-shot learning aims to bridge the sample-efficiency gap between deep learning and human learning in fields such as computer vision , reinforcement learning and speech recognition ( Santoro et al. , 2016 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ; Vinyals et al. , 2016 ; Wang et al. , 2019a ) . These methods fall under the framework of meta-learning , in which a meta-learner extracts knowledge from many related tasks ( in the meta-training phase ) and leverages that knowledge to quickly learn new tasks ( in the meta-testing phase ) . In this paper , we focus on the few-shot classification problem where each task is defined as a N -way classification problem with k samples ( shots ) per class available for training . Many meta-learning methods use the episodic training setup in which the meta-learner iterates through episodes in the meta-training phase . In each episode , a task is drawn from some population and a limited amount of support and query data from that task is made available . The meta-learner then learns a task-specific classifier on the support data and the classifier predicts on the query data . Updates to the meta-learner is computed based on the performance of the classifier on the query set . Evaluation of the meta-learner ( during a phase called meta-testing ) is also carried out in episodes in a similar fashion , except that the meta-learner is no longer updated and the performance on query data across multiple episodes is aggregated . In the episodic setup , the selection of k during meta-training time can have significant effects on the learning outcomes of the meta-learner . Intuitively , if support data is expected to be scarce , the meta-learner needs to provide strong inductive bias to the task-specific learner as the danger of overfitting is high . In contrast , if support data is expected to be abundant , then the meta-learner can provide generally more relaxed biases to the task-specific learner to achieve better fitting to the task data . Therefore it is plausible that a meta-learner trained with one k value can be suboptimal at adapting to tasks with a different k value and thus exhibit meta-overfitting to k. In experiments , k is often simply kept fixed between meta-training and meta-testing , but in real-world usage , one can not expect to know beforehand the amount of support data from unseen tasks during deployment . In this paper we will focus on Prototypical networks ( Snell et al. , 2017 ) , a.k.a . ProtoNet . ProtoNet is of practical interest because of its flexibility : a single trained instance of ProtoNet can be used on new tasks with any k and N . However , ProtoNet exhibits performance degradation when the k used in training does not match the k used in testing.1 First , we will undertake a theoretical investigation to elicit the connection from k to a lower bound of expected performance , as well as to the intrinsic dimension of the learned embedding space . Then , we conduct experiments to empirically verify our theoretical results across various settings . Guided by our new understanding of the effects of k , we propose an elegant method to tackle performance degradation in mismatched k cases . Our contributions are threefold : • We provide performance bounds for ProtoNets given an embedding function . From which , we argue that k affects learning and performance by scaling the contribution of intra-class variance . • Through VC-learnability theory , we connect the value of k used in meta-training to the intrinsic dimension of the embedding space . • The most important contribution of this paper ( introduced in Section 3.3 ) is a new method that improves upon vanilla ProtoNets by eliminating the performance degradation in cases where the k is mismatched between meta-training and meta-testing . Our evaluation protocol more closely adheres to real-world scenarios where the model is exposed to different numbers of training samples . 2 BACKGROUND . 2.1 PROBLEM SETUP . The few-shot classification problem considered in this paper is set up as described below . Consider a space of classes C with a probability distribution τ , N classes c = { c1 , ... , cN } are sampled i.i.d . from τ to form a N -way classification problem . For each class ci , k support data are sampled from class-conditional distribution Si = { sx1 , .. , sxk } iid∼ P ( x|Y ( x ) = ci ) , where x ∈ RD , D denotes the dimension of data , and Y ( x ) denotes the class assignment of x . Note that we assume that Y ( x ) is singular ( e.g . each x can only have 1 label ) , and does not depend on N ( e.g . a data point with a label “ cat ” will always have the label “ cat ” ) , in contrast to y defined below . Additionally , the set Q = { qx1 , ... , qxl } containing l query data is sampled from the joint distribution 1 N ∑N i=1 P ( x|ci ) .2 For each x , let y ∈ { 1 , ... , N } denote its label in the context of the few-shot classification task . Define Ŝi = { ( sx1 , y1 = i ) , ... , ( sxk , yk = i ) } as the augmented set of supports for class ci , and denote the union of support sets for N classes as S = ⋃N i=1 Ŝi . The few-shot classification task is to predict y for each x in Q given S. During meta-training , the ground truth label for Q is also available to the learner . 2.2 META-LEARNING SETUP . Meta-learning approaches train on a distribution of tasks to obtain information that generalizes to unseen tasks . For few-shot classification , a task is determined by which classes are involved in the N -way classification task . During meta-training , the meta-learner observes episodes of few-shot classification tasks consisting of N classes , k labelled samples per class , and l unlabelled samples , as previously described . The collection of all classes observed during meta-training forms the metatraining split Dtr = { trc1 , ... , trcR } . Critically , we assume that every unseen class that the learner is evaluated upon ( during meta-testing ) is also drawn from the same distribution τ . 1For example , 1-shot accuracy of a 10-shot trained network performs 9 % ( absolute ) worse than the 1-shot trained network 2We assume equal likelihood of drawing from each class for simplicity 2.3 PROTOTYPICAL NETWORKS . ProtoNets ( Snell et al. , 2017 ) compute E-dimensional embeddings for all samples in S and Q . The embedding function φ : RD → RE is usually a deep network.The prototype representation for each class is formed by averaging the embeddings for all supports of said class : φ ( Si ) = 1k ∑ x∈Si φ ( x ) . Classification of any input x ( e.g . x ∈ Q ) is performed by computing the softmax over squared Euclidean distances of the input point ’ s embedding to the prototypes . Let ŷ denote the prediction of the classifier for one of the categories j ∈ { 1 , · · · , N } : pφ ( ŷ = j|x , S ) = e−‖φ ( x ) −φ ( Sj ) ‖ 2 ∑N i=1 e − ∥∥∥φ ( x ) −φ ( Si ) 2∥∥∥ , where ‖v‖2 = E∑ d=1 v2d ( 1 ) The parameters of the embedding functions are learned through meta-training . Negative log-likelihood J ( φ ) = − log ( p ( ŷ = y|x ) ) of the correct class y is minimized on the query points through SGD . As explained in ( Law et al. , 2019 ) , ProtoNets can be seen as a metric learning approach optimized for the supervised hard clustering task ( Law et al. , 2016 ) . The model φ is learned so that the representations of similar examples ( i.e . belonging to a same category ) are all grouped into the same cluster in RE . We propose in this paper a subsequent metric learning step which learns a linear transformation that maximizes inter-to-intra class variance ratio . 3 PROPOSED METHOD . We first present theoretical results explaining the effect of the shot number on ProtoNets , and then introduce our method for addressing performance degradation in cases of mismatched shots . 3.1 RELATING k TO LOWER BOUND OF EXPECTED ACCURACY To better understand the role of k on the performance of ProtoNets , we study how it contributes to the expected accuracy across episodes when using any kind of fixed embedding function ( e.g . the embedding function obtained at the end of the meta-training phase ) . With I denoting the indicator function , we define the expected accuracy R as : R ( φ ) = EcES , x , yI [ arg max j { pφ ( ŷ = j|x , S ) } = y ] ( 2 ) Definitions : Throughout this section , we will use the following symbols to denote the means and variances of embeddings under different expectations : µc , Ex [ φ ( x ) | Y ( x ) = c ] Σc , Ex [ ( φ ( x ) − µc ) ( φ ( x ) − µc ) T | Y ( x ) = c ] µ , Ec [ µc ] Σ , Ec [ ( µc − µ ) ( µc − µ ) T ] Remark . µc is the expectation of the embedding conditioned on class c. µ is the ( full ) expectation of the embedding , which can be expressed as the expectation of µc over classes . Σ is the variance of class means in the embedding space - it can be interpreted as the signal of the input to the classifier , as larger Σ implies larger distances between classes . Σc is the expected intra-class variance - it represents the noise in the above signal . Modelling assumptions of ProtoNets : The use of the squared Euclidean distance and softmax activation in ProtoNets implies that classification with ProtoNets is equivalent to a mixture density estimation on the support set with spherical Gaussian densities ( Snell et al. , 2017 ) . Specifically , we adopt the modelling assumptions that the distribution of φ ( x ) given any class assignment is normally distributed ( p ( φ ( x ) |Y ( x ) = c ) = N ( µc , Σc ) ) , with equal covariance for all classes in the embedding space ( ∀ ( c , c′ ) , Σc = Σc′ ) 3 . We present the analysis for the special case of episodes with binary classification ( i.e . with N = 2 ) for ease of presentation , but the conclusion can be generalized to arbitrary N > 2 ( see appendix ) . 3The second assumption is more general than the one used in the original paper as we do not require Σ to be diagonal Also , as noted in Section 2.1 , we assume equal likelihood between the classes . We would like to emphasize that the assignment of labels can be permuted freely and the classifier ’ s prediction would not be affected due to symmetry . Hence , we only need to consider one case for the ground truth label without loss of generality . Let a and b denote any pair of classes sampled from τ . Let x be drawn from a , and overload a and b to also indicate the ground truth label in the context of that episode , then equation 2 can be written as : R ( φ ) = Ea , b∼τEx , SI [ ŷ = a ] ( 3 ) Additionally , noting that p ( ŷ = a ) can be expressed as a sigmoid function σ : p ( ŷ = a|x ) = e −‖φ ( x ) −φ ( Sa ) ‖2 e−‖φ ( x ) −φ ( Sb ) ‖ 2 + e−‖φ ( x ) −φ ( Sa ) ‖ 2 = σ ( ∥∥∥φ ( x ) − φ ( Sb ) ∥∥∥2 − ∥∥∥φ ( x ) − φ ( Sa ) ∥∥∥2 ) = σ ( α ) , where α , ∥∥∥φ ( x ) − φ ( Sb ) ∥∥∥2 − ∥∥∥φ ( x ) − φ ( Sa ) ∥∥∥2 We can express equation 3 as a probability : R ( φ ) = Pr a , b , x , S ( ŷ = a ) = Pr a , b , x , S ( α > 0 ) ( 4 ) We will introduce a few auxiliary results before stating the main result for this section . Proposition 1 . From the one-sided Chebyshev ’ s inequality , it immediately follows that : R ( φ ) = Pr ( α > 0 ) ≥ E [ α ] 2 Var ( α ) + E [ α ] 2 ( 5 ) In Lemma 1 and Lemma 2 , we derive the expectation and variance of α when conditioned on the classes sampled in a episode . Then , in Theorem 3 , we compose them into the RHS of Proposition 1 through law of total expectation . Lemma 1 . Consider space of classes C with sampling distribution τ , a , b iid∼ τ . Let S = { Sa , Sb } Sa = { ax1 , ... , axk } , Sb = { bx1 , ... , bxk } , k ∈ N is the shot number , and Y ( x ) = a . Define φ ( Sa ) , 1k ∑ x∈Sa φ ( x ) and φ ( Sb ) , 1 k ∑ x∈Sb φ ( x ) . Consider Σ as defined earlier . Assume p ( φ ( x ) |Y ( x ) = c ) = N ( µc , Σc ) and Σc = Σc′ for any choice of c , c′ ∈ C , then , Ex , S|a , b [ α ] = ( µa − µb ) T ( µa − µb ) , and Ea , b , x , S [ α ] = 2Tr ( Σ ) . Lemma 2 . Under the same notation and assumptions as Lemma 1 , additionally invoking definition for Σc , then , Ea , b [ Var ( α|a , b ) ] ≤ 8 ( 1 + 1 k ) Tr ( Σc ( ( 1 + 1 k ) Σc + 2Σ ) ) . ( 6 ) The proofs of the above lemmas are in the appendix . With the results above , we are ready to state our main theoretical result in this section . Theorem 3 . Under the conditions where Lemma 1 and 2 hold , we have : R ( φ ) ≥ 4Tr ( Σ ) 2 8 ( 1 + 1/k ) 2Tr ( Σ2c ) + 16 ( 1 + 1/k ) Tr ( ΣΣc ) + Ea , b [ ( ( µa − µb ) T ( µa − µb ) ) 2 ] . ( 7 ) Proof . First , we use decompose the Var ( α ) term in Proposition 1 by Law of Total Expectation . Var ( α ) = Ea , b , x , S [ α2 ] − Ea , b , x , S [ α ] 2 = Ea , bEx , S [ α2|a , b ] − Ea , b , x , S [ α ] 2 ( 8 ) = Ea , b [ Var ( α|a , b ) + Ex , S [ α|a , b ] 2 ] − Ea , b , x , S [ α ] 2 . ( 9 ) Hence , Proposition 1 can also be expressed as R ( φ ) ≥ E [ α ] 2 Ea , b [ Var ( α|a , b ) + Ex , S [ α|a , b ] 2 ] . ( 10 ) Finally , we arrive at Theorem 3 by plugging Lemma 1 and 2 into equation 10 . Several observations can be made from Theorem 3 : 1 . The shot number k only appears in the first two terms of the denominator , implying that the bound saturates quickly with increasing k. This is also in agreement with the empirical observation that meta-testing accuracy has diminishing improvements when more support data is added . 2 . By observing the degree of terms in equation 7 ( and treating the last term of the denominator as a constant ) , it is clear that increasing k will decrease the sensitivity ( magnitude of partial derivative ) of this lower bound to Σc , and increase its sensitivity to Σ . 3 . If one postulates that meta-learning updates on φ are similar to gradient ascent on this accuracy lower bound , then learning with smaller k emphasizes minimizing noise , while learning with higher k emphasizes maximizing signal . In conclusion , these observations give us a plausible reason for the performance degradation observed in mismatched shots : when an embedding function has been optimized ( trained ) for ktrain > ktest , the relatively high Σc is amplified by the now smaller k , resulting in degraded performance . Conversely , an embedding function trained for ktrain < ktest already has small Σc , such that increasing k during testing has further diminished improvement on performance .

Summary: The author performs theoretical analysis of the number-of-shot problem in the case study of prototypical network which is a typical method in few-shot learning. To facilitate analysis, the paper assumes 2-way classification (binary classification) and equal covariance for all classes in the embedding space, and finally derives the lower bound of the expected accuracy with respect to the shot number k. The final formula of the lower bounding indicates that increasing k will decrease the sensitivity of this lower bound to Σc (expected intra-class variance), and increase its sensitivity to Σ (variance of class means). To reduce the meta-overfitting (when training are test shot are the same, the performance becomes better), the author designed Embedding Space Transformation (EST) to minimize Σc and maximize Σ through a transformation that lies in the space of non-dominant eigenvectors of Σc while also being aligned to the dominant eigenvectors of Σ. The experimental results on 3 commonly used datasets for few-shot learning, i.e. Omniglot, Mini-ImageNet and Tiered-ImageNet demonstrate promising results and desired properties of the method.