Learning Flexible Classifiers with Shot-CONditional Episodic (SCONE) Training

1 INTRODUCTION . Few-shot classification is the problem of learning a classifier using only a few examples . Specifically , the aim is to utilize a training dataset towards obtaining a flexible model that has the ability to ‘ quickly ’ learn about new classes from few examples . Success is evaluated on a number of test episodes , each posing a classification task between previously-unseen test classes . In each such episode , we are given a few examples , or “ shots ” , of each new class that can be used to adapt this model to the task at hand , and the objective is to correctly classify a held-out set of examples of the new classes . A simple approach to this problem is to learn a classifier over the training classes , parameterized as a neural network feature extractor followed by a classification layer . While the classification layer is not useful at test time due to the class shift , the embedding weights that are learned during this “ pre-training ” phase evidently constitute a strong representation that can be used to tackle test tasks when paired with a simple “ inference algorithm ” ( e.g . nearest-neighbour , logistic regression ) to make predictions for each example in the test episode given the episode ’ s small training set . Alternatively , early influential works on few-shot classification ( Vinyals et al. , 2016 ) advocate for episodic training , a regime where the training objective is expressed in terms of performance on a number of training episodes of the same structure as the test episodes , but with the classes sampled from the training set . It was hypothesized that this episodic approach captures a more appropriate inductive bias for the problem of few-shot classification and would thus lead to better generalization . However , there is an ongoing debate about whether episodic training is in fact required for obtaining the best few-shot classification performance . Notably , recent work ( Chen et al. , 2019 ; Dhillon et al. , 2020 ) proposed strong “ pre-training ” baselines that leverage common best practices for supervised training ( e.g . normalization schemes , data augmentation ) to obtain a powerful representation that works well for this task . Interestingly , other recent work combines the pre-training of a single classifier with episodic fine-tuning by removing the classification head and continuing to train the embedding network using the episodic inference algorithm that will be applied at test time ( Triantafillou et al. , 2020 ; Chen et al. , 2020 ) . The success of this hybrid approach suggests that perhaps the two regimes have complementary strengths , but the role of this episodic fine-tuning is poorly understood : what is the nature of the modification it induces into the pre-trained solution ? Under which conditions is it required in order to achieve the best performance ? As a step towards answering those questions , we investigate the effect of the shot used during episodic fine-tuning on the resulting model ’ s performance on test tasks of a range of shots . We are particularly interested in understanding whether the shot of the training episodes constitutes a source of information that the model can leverage to improve its few-shot classification performance on episodes of that shot at test time . Our analysis reveals that indeed a particular functionality that this fine-tuning phase may serve is to specialize a pre-trained model to solving tasks of a particular shot ; accomplished by performing the fine-tuning on episodes of that shot . However , perhaps unsurprisingly , we find that specializing to a given shot comes at the expense of hurting performance for other shots , in agreement with ( Cao et al. , 2020 ) ’ s theoretical finding in the context of Prototypical Networks ( Snell et al. , 2017 ) where inferior performance was reported when the shot at training time did not match the shot at test time . Given those trade-offs , how can our newfound understanding of episodic fine-tuning as shotspecialization help us in practice ? It is unrealistic to assume that we will always have the same number of labeled examples for every new class we hope to learn at test time , so we are interested in approaches that operate well on tasks of a range of shots . However , it is impractical to fine-tune a separate episodic model for every shot , and intuitively that seems wasteful as we expect that tasks of similar shots should require similar models . Motivated by this , we propose to train a single shot-conditional model for specializing the pre-trained solution to a wide spectrum of shots without suffering trade-offs . This leads to a compact but flexible model that can be conditioned to be made appropriate for the shot appearing in each test episode . In what follows we provide some background on few-shot classification and episodic models and then introduce our proposed shot-conditioning approach and related work . We then present our experimental analysis on the effect of the shot chosen for episodic fine-tuning , and we observe that our shot-conditional training approach is beneficial for obtaining a general flexible model that does not suffer the trade-offs inherent in naively specializing to any particular shot . Finally , we experiment with our proposed shot-conditional approach in the large-scale Meta-Dataset benchmark for few-shot classification , and demonstrate its effectiveness in that challenging environment . 2 BACKGROUND . Problem definition Few-shot classification aims to classify test examples of unseen classes from a small labeled training set . The standard evaluation procedure involves sampling classification episodes by picking N classes at random from a test set of classes Ctest and sampling two disjoint sets of examples from the N chosen classes : a support set ( or training set ) of k labeled examples per class , and a query set ( or test set ) of unlabeled examples , forming N -way , k-shot episodes . The model is allowed to use the support set , in addition to knowledge acquired while training on a disjoint set of classes Ctrain , to make a prediction for examples in the query set , and is evaluated on its query set accuracy averaged over multiple test episodes . Episodic training Early few-shot classification approaches ( Vinyals et al. , 2016 ) operate under the assumption that obtaining a model capable of few-shot classification requires training it on ( mini-batches of ) learning episodes , instead of ( mini-batches of ) individual examples as in standard supervised learning . These learning episodes are sampled in the same way as described above for test episodes , but with classes sampled from Ctrain this time . In other words , the model is trained to minimize a loss of the form : ES , Q∼PN , ktrain 1 |Q| ∑ ( x∗ , y∗ ) ∈Q − log pθ ( y∗ | x∗ , S ) ( 1 ) where S and Q are support and query sets sampled from the distribution PN , ktrain of N -way , k-shot training episodes induced by Ctrain , and θ represents the model ’ s parameters . This training regime is often characterized as meta-learning or learning to learn , i.e . learning over many episodes how to learn within an episode ( from few labeled examples ) . Episodic models differ by their “ inference algorithm ” , i.e . the manner in which pθ ( y∗ | x∗ , S ) is computed to classify query examples based on the support set . Prototypical Networks Prototypical Networks ( Snell et al. , 2017 ) is a simple but effective episodic model which constructs a prototype φc for each class c in an episode as φc = 1 |Sc| ∑ x∈Sc fθ ( x ) , ( 2 ) where f is an embedding function parametrized by θ and Sc represents the set of support examples belonging to class c , and classifies a given query example as p ( y∗ = c | x∗ , S ) = exp ( −||x ∗ − φc||22 ) ∑ c′ exp ( −||x∗ − φc′ ||22 ) . ( 3 ) 3 SHOT CONDITIONAL EPISODIC ( SCONE ) TRAINING . In this section we introduce Shot CONditional Episodic ( SCONE ) training for the purpose of specializing a strong pre-trained model to solving few-shot classification tasks of a range of different shots , without suffering disproportionately for any shot . Training objective Training episodically involves minimizing the objective shown in Equation 1 . We first sample an episode from P k , Ntrain and compute a prediction pθ ( y ∗ | x∗ , S ) for each query example x∗ . We then compute the cross-entropy loss on the query set using those predictions and perform a parameter update by backpropagating its gradient with respect to θ into the inference algorithm . In this work we concern ourselves with models that use an embedding function fθ to obtain a representation for the support and query examples of each episode on top of which the inference algorithm is applied . In Prototypical Networks , for instance , fθ contains all of the model ’ s learnable parameters . SCONE trains on episodes of varying shots and conditions the model on each episode ’ s shot distribution . ( Figure 1 ) by minimizing Ek∼Pk ES , Q∼PN , ktrain 1 |Q| ∑ ( x∗ , y∗ ) ∈Q − log pθk ( y∗ | x∗ , S ) , ( 4 ) where Pk is the distribution over shots at training time and θk depends on an episode ’ s sampled shots . In the Appendix , we include an algorithm box outlining SCONE fine-tuning . Conditioning mechanism Rather than learning a separate set of model parameters for each shot setting , we modulate a subset of its parameters using FiLM ( Perez et al. , 2018 ) , a simple conditioning mechanism which performs an affine feature-wise transformation of its input x based on conditioning information k ( in our case , the episode ’ s number of shots ) : FiLM ( x ) = γ ( k ) x+ β ( k ) . ( 5 ) The dependency of γ and β on k is handled by maintaining distinct values for each shot setting and selecting the appropriate γ and β based on an episode ’ s shot . Equivalently , we can think of our approach as a compact representation of many shot-specific feature extractors which share all but their FiLM layer parameters . More concretely , we maintain a set of FiLM parameters for each shot in the [ 1 , MAX-SHOT ] range ( where MAX-SHOT is a hyperparameter ) and let all shots settings greater than or equal to MAXSHOT share the same FiLM parametrization . As is often the case in practice , instead of inserting FiLM layers in the network ’ s architecture , we modulate the scaling and shifting parameter values of existing batch normalization layers ( Dumoulin et al. , 2017 ; De Vries et al. , 2017 ) . When performing episodic fine-tuning , we initialize all sets of FiLM parameters to those learned during pre-training ( i.e . the learned batch normalization scaling and shifting coefficients ) . These different sets of FiLM parameters are then free to deviate from each other as a result of fine-tuning . We found it beneficial to penalize the L2-norm of β ( regularizing the offset towards 0 ) and the L2 norm of γ − 1 ( regularizing the scaling towards 1 ) . For this purpose , we introduce a hyperparameter that controls the strength of this FiLM weight decay . Handling class-imbalanced episodes SCONE can also be used on imbalanced episodes , where different classes have different shots . In that case , instead of selecting a single set of FiLM parameters , we compute the FiLM parameters for an episode as the convex combination of the FiLM parameters associated with all shots found in the episode , where the weights of that combination are determined based on the frequency with which each shot appears in the episode . Concretely , the episode ’ s “ shot distribution ” s ( a vector of length MAX-SHOT ) is obtained by averaging the one-hot representations of the shots of the classes appearing in an episode . In the special case of a class-balanced episode , the resulting average will be exactly a one-hot vector . This shot distribution is then used for the purpose of selecting the episode ’ s FiLM parameters . This can be thought of as an embedding lookup sTF in a matrix F of FiLM parameters using a shot distribution s. Smoothing the shot distribution We expect similar shot values to require similar FiLM parameters , which we incorporate as an inductive bias by smoothing the shot distribution . We outline our SMOOTHSHOT procedure in the Appendix in Algorithm 1 , which receives the shot s of a class ( an integer ) , and a smoothing hyperparameter m ( a float in [ 0 , 1 ] ) and returns the smoothed shot for that class , which is a vector of length MAX-SHOT . Essentially , the result of smoothing is that the returned vector representation of s is not strictly one-hot with only the position corresponding to the observed shot s being ‘ on ’ . Instead , some entries surrounding that position are also non-zero . Specifically , the entries that are directly adjacent to s receive the value m , the entries two spots away from s the value m2 , and so on , with entries further away from s receiving exponentially-decaying values .

Summary: The paper proposes a solution to few-shot meta learning approaches overfitting to the number of shots they are finetuned on, and not generalizing as well as expected to novel shots. In order to mitigate this problem, the paper suggests a parameterization of the meta learner which also conditions on the number of shots the model trains on. In practice, this is done via manipulation of the batch normalization parameters based on the number of shots. With this conditioning, the paper shows that the models perform better across a range of shots that they are evaluated on, compared to various sensible baselines.