Learning Flexible Classifiers with Shot-CONditional Episodic (SCONE) Training

1 INTRODUCTION . Few-shot classification is the problem of learning a classifier using only a few examples . Specifically , the aim is to utilize a training dataset towards obtaining a flexible model that has the ability to ‘ quickly ’ learn about new classes from few examples . Success is evaluated on a number of test episodes , each posing a classification task between previously-unseen test classes . In each such episode , we are given a few examples , or “ shots ” , of each new class that can be used to adapt this model to the task at hand , and the objective is to correctly classify a held-out set of examples of the new classes . A simple approach to this problem is to learn a classifier over the training classes , parameterized as a neural network feature extractor followed by a classification layer . While the classification layer is not useful at test time due to the class shift , the embedding weights that are learned during this “ pre-training ” phase evidently constitute a strong representation that can be used to tackle test tasks when paired with a simple “ inference algorithm ” ( e.g . nearest-neighbour , logistic regression ) to make predictions for each example in the test episode given the episode ’ s small training set . Alternatively , early influential works on few-shot classification ( Vinyals et al. , 2016 ) advocate for episodic training , a regime where the training objective is expressed in terms of performance on a number of training episodes of the same structure as the test episodes , but with the classes sampled from the training set . It was hypothesized that this episodic approach captures a more appropriate inductive bias for the problem of few-shot classification and would thus lead to better generalization . However , there is an ongoing debate about whether episodic training is in fact required for obtaining the best few-shot classification performance . Notably , recent work ( Chen et al. , 2019 ; Dhillon et al. , 2020 ) proposed strong “ pre-training ” baselines that leverage common best practices for supervised training ( e.g . normalization schemes , data augmentation ) to obtain a powerful representation that works well for this task . Interestingly , other recent work combines the pre-training of a single classifier with episodic fine-tuning by removing the classification head and continuing to train the embedding network using the episodic inference algorithm that will be applied at test time ( Triantafillou et al. , 2020 ; Chen et al. , 2020 ) . The success of this hybrid approach suggests that perhaps the two regimes have complementary strengths , but the role of this episodic fine-tuning is poorly understood : what is the nature of the modification it induces into the pre-trained solution ? Under which conditions is it required in order to achieve the best performance ? As a step towards answering those questions , we investigate the effect of the shot used during episodic fine-tuning on the resulting model ’ s performance on test tasks of a range of shots . We are particularly interested in understanding whether the shot of the training episodes constitutes a source of information that the model can leverage to improve its few-shot classification performance on episodes of that shot at test time . Our analysis reveals that indeed a particular functionality that this fine-tuning phase may serve is to specialize a pre-trained model to solving tasks of a particular shot ; accomplished by performing the fine-tuning on episodes of that shot . However , perhaps unsurprisingly , we find that specializing to a given shot comes at the expense of hurting performance for other shots , in agreement with ( Cao et al. , 2020 ) ’ s theoretical finding in the context of Prototypical Networks ( Snell et al. , 2017 ) where inferior performance was reported when the shot at training time did not match the shot at test time . Given those trade-offs , how can our newfound understanding of episodic fine-tuning as shotspecialization help us in practice ? It is unrealistic to assume that we will always have the same number of labeled examples for every new class we hope to learn at test time , so we are interested in approaches that operate well on tasks of a range of shots . However , it is impractical to fine-tune a separate episodic model for every shot , and intuitively that seems wasteful as we expect that tasks of similar shots should require similar models . Motivated by this , we propose to train a single shot-conditional model for specializing the pre-trained solution to a wide spectrum of shots without suffering trade-offs . This leads to a compact but flexible model that can be conditioned to be made appropriate for the shot appearing in each test episode . In what follows we provide some background on few-shot classification and episodic models and then introduce our proposed shot-conditioning approach and related work . We then present our experimental analysis on the effect of the shot chosen for episodic fine-tuning , and we observe that our shot-conditional training approach is beneficial for obtaining a general flexible model that does not suffer the trade-offs inherent in naively specializing to any particular shot . Finally , we experiment with our proposed shot-conditional approach in the large-scale Meta-Dataset benchmark for few-shot classification , and demonstrate its effectiveness in that challenging environment . 2 BACKGROUND . Problem definition Few-shot classification aims to classify test examples of unseen classes from a small labeled training set . The standard evaluation procedure involves sampling classification episodes by picking N classes at random from a test set of classes Ctest and sampling two disjoint sets of examples from the N chosen classes : a support set ( or training set ) of k labeled examples per class , and a query set ( or test set ) of unlabeled examples , forming N -way , k-shot episodes . The model is allowed to use the support set , in addition to knowledge acquired while training on a disjoint set of classes Ctrain , to make a prediction for examples in the query set , and is evaluated on its query set accuracy averaged over multiple test episodes . Episodic training Early few-shot classification approaches ( Vinyals et al. , 2016 ) operate under the assumption that obtaining a model capable of few-shot classification requires training it on ( mini-batches of ) learning episodes , instead of ( mini-batches of ) individual examples as in standard supervised learning . These learning episodes are sampled in the same way as described above for test episodes , but with classes sampled from Ctrain this time . In other words , the model is trained to minimize a loss of the form : ES , Q∼PN , ktrain 1 |Q| ∑ ( x∗ , y∗ ) ∈Q − log pθ ( y∗ | x∗ , S ) ( 1 ) where S and Q are support and query sets sampled from the distribution PN , ktrain of N -way , k-shot training episodes induced by Ctrain , and θ represents the model ’ s parameters . This training regime is often characterized as meta-learning or learning to learn , i.e . learning over many episodes how to learn within an episode ( from few labeled examples ) . Episodic models differ by their “ inference algorithm ” , i.e . the manner in which pθ ( y∗ | x∗ , S ) is computed to classify query examples based on the support set . Prototypical Networks Prototypical Networks ( Snell et al. , 2017 ) is a simple but effective episodic model which constructs a prototype φc for each class c in an episode as φc = 1 |Sc| ∑ x∈Sc fθ ( x ) , ( 2 ) where f is an embedding function parametrized by θ and Sc represents the set of support examples belonging to class c , and classifies a given query example as p ( y∗ = c | x∗ , S ) = exp ( −||x ∗ − φc||22 ) ∑ c′ exp ( −||x∗ − φc′ ||22 ) . ( 3 ) 3 SHOT CONDITIONAL EPISODIC ( SCONE ) TRAINING . In this section we introduce Shot CONditional Episodic ( SCONE ) training for the purpose of specializing a strong pre-trained model to solving few-shot classification tasks of a range of different shots , without suffering disproportionately for any shot . Training objective Training episodically involves minimizing the objective shown in Equation 1 . We first sample an episode from P k , Ntrain and compute a prediction pθ ( y ∗ | x∗ , S ) for each query example x∗ . We then compute the cross-entropy loss on the query set using those predictions and perform a parameter update by backpropagating its gradient with respect to θ into the inference algorithm . In this work we concern ourselves with models that use an embedding function fθ to obtain a representation for the support and query examples of each episode on top of which the inference algorithm is applied . In Prototypical Networks , for instance , fθ contains all of the model ’ s learnable parameters . SCONE trains on episodes of varying shots and conditions the model on each episode ’ s shot distribution . ( Figure 1 ) by minimizing Ek∼Pk ES , Q∼PN , ktrain 1 |Q| ∑ ( x∗ , y∗ ) ∈Q − log pθk ( y∗ | x∗ , S ) , ( 4 ) where Pk is the distribution over shots at training time and θk depends on an episode ’ s sampled shots . In the Appendix , we include an algorithm box outlining SCONE fine-tuning . Conditioning mechanism Rather than learning a separate set of model parameters for each shot setting , we modulate a subset of its parameters using FiLM ( Perez et al. , 2018 ) , a simple conditioning mechanism which performs an affine feature-wise transformation of its input x based on conditioning information k ( in our case , the episode ’ s number of shots ) : FiLM ( x ) = γ ( k ) x+ β ( k ) . ( 5 ) The dependency of γ and β on k is handled by maintaining distinct values for each shot setting and selecting the appropriate γ and β based on an episode ’ s shot . Equivalently , we can think of our approach as a compact representation of many shot-specific feature extractors which share all but their FiLM layer parameters . More concretely , we maintain a set of FiLM parameters for each shot in the [ 1 , MAX-SHOT ] range ( where MAX-SHOT is a hyperparameter ) and let all shots settings greater than or equal to MAXSHOT share the same FiLM parametrization . As is often the case in practice , instead of inserting FiLM layers in the network ’ s architecture , we modulate the scaling and shifting parameter values of existing batch normalization layers ( Dumoulin et al. , 2017 ; De Vries et al. , 2017 ) . When performing episodic fine-tuning , we initialize all sets of FiLM parameters to those learned during pre-training ( i.e . the learned batch normalization scaling and shifting coefficients ) . These different sets of FiLM parameters are then free to deviate from each other as a result of fine-tuning . We found it beneficial to penalize the L2-norm of β ( regularizing the offset towards 0 ) and the L2 norm of γ − 1 ( regularizing the scaling towards 1 ) . For this purpose , we introduce a hyperparameter that controls the strength of this FiLM weight decay . Handling class-imbalanced episodes SCONE can also be used on imbalanced episodes , where different classes have different shots . In that case , instead of selecting a single set of FiLM parameters , we compute the FiLM parameters for an episode as the convex combination of the FiLM parameters associated with all shots found in the episode , where the weights of that combination are determined based on the frequency with which each shot appears in the episode . Concretely , the episode ’ s “ shot distribution ” s ( a vector of length MAX-SHOT ) is obtained by averaging the one-hot representations of the shots of the classes appearing in an episode . In the special case of a class-balanced episode , the resulting average will be exactly a one-hot vector . This shot distribution is then used for the purpose of selecting the episode ’ s FiLM parameters . This can be thought of as an embedding lookup sTF in a matrix F of FiLM parameters using a shot distribution s. Smoothing the shot distribution We expect similar shot values to require similar FiLM parameters , which we incorporate as an inductive bias by smoothing the shot distribution . We outline our SMOOTHSHOT procedure in the Appendix in Algorithm 1 , which receives the shot s of a class ( an integer ) , and a smoothing hyperparameter m ( a float in [ 0 , 1 ] ) and returns the smoothed shot for that class , which is a vector of length MAX-SHOT . Essentially , the result of smoothing is that the returned vector representation of s is not strictly one-hot with only the position corresponding to the observed shot s being ‘ on ’ . Instead , some entries surrounding that position are also non-zero . Specifically , the entries that are directly adjacent to s receive the value m , the entries two spots away from s the value m2 , and so on , with entries further away from s receiving exponentially-decaying values .

Summary: This paper proposed an implementation method of using different numbers of shots of data for few-shot learning such as to mitigate the negative effect of "different shots". It optimized the FiLM parameters using meta gradient descent during episodic meta-training with different-shot learning tasks. It conducted the experiments on quite a set of "meta-dataset benchmarks".