Learning Flexible Classifiers with Shot-CONditional Episodic (SCONE) Training
Authors: Eleni Triantafillou
Vincent Dumoulin
Hugo Larochelle
Richard Zemel
Venue:  (2021)
Keywords: few-shot classification
few-shot learning
episodic training
meta-learning

Abstract: Early few-shot classification work advocates for episodic training, i.e. training over learning episodes each posing a few-shot classification task. However, the role of this training regime remains poorly understood, and its usefulness is still debated. Standard classification training methods (``pre-training'') followed by episodic fine-tuning have recently achieved strong results. This work aims to understand the role of this episodic fine-tuning phase through an exploration of the effect of the ``shot'' setting (number of examples per class) that is used during fine-tuning. We discover that fine-tuning on episodes of a particular shot can specialize the pre-trained model to solving episodes of that shot at the expense of performance on other shots, in agreement with a trade-off recently observed in the context of end-to-end episodic training. To amend this, we propose a shot-conditional form of episodic fine-tuning, inspired from recent work that trains a single model on a distribution of losses. Our investigation shows that this improves overall performance, without suffering disproportionately on any shot. We also examine the usefulness of this approach on the large-scale Meta-Dataset benchmark where test episodes exhibit varying shots and imbalanced classes. We find that our flexible model improves performance in that challenging environment.

Under review as a conference paper at ICLR 2021
LEARNING FLEXIBLE CLASSIFIERS WITH SHOT- CONDITIONAL EPISODIC (SCONE) TRAINING
Anonymous authors Paper under double-bling review
ABSTRACT
Early few-shot classification work advocates for episodic training, i.e. training over learning episodes each posing few-shot classification task. However, the role of this training regime remains poorly understood, and its usefulness is still debated. Standard classification training methods ("pre-training") followed by episodic fine- tuning have recently achieved strong results. This work aims to understand the role of this episodic fine-tuning phase through an exploration of the effect of the "shot" setting (number of examples per class) that is used during fine-tuning, We discover that fine-tuning on episodes of particular shot can specialize the pre-traines model to solving episodes of that shot at the expense of performance on other shots, in agreement with trade-off recently observed in the context of end-to-end episodic training. To amend this, we propose shot-conditions form of episodic fine-tuning. inspired from recent work that trains single model on a distribution of losses. Our investigation shows that this improves overall performance, without suffering disproportionately on any shot. We also examine the usefulness of this approach on the scale Meta- Dataset benchmark where test episodes exhibit varying shots and imbalanced classes. We find that our flexible model improves performance that challenging environment.
INTRODUCTION
Few-shot classification is the problem of learning classifier using only a few examples. Specifically, the aim is to utilize training dataset towards obtaining flexible model that has the ability to 'quickly' learn about new classes from few examples. Success evaluated on a number of test episodes, each posing a classification task between previously-unsee given a few examples, or "shots", of each new class that can be used to adapt this model to the task at hand, and the objective is to correctly classify held-out set of examples of the new classes.
test classes. In each such episode, we are
simple approach to this problem is to learn a classifier over the training classes, parameterized as neural network feature extractor followed by classification layer. While the classification layer is not useful at test time due to the class shift, the embedding weights that are learned during this "pre-training" phase evidently constitute a strong representation that can be used to tackle test tasks when paired with a simple "inference algorithm" (e.g. nearest- neighbtr, logistic regression) to make predictions for each example the test episode given the episode's 's small training set. Alternatively, early influential works on few-shot classification (Vinyals et al. 2016) advocate for episodic training, regime where the training objective is expressed in terms of performance on number of training episodes of the same structure as the test episodes, but with the classes sampled from the training set. was hypothesized that this episodic approach captures a more appropriate inductive bias for the problem of few-shot classification and would thus lead to better generalization.
However, there an ongoing debate about whether episodic training is in fact required for obtaining the best few-shot classification performance. Notably, recent work (Chen et al.) 2019 Dhillon et al. 2020) proposed strong "pre-training" baselines that leverage common best practices for supervised training (e.g. normalization schemes, data augmentation) to obtain powerful representation that works well for this task. Interestingly, other recent work combines the pre-training of a single classifier with episodic fine-tuning by removing the classification head and continuing to train the embedding network using the episodic inference algorithm that will be applied at test time (Triantafillou et al.) 2020; Chen et al. 2020). The success of this hybrid approach suggests that perhaps the two regimes

---

Under review as a conference paper at ICLR 2021
have complementary strengths, but the role of this episodic fine-tuning is poorly understood: what is the nature of the modification induces into the pre-trained solution? Under which conditions is required in order to achieve the best performance?
As a step towards answering those questions, we investigate the effect of the shot used during episodic fine-tuning on the resulting model's performance test tasks of range of shots. We are particularly interested in understanding whether the shot of the training episodes constitutes a source of information that the model can leverage to improve its few-shot classification performance on episodes of that shot at test time, Our analysis reveals that indeed particular functionality that this fine-tuning phase may serve is specialize a pre- pre-trained model solving tasks of particular shot; accomplished by performing the fine-tuning on episodes of that shot. However, perhaps unsurprisingly, we find that specializing to a given shot comes at the expense of hurting performance for other shots, in agreement with (Cao et al., 2020y's s theoretical finding in the context of Prototypical Networks (Snell et al., 2017) where inferior performance was reported when the shot at training time did not match the shot at test time.
Given those trade-offs, how can our newfound understanding episodic fine-tuning as shot- specialization help us in practice? It is unrealistic to assume that we will always have the same number of labeled examples for every new class we hope to learn at test time, so we are interested in approaches that operate well tasks of range of shots. However, it is impractical to fine-tune separate episodic model for every shot, and intuitively that seems wasteful as we expect that tasks of similar shots should require similar models. Motivated by this, we propose train single noot-conditional model for specializing the pre-trained solution to wide spectrum of shots without suffering trade-off's. This leads compact but flexible model that can be conditioned be made appropriate for the shot appearing in each test episode.
In what follows we provide some background on few-shot classification and episodic models and then introduce our proposed shot-come conooooodiiioingggggo approach and related work. We then present our experimental analysis on the effect of the shot chosen for episodic fine-tuning, and we observe that our shot-conditional training approach is beneficial for obtaining general flexible model that does not suffer the trade-offs inherent in naively specializing to any particular shot. Finally, we experiment with our proposed shot-conditional approach in the large-scale Meta-Dataset benchmark for few-shot classification, and demonstrate its effectiveness in that challenging environment.
BACKGROUND
Problem definition Few-shot classification aims to classify test examples of unseen classes from small labeled training set. The standard evaluation procedure involves sampling classification episodes by picking N classes at random from a test set of classes Crest and sampling two disjoint sets of examples from the N' chosen classes: a support set (or training set) of k labeled examples per class, and a query set (or test set) of unlabeled examples, forming N-way, k-shot episodes. The model is allowed to use the support set, in addition knowledge acquired while training disjoint set of classes Crrain, to make a prediction for examples in the query set, and is evaluated on its query accuracy averaged over multiple test episodes.
Episodic training Early few-shot classification approaches (Vinyals et al. 2016) operate under the assumption that obtaining a model capable of few-shot classification requires training it on (mini-batches of) learning episodes, instead of (mini-batched of) individual examples as in standard supervised learning. These learning episodes are sampled in the same way as described above for test episodes, but with classes sampled from Ctrain this time. In other words, the model is trained to minimize a loss of the form:
Eg Q- P.N.
Σ (=*
l Poly""""" x", ())
trucn
Q
where S and Q are support and query sets sampled from the distribution P.Nkk of N-way, k-shot training episodes induced by Ctraén and 0 represents the model's parameters. This training regime is often characterized as meta-learning or learning to learn, i.c. learning over many episodes how to learn within episode (from few labeled examples). Episodic models differ by their "inference
2

---

Under review as a conference paper at ICLR 2021
shots distribution
FiLM parameter selection
sub-network
FiLM layer
sub-nehwor
FILM layer
sub-network
x E S, Q
fo(x)
feature extractor
Figure 1: SCONE conditions the feature extractor fe on an episode's shot distribution.
algorithm", i.e. the manner in which pa(y") (8) is computed to classify query examples based on the support set.
Prototypical Networks Prototypical Networks (Snell et al. 2017) is a simple but effective episodic model which constructs prototype de for each class in an episode as
de
fa(x),
S
FES.
where f is an embedding function parametrized by 0 and S. represents the set of support examples belonging to class and classifies a given query example as
exp(- 6cl12)
ply" = S) Ee exp(-1|1rr --1|-" - de 12)
(3)
3 SHOT CONDITIONAL EPISODIC (SCONE) ) TRAINING
In this section introduce Shot CONditional Episodic (SCONE) ) training for the purpose of specializing a strong pre-trained model to solving few-shot classification tasks of a range different shots, without suffering disproportionately for any shot.
Training episodically involves minimizing the objective shown in Equation We first sample an episode from P&. and compute a prediction po(y" x²,S) for each query example x². We then compute the cross-entropy loss the query set using those predictions and perform a parameter update by backpropagating its gradient with respect to into the inference algorithm. In this work concern ourselves with models that an embedding function to obtain a representation for the support and query examples of each episode on top of which the inference algorithm is applied. In Prototypical Networks, for instance, fo contains all of the model's leamable parameters.
Training objective
SCONE trains episodes of varying shots and conditions the model on each episode's shot distribution. (Figure by minimizing
Ex~Ph P.N.
log pe, (w" xx", S)
(E"
where Ps is the distribution over shots at training time and 0₁ depends on an episode's 's sampled shots. In the Appendix, we include algorithm box outlining SCONE fine-tuning.
3

---

Under review as a conference paper at ICLR 2021
Conditioning mechanism Rather than learning separate set of model parameters for each shot
setting, we modulate a subset of its parameters using FiLM (Perez et al. (2018), a simple conditioning mechanism which performs an affine feature-wist transformation of its input x based on conditioning information k (in our case, the episode's 's number of shots):
FiLM(x) = (kk) O x + (()).
(5)
The dependency oy and 3 on k is handled by maintaining distinct values for each shot setting and selecting the appropriate 7 and 3 based on an episode's shot. Equivalently, we can think of our approach as a compact representation of many shot-specific feature extractors which share all but their FiLM layer parameters.
More concretely, we maintain a set of FiLM parameters for each shot in the [1. MAX-SHOT] range (where MAX-SHOT is hyperparameter) and let all shots settings greater than or equal to MAX- SHOT share the same FiL.M parametrization. As is often the case in practice, instead of inserting FiLM layers in the network's 's architecture, we modulate the scaling and shifting parameter values of existing batch normalization layers (Dumoulin et al. [017 De Vries et al. 2017). When performing episodic fine-tuning, we initialize all sets of FiLM parameters to those learned during pre-training (i.e. the learned batch normalization scaling and shifting coefficients). These different sets of FiLM parameters are then free to deviate from each other as a result of tine-tuning. We found it beneficial to penalize the L2-norm of 3 (regularizing the offset towards 0) and the L2 norm of (regularizing the scaling towards 1). For this purpose, we introduce a hyperparameter that controls the strength of this FiLM weight decay.
Handling class-imbalanced episodes SCONE can also be used on imbalanced episodes, where different classes have different shots. In that case, instead of selecting a single set of FiLM parameters, we compute the FiLM parameters for an episode as the convex combination of the FiLM parameters associated with all shots found in the episode, where the weights of that combination are determined based on the frequency with which each shot appears in the episode.
Concretely, the episode's 's "shot distribution" s (a vector of length MAX-SHOT) is obtained by averaging the one-hot representations of the shots of the classes appearing in an episode. In the special case of a bass-balaance episode, the resulting average will be exactly a one-hot vector. This shot distribution is then used for the purpose of selecting the episode's ' FiLM parameters. This can be thought of as an embedding lookup s F in a matrix F of FiLM parameters using a shot distribution s.
Smoothing the shot distribution We expect similar shot values to require similar FiLM parameters, which we incorporate as an inductive bias by smoothing the shot distribution. We outline our SMOOTH- SHOT procedure in the Appendix in Algorithm 1, which receives the shot s of a class (an integer), and a smoothing hyperparameter m (a float in [0, 1)) and returns the smoothed shot for that class, which is vector of length MAX-SHOT. Essentially, the result of smoothing is that the returned vector representation of is not strictly one-hot with only the position corresponding to the observed shot being 'on'. Instead, some entries surrounding that position are also non-zero. Specifically, the entries that are directly adjacent to receive the value m, the entries two spots away from the value m? and so on, with entries further away from 8 receiving appoentially-decayiin values.
4 RELATED WORK
Few-shot classification A plethora of models have been recently proposed for few-shot classifi- cation, and we refer the reader to (Hospedales al.) 2020) for a broad survey. Before episodic training was introduced, few-shot classifiers often relied on metric learning (Koch et al. 2015 Triantafillou et al. 2017). This theme persisted in early episodic models like Matching Networks (Vinyals al. 2016) and Prototypical Networks (Snell al. (017) where classification is made via necaest--ncighbov comparisons in the embedding space. Matching Networks apply a soft A-NN algorithm where the label of a query example is predicted to be the weighted average of the (one-hot) support labels with the weights determined by the similarity of that query to each support example.
Gradient- based episodic models are another popular family of approaches following the influential MAML paper (Finn et al. (017). To create a classifier for each given episode, this approach fine- tee-tunes the embedding weights along with linear classifier head using gradient descent on the support set.
4

---

Under review as a conference paper at ICLR 2021
Intuitively, this results in learning embedding space that serves as a useful starting point from which a few steps of gradient descent suffice to adapt the model to each episode's 's classification task. Proto-MAML (Triantafillou et al.] 2020) is simple extension that initializes the linear classifier for each episode from the prototypes of the classes appearing in that episode.
Recently, the field has shifted towards studying few-shot classification in more realistic environments like ImageNet (Ren et al.) 2018) and Meta-Datasse (Trantafillou et al.] 2020), which has encouraged research into newly-introduced challenges, such as accounting for multiple diverse datasets. Along these lines, Requeima et al. (2019); [Bateni et al. (2019) proposed novel task (aaauccces,SSikiieaaee
conditioning approaches, (Saiki al. (2020) introduced an improved hy hyperparameter tuning approach, and Dvornik et al. (2020) proposed a method for selecting an appropriate set of features for each test episode out of universal feature representation.
Understanding episodic learning Our work inscribes itself in a recent line of work attempting
to understand the differences between episodic and non-episodic learning. [Goldllum et al. (2020)
attempts to understand episodic learning from the perspective of how classes cluster feature-space
(for models that learn final classification layer on top of a feature extractor) well as from the perspective of local minima clusters (for gradient- based meta-learners). Huang et al. (2020); g Chao
et al. (2020) draw parallels between learning episodes and supervised learning examples, Bronskill et al. (2020) discusses batch normalization in episodic learning, drawing parallels from its use in non-episodic learning and Chen et al. (2020) contrasts episodic and non-episodic learning in their ability to generalize to new examples of previously seen classes or new examples of unseen classes. Finally, Cao al.] (2020) theoretically investigates the role of the shot in Prototypical Networks to explain the observed performance drop when there is a mismatch between the shots at training and test time. Instead, we empirically study the effect of the shot chosen during episodic fine-tuning of a pre-trained solution, in larger-scale and more diverse environment.
Feature-wise conditioning Feature-wise transformations such as FiLM (Perez et al. 201 8) are used as conditioning mechanism in variety of problem settings; see Dumoulin et al. (2018) for a survey on the topic. (Shu et al. (2019) devise loss re- weighting scheme that conditions on the loss
at each time-step, which is a scalar, thus bearing similarity to our approach when conditioning on a scalar shot setting. In few-shot classification, (Sun al. 2019)
feature- wise transformations
means of transfer to new tasks. (Oreshkin et al.) [218]] Requeima et al.. 2019 [ateni et al. (2019] use
FiLM to condition metric learners' backbones the support set, while (Dvornik et al. (2020) uses it as a way to represent many pre-trained classifiers using a shared parametrization. FiLM has also been used successfully for class-incremental learning Liu al. (2020) and semi-supervised few-shot learning (Li et al. 2019). Notably, TADAM (Oreshkin et al. 2018), CNAPs (Requeima et al.) 2019) and Simple-CNAP (Bateni et al. 2019) also use task conditioning, but they use the mean of the support set for this and thus the "shot' information is discarded. The purpose of our conditioning mechanism instead to make the backbone shot-aware. The idea of shot-conditional learners inspired by recent work that investigates loss-conditional training using feature-wise transformations (Dosovitskiy & Djolonga 2020 Babaeizadeh & Ghiasi 2020).
5 EXPERIMENTS
5.1 EXPLORING THE ROLE OF 'SHOTS' DURING EPISODIC FINE-TUNING
In this subsection, we examine the effect of the 'shot' that is used during the episodic fine-tuning phase, and in particular how it impacts the resulting model's 's ability to solve test episodes of different shots. We consider either using fixed shot l throughout the fine-tuning phase, or fine-tuning on episodes of distribution of shots. In the latter case, we explore both standard fine-tuning as well as SCONE fine-tuning that equips the model with the shot-conditioning mechanism described in the previous section. We also compare against EST (Cao et al. 2020).
Experimental setup We ran this round of experiments on ImageNet using the class splits proposed in Detta-Dataset... First, we pre-trained standard classifier on the set of training classes of ImageNet. We then removed the topmost classification layer, leaving us with a pre-trained backbone that used the initialization for the subsequent episodic fine-tuning round. We ran the following
5

---

Under review as a conference paper at ICLR 2021
1-shot accuracy
5-shot accuracy
40-shot accuracy
Accuracy
0.50
0.45
Accuracy
0. 88
Acouracy
0.82
0.80
Finekune shots
Fine tune on 1 shot
Fine-tuee on 60 60-shot
File-tune on 5-shot
SCONE Fine
shots
0:788
0.40
0.65
0.76
Figure 2: Test accuracy on three different evaluation shots. Fine-tuning exclusively on particular shot leads to the best test accuracy on that shot but poor aceuracy on different shots. Fine-tuning on a range of shots is reasonable general solution, but its performance can be improved when using SCONE, thanks to its conditioning mechanism that offers a compact form of shot specialization.
variants of episodic fine-tuning: exclusively on I-shot episodes ('Fine-tune on I-shot'), exclusively on 5-shot episodes ('Fine-tune on 5-shot"), on episodes whose shot is drawn uniformly from the range [1, 40] ('Fine-tune on all shots'), ), and on episodes with that same shot distribution but using SCONE ('SCONE Fine-tune on all shots'), which additionally equips the backbone with the shot conditioning mechanism described in the previous section. We also consider 'Fine-tune on best k-shot', an additional baseline that fine-tunes exclusively on the shot & that is found to work best on average on the validation set (on the range of shots 1 - 40). For this, we trained models for k 1, 5, 10, 15, 20, 30, 40 and found the best to be k- = 15.
As mentioned in the previous section, when applying SCONE training, we penalize the L2 norm of FiLM parameters. For a fair comparison with the other models, we applied the same regularization to the batch normalization parameters of all models during the episodic tine-tuning phase, and we found this to be generally helpful. We tuned the strength of this regularization separately for each model and picked the variant that worked best on the validation set, which we report in the Appendix. We set the SCONE 's MAX-SHOT hyperparameter to be 40 for this experiment.
We also compare to EST (Cao et al. 2020) which is a peporeticallyggroundde method for building shot resiliency in Gaussian classifiers. This involves applying a linear transformation on top of the learned embeddings that aims to strike a good balance between maximizing the inter-class variance and minimizing the intra-class variance. In practice, that trade-off is controlled via a hyperparameter p. We applied EST on top of the embeddings of Finne-tune on all shots' and we tuned p and the hyperparameter d controlling the projection dimensionality very extensively. The values that we found worked best (selected on the validation set of ImageNet on the range of shots 1-40) are substantially different than those used in the original EST paper: d = 480 and p = 5e 8 (versus the original d 120 and p lE- 3). We believe that this discrepancy may be due to deeper backbones and larger range of shots. The EST configuration that worked best for us yields a minimal reduction in the embedding dimensionality, and primarily favours maximizing the inter-class variance, with the term that minimizes the intra-class variance having minimal effect.
In all cases, we fix the 'way' to 5. We use Prototypical Networks the episodic model and perform early stopping and model selection on the validation set of classes, where the validation performance of a variant is computed on episodes of the same (distribution of) shot(s) that it is trained on. All models are tested on a held-out test set of classes that is not seen during pre- vee-training nor episodic fine-tuning, on 5-way episodes of different shot settings.
Findings We observe from Figure 2 that fine-tuning on a fixed shot yields the best results on test episodes of that shot. For example, 1-shot accuracies show that 'Fine-tune on 1-shot" surpasses the performance of all other variants on 1-shot test episodes, with the analogous findings in 1-shot and 5-shot accuracies for 5-shot and 40-shot, respectively. Therefore, particular functionality that the episodic fine-tuning phase may serve is to specialize the pre-trained model for performing well on tasks of particular shot. However, as illustrated in all sub-plots of Figure this shot specialization comes at the cost of severely reduced performance on tasks of very different shots. For instance, the model that is specialized for 40-shot tasks ('Fine-tune test tasks and vire-versa. We also note that the Fine-tune on best k-shot' model does not suffice to perform well in all settings either, since lo = 15 there, it performs really poorly 1-shot for instance.
40-shot') performs very poorly on 1-shot
In practice, it may be desirable to perform well on more than a single shot setting at test time, without having to fine-tune multiple separate shot-specialized models. A reasonable approach to that is

---

Under review as a conference paper at ICLR 2021
Prototypical Networks (ImageNet only)
Dataset
Standard
1.2 RN
EST
Best k-shot
SCONE w/o
SCONE
ILSVRC-2012
1: 1.12%
1: 1.06%
52.17
1.09%
1.8%
52.98%
52.51
1.11%
Omniglat
63.12 +: 1.37%
63.14 +: 1.32%
66.17
+ 1.29%
65.94
+ 1.33%
64.71%
+ 1.32%
65.60 = 1.34%
Aircraft
54,30 + 0.97%
53.26 + 0,97%
55.64 + 0.88%
56.03
0.95%
55.38%
0.96%
55.38
0.96%
Birds
68.22 + 0.97%
69.21 = 1.01%
67.17 + 1.02%
68.63
1.11%
68.98%
1.04%
69,70 = 1.01%
DTD
66.62 1: 0.90%
68.33 +: 0.81%
68.20 0.77%
69.61
0.80%
68.68%
1 0.78%
69,58 0.77%
Quickdraw
59.79 0.98%
59.17 0.96%
60.05
0.97%
60.68
0.97%
60.00%
1.00%
60.81
0.95%
Fungi
36.77 +: 1.07%
38,96 +: 1.10%
39,50 + 1.12%
37.96 1: 1.08%
39.19% + 1.15%
39,66 = 1.12%
VGG Flower
:::
87.70 1:
88.55 0.65%
0.86%
98%
88.03 0.73%
Traffic Signs
48.64 + 1.06%
46.54 + 1.03%
48.41 + 1.07%
50.26 1.16%
47.61%
1.05%
48, 48,24 + 1.09%
MSCOCO
43.02 1.09% %
43.11 + 1.05%
43,45
1.05%
43,20
1.18%
43.43%
1.08%
44.25
1.11%
Average
$7.80 %
58. 12 %
58, 92%
59.21%
58.79%
59. 38%
Avorago ranks
4,90
4.10
3.25
3.05
3.00
2.70
Table I: Prototypical Networks fine-tuned on ImageNet ( ('Standard") with the addition of L2 regular- ization on the batch normalization weights ('1.2 ). EST (Cao et al. 2020), the 'Fine-tune on best k-shot" baseline ('Best k-shot') and SCONE. including the ablation that omits the shot smoothing ('SCONE w/o S'). The reported numbers are query set accuracies averaged over 600 test episodes and 95% confidence intervals. We also show the average ranks (lower is better). We report details in the Appendix on rank computation and statistical testing.
episodically fine-tuning on a range of shots, to obtain a general model. Indeed, Figure 2 shows that 'Fine-tune on all shots' does not perform too poorly any shot but, perhaps unsurprisingly, in any given setting, it falls short of the performance of the corresponding shot-specialized model.
Finally, we observe that SCONE fine-tuning outperforms its shot-unaware counterpart in all settings ('SCONE Fine-tune on all shots' vs 'Fine-tune on all shots'). This constitutes evidence that SCONE fine-tuning indeed leads to a more flexible model that can adapt to the shot of each episode via its conditioning mechanism, without suffering the trade-offs inherent in naively specializing a model exclusively to any particular shot. We can view SCONE model as very compact way of representing multiple shot-specialized models, where the information required for that specialization resides in the light-weight FiLM parameters. SCONE also outperforms the EST approach in this setting which also strives for shot resiliency, but does so by encouraging invariance to the shot setting rather than shot awareness in SCONE.
5.2 LARGE-SCALE EXPERIMENTS ON
In what follows, we apply SCONE few-shot classification (Triantafilouu et al. 2020). Meta-Dataset is comprised of ten distinct image datasets, including natural images, handwritten characters and sketches. It also defines generative process for episodes that varies the way and shot across episodes, and within a particular episode varies the shot for different classes, introducing imbalance. The range of shots induced by this episode generator is also larger than what we considered in the previous section. It is long-tailed distribution under which small and medium shots are more likely but is possible to also encounter very large shots (e.g. >400), though this would happen very infrequently. We include histograms of the shot distributions of Meta-Dataset's s training, validation and test episodes in the Appendix. These experiments aim to investigate whether SCONE is effective on this broader shot distribution and imbalanced episodes.
the diverse and challenging Meta-Dataset benchmark for
Prototypical Network on ImageNet For our first of experiments on Meta-Datasel we explore different strategies of episodic fine-tuning of the pre-trained classifier's 's embedding weights using Prototypical Networks. For this, we use Meta-Dataset's 's sampling algorithm to draw training episodes of varying shots and ways from ImageNet. We compare standard episodic fine-tuning ('St ('Standard') to SCONE episodic fine-tuning ('SCONE '). Since SCONE uses L2-regularizztion on the sets of FiLM parameters, for a fair comparison we include a variant of standard episodic fine-tuning with 12-regularization on the batch normalization parameters (L22 BN'). We also include an ablation of our method that does not use any smoothing of the shot distribution. Finally, compare to EST as well where we computed the EST transformation on the 'L2 BN' instead of the 'Standard' Prototypical Network variant, since that worked best. We tuned EST's hyperparameters very extensively, as
7

---

Under review as a conference paper at ICLR 2021
Meta-Bascline (All darasets)
Dataset
Classifier Raseline
Control
SCONE
ILSVRC-200
53.44
1:
Omniglat
81.66 + 0.73%
89.28
0.51%
90.01
0.49%
Aircraft
70.65 0.62%
81.60 0.49%
78.27
0.54%
Birds
76.99 + 0.64%
78.75
0.59%
79.62
0.58%
DTD
71.28 1: 0.56%
70.47 1: 0.58%
71.89
0.39%
Quickdraw
0.67%
72.79 0.59%
71.95
0.56%
Fungi
50.23 + 0.81%
55.28 +: 0.73%
57.04
0.74%
Traffic Signs
89.14 + 0.44%
90.13 + 0.43%
91.09
0.39%
MSCOCO
53,92 0.78%
47.85 0.81%
52.94
0.82%
Average
68.03%
70.63%
71.68%.
Average
2.55
1.45
Table 2: Our reproduction of the Classifier-Baseling (Chen et al. 2020) trained on all datasets, and two variants that freeze those weights and episodically fine-tune using Meta-Baseline (Chen et al.) 2020) to optimize either only the batch norm parameters ("Control"), ) only SCONE 's parameters ("SCONE '). In all cases, the reported numbers are query set accuracies averaged over 1K test episodes and 95% confidence intervals. We also show the average ranks (lower is better).
described in Section 5.1, this time model-selecting the validation sets of all datasets of Meta- Dataset. The values that worked best this case are = 480 and = 5e As noted Section deeper
5.1, these are substantially different than those used in the original EST paper, likely due backbones and significantly broader range of shots explored. Finally, we ran the same 'Fine-tune on best k-shot" baseline described in Section 5.1. In this case we found that the best k was 20.
We evaluate these models on the held-out set of ImageNet classes as well as the remaining datasets. We SCONE 's MAX-SIIOT 200. We tune the learning rate and decay schedule separately for each variant and we perform model selection of SCONE 's hyperparameters using the validation set. All additional details are reported in the Appendix, and we plan to open source our code upon publication.
Bettl-Bseeline on all datasets Next, we experiment with the recent Meta- Baaaa-BBseieeee model (Chen al. 2020).. Meta-Bass DetaaBBaeeline also consists pre-training phase ('Classifier-Baase Baseline') followed by an episodic fine-tuning phase "MMet-Baseling classifier on the set of training classes. This variant is evaluated on few-shot episodes by discarding the ultimate classification layer and utilizing a cosine similarity- -based nearest-centroid inference algorithm on the learned embeddings, Meta-Baseling then fine-tunes ClassifierBasseline 's pre-trained embeddings the episodic objective of the aforementioned nearest-centroid algorithm.
ClassifierBBaselii refers to simply training
When training on all datasets of Meta-Dataset, they obtained strong results using their Classifier-Baselinee Baseline which is in this case trained a multi-task setup with separate output heads for the different datasets. They found that episodically fine-tuning that solution on all datasets did not help in general (it improved performance on some datasets but hurt performance larger number of datasets).
66
200
175
64
150
62
125
100
60
Inspired by that finding, we experimented with SCONE training phase on top of Classifier- strong pre-trained solution where froze the embedding weights to that powerful representation and we optimized only the set of SCONE 's FiLM parameters for shot conditioning. We performed this fine-tuning on training episodes from all datasets, using Meta- Baseline's nearest centroid method as the episodic model. As control experiment, performed the same episodic fine-tuning but without chot-conditioning, where we optimized only the batch normalization parameters, keeping the remainder of the embedding weights frozen ("Control'), This control can be thought of as a special case of SCONE where MAX-SHOT is
58
56
25
20
21
22
23
24
23
Figure 3: UMAP projection of the learned FiLM parameters for each "shot" setting, color-coded by shots.
8

---

Under review as a conference paper at ICLR 2021
0.02
0.01
150
Figure 4: The shot distribution produced according to our smoothing procedure for a hypothetical 4-way episode where the shots for the four classes are: 1, 10, 23, and 103.
Findings The results of this investigation are shown in Table and Table (as well as their more heavily-annotated counterparts in the Appendix, Tables 4 and 5 that show the per-row rank computa- tion). Following (Triantafillou et al. 2020), we run a test of statistical significance described in the Appendix to determine when to bold entry. Table T shows that SCONE fine-tuning outperforms standard episodic ane-tuning in the context of Prototypical Networks. Interestingly, penalizing the L2-norm of batch normalization parameters during episodic fine-tuning beneficial even when not using SCONE, but it does not reach the performance obtained by our shot-conditioning. The ablation of SCONE that does not use any smoothing of the shot distribution is also competitive, but performs worse than full SCONE. We also observe that EST is competitive in this setting, only slightly worse than SCONE, though we note that SCONE is more general approach that is not tied Gaussian classifiers. Similarly, in the context of Sata-Baseline, Table 2 shows that episodically fine-tuning the batch normalization parameters of the otherwise-frozee frozzen embedding is helpful ('Control'), but using SCONE to learn a separate set of FiLM parameters for each shot yields additional gains in this setting too. Overall, despite the simplicity of SCONE, these results demonstrate its effectiveness on different shot distributions, and in different backbones.
FiLM parameter visualization Finally, tion (Mclnnes et al., 2018) of the learned FiLM parameters for each shot setting (Figure 3). As expected, similar shot settings tend to learn similar sets of FiLM parameters, which is reflective of the fact that they rely on similar features for classification.
sanity check, we perform UMAP projec-
Example smoothed shot distribution To gain an intuition on the effect of our smoothing pro- cedure, illustrate in Figure 4 the result of smoothing example shot distribution using m = le 06, which is the value of the smoothing hyperparameter that we used for our Prototypical Network experiments on Meta-Dataset. For this, we consider hypothetical 4-way episode where the shots for the four classes are: 1, 10, 23, and 103. We observe that the largest peak the range of small values, due the first three shots of the episode, with the fourth shot causing second peak around the value 103. As reminder, this shot distribution defines the weights of the convex combination of FiLM parameters that will be used for episode. In practice therefore, we are activating 'blocks' of FiLM parameters that are relevant for each episode, instead of strictly activating only the FiLM parameters of the observed shots.
6 CONCLUSION
In summary, we present analysis aiming understand the role of episodic fine-tuning on top of pre-trained model for few-shot classification. We discover that this fine-tuning phase can be used to specialize the pre-trained model episodes of given shot, leading strong performance on test episodes of that shot at the expense of inferior performance on other shots. To eliminate that trade-off, we propose a shot-conditional episodic training approach that trains a model on episodes of a range of shots and can be conditioned at test time to modify its behavior appropriately depending on the shot of the given test episode. Our experimental analysis suggests that our proposed shot- conditioning mechanism is beneficial both in amaller-scale experiments, as well as in the large-scale Meta-Dataset Dataset benchmark, the context of two different episodic models. Future work could explore how to incorporate shot-awareness in other few-shot classification models. In addition to the architectural modification of FiLM conditioning on the shot distribution, are there algorithmic adjustments that can yield additional performance gains, such mechanism of determining the number of lnner-loop updates to perform for gradient- based meta-learners based on the number of available shots?
and diverse Meta-

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mohammad Babaeizadeh and Golnaz Ghiasi. Adjustable real-time style transfer. In Proceedings of the International Conference on Learning Representations. 2020.
Peyman Bateni, Raghav Goyal, Vaden Masrani. Frank Wood, and Leonid Sigal. Improved few-shot visual classification. arXiv preprint arXiv:1912.03433 2019.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493-14502, 2020.
John Bronskill, Jonathan Gordon, James Requeima, Sebastian Nowozin, and Richard E Turner. Tasknorm: Rethinking batch normalization for meta-learning. arXiv preprint arXiv:2003.03284 2020.
Tianshi Cao, Mare Law, and Sanja Fidler. A theoretical analysis of the number of shots in few-shot learning. Proceedings of the International Conference on Learning Representations, 2020.
Wei-Lun Chao, Han-Jia Ye, De-Chuan Zhan, Mark Campbell, and Kilian Q Weinberger. Revisiting meta-learning supervised learning. arXiv preprint arXiv:2002.00557 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In Proceedings of the International Conference on Learning Representations, 2019.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.04590 2020.
Harm De Vries, Florian Strub, Jérémic Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In Proceedings of the International Conference on Learning Representations, 2020.
Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In Proceedings of the International Conference on Learning Representations, 2020.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In Proceedings of the International Conference on Learning Representations, 2017.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018.
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Selecting relevant features from a universal representation for few-shot classification. arXiv preprint 0tti::200000933338 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- -olume 70, pp. 1126-1135. JMLR. org, 2017.
Micah Goldblum, Steven Reich, Liam Fowl, Renkun Ni, Valeriaa Cherepanova, and Tom Gold- stein. Unraveling meta-learning: Understanding feature representations for few-shot tasks. In Proceedings of the International Conference on Machine Learning, 2020.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, arXiv preprint arXiv:2004.054: 055399, 2020.
Mciyu Huang, Xueshuang Xiang, and Yao Xu. Training few-shot classification via the perspective of minibatch and pretraining. arXiv preprint arXiv:2004.05910, 2020.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
10

---

Under review as a conference paper at ICLR 2021
Xinzhe Li. Qianru Sun, Yaoyao Liu, Qin Zhou, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised shot classification. Advances in Neural Informa- fion Processing Systems, pp. 10286-10286, 2019.
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12245-12254, 2020.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint urXiv:1802.03420 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 18X3v 306.000776 2018.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Advances in Neural Information Processing Systems, pp. 7957-7968, 2019.
Tommoy Saikia, Thomas Brox, and Cordelia Schmid. Optimized generic feature learning for few-shot classification across domains. arXiv preprint arXiv:2001.009266, 2020.
Jun Shu, Qi Xic, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pp. 1919-1930, 2019.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. deca-trannsfer learning for few-shot learning. Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 403-412, 2019.
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. Advances in Neural Information Processing Systems, pp. 2255-2266 2017.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-Dataset: A dataset of datasets for learning learn from few examples. Proceedings of the International Conference on Learning Representations, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicaap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
A APPENDIX
SCONE 'S TRAINING ALGORITHM IN MORE DETAIL
For clarity, we provide pseudocode for SCONE 's training algorithm including our procedure for shot smoothing in Algorithm We will also release code upon publication for reproducibility.

---

Under review as a conference paper at ICLR 2021
Algorithm SCONE training
Input:
Distributions of training episodes Ptrain. pre-trained embedding weights 8, pre-trained batch norm weights y and B, embedding function f. learning rate e (a float), smoothing co-efficient m (a float in the range [0, 1J) and maximum supported shot MAX-SHOT (an int).
Output: Finetuned embedding weights O' and FiL.M parameters F = 8''
procedure SMOOTH-SHOT(s, MAX-SHOT) if s MAX-SHOT then
8 f MAX-SHOT
> Cap to the max supported shot
end if
D So that is in the range [0, MAX-SHOT - 1] D Init the smoothed shot
6 ONE-HOT DEPTHEMAX-SHOT)
0 و MAX-SHOT do
(---j-1
The index j slots to the left of b Outputs the zero vector if ( < 0 D The index j slots to the right of
{
r r -
1 ONE-HOT(T, DEPTHHMA-HHHT -NNEHHHTTTYDBBPPTH-AAXXSSS))
Outputs the zero vector if <
8 + -+++++ m²
Adjust the iteration's 's smoothing
end for end procedure
0' +- 0
Init the embedding weights from the pre-trained embeddings
for < k < MAX-SHOT do
b Init the FiLM params from the pre-trained batch norm params
(k) f 8k(k) +
end for while validation accuracy is improving do Sample training episode with support set S and query set Q Let kj -. KN be the shots of the episode's classes.
8 < ZEROS ZEROS (MAX-SHOT)
> Init the (unnormalized) shot distribution
for each class do
+ ONE-HOT DEPTH = MAX-SHOT)
8; SMOOTH (HOTTH, m, MAX-SHOT)
Smooth the one-hot shot of class
8 +
$ +
end for
SUM(s)
D Normalize to get the episode's shot distribution Select the FiLM params for the episode
7/ + 5 =" 3r + sT
Let St
{z(x; 04 2 3)), vawles
The embedded support set > The embedded query set Compute the episode's 's loss
Let Q = [f(1; 0', Ne (.), Wian)e
L. - 1
n Σ ((rr logp(y"" SH)
dL
0 00 aq
Update the model via gradient descent
7/ dL
On'
Oc
8r - 8
03'
end while
12

---

Under review as a conference paper at ICLR 2021
1 1-shot accuracy
5-shet sccuracy
10:shot accuracy
0.50
0.70
Accuracy
Accuracy
Accuracy
0.45
0.74
0.66
0.72
20-shot accuracy
accuracy
40-shot accuracy
0.600
Accuracy
Accuracy
50.50
Accurecy
Fine-taee 40-ahot
0.775
0.80
Finertune shiiss
SCONE Fine tuns on all shots
0.750
00.78
ES1 Fineed une on shots
0.78
Fine tune best leshot
0.76
Figure 5: Additional evaluation shot settings complement those in Figure the main paper. We refer the reader to Section 5.1 for detailed description of these plots.
1-4hot accuracy
12-10)-shot accuracy
(20-100)-shot accuracy
Accurary
0.22
Accuracy
0.39
Accuracy
0.54
#ST
Standard
020
0.38
0.53
0.37
Figure 6: Break down of the performance in Table in different evaluation shot ranges. We find that while the 'Best k-shot baseline performs well for large shots (the third subplot), it performs poorly on low shots (first two subplots).
Hypothesis testing We follow the same procedure as in (Triantafillou et al. 2020) to compute ranks for different methods that in turn determine which entries to bold in our tables. Specifically, we perform a 95% confidence interval statistical test on the difference between the mean accuracies of pairs of entries of each row. If for two entries we are not able reject the null hypothesis that the difference between their means is 0, they will receive the same rank. For example, if model A and model are tied for first place according to that test, they will each receive rank 1.5 (the average of the ranks and 2). If larger mean accuracy will receive a higher rank than the other. In each row, we bold the entries that tied for the highest rank. For convenience, in the last section of the Appendix, show more saavily-annotater copy each table the paper to make the rank computation procedure more transparent.
able to reject that hypothesis, however, the entry with the
ADDITIONAL RESULTS
First, provide in Figure 5 addditioaal plots in Figure in the main paper. The setup for this is exactly the same as for Figure 2
evaluation shot settings than those shown
Next, since we observe that the 'Best k-shot baseline performs well in Table [ which reflects average performance across episodes of varying shots, we further break down its performance different ranges of shots in Figure We find that while indeed 'Best k-shot" performs well for large shots, it actually performs poorly for low shots. This finding strengthens our case against this baseline: not only is it computationally expensive, requiring training multiple different models to pick the one that performs best, but is also is not as consistent SCONE in its performance different shot ranges.
Finally, to place our results into context, we display the results of Meta- Deta-Baselline with SCONE alongside the performance of recent work, controlling for model capacity. in T The approaches we compare against are: Passifier-Baseling (Chen et al. 2020). SUR-pf (Dvornik et al.) 2020), TaskNorm (Bronskill et al. 2020) and Simple CNAPs (Bateni al. 2020). In particular, we report the performance of the parametric family SUR ('SUR-pf pf instead of full SUR (which has 8x more parameters), in order to make aaples-to-apples comparisons with the remaining approaches. We find that the Meta- Baseline method, when combined with SCONE, achieves state-of ofte---f-hhatt on Meta-Dataset in this context, according to the average rank metric.
13

---

Under review as a conference paper at ICLR 2021
Dataset
Classifer-Baselinn:
Meta-Baseline SCONE
SUR-pf
TaskNorm
Simple CNAPs
II. 1LSV
53.44
0.82%
53.69 0.83%
56.40 :: 1.20%
50.60 +: 1.10%
58.60 1: 1.10%
Omniglot
81.66 1: 0.73%
90.01 1: 0.49%
88.50 1: 0.80%
90.70 1: 0.60%
91.70 1: 0.60%
Aircraft
0.62%
0.54%
79.50 0.80%
0.60%
82.40 1:
Birds
76,99 0.64%
79.62 0.58%
76.40 - 0.90%
74.60 + 0.80%
74.90 + 0.80%
DTD
71.28 0.56%
71.89 0,59%
73.10 0.70%
62.10 + 0.70%
67.80 + 0.80%
Quickdraw
64.09 0.67%
71.95 0.56%
75.70 0.70%
74.80 +: 0.70%
77.70 1: 0.70%
Fungii
50.23 0.81%
$7.04 0.74%
48.20 0.90%
48.70 1: .00%%
46.90 1: 1.00%%
VGG Flower
89.14 0.44%
91.09 0.39%
90.60 0.50%
89.60
0.60%
90.70 : 0.50%
Traffic Signs
89.14 = 0.44%
70.33 + 0.56%
65. 10 0.80%
67.00 + 0.70%
73.50 + 0.70%
MSCOCO
53.92
0.78%
0.82%
10
1.00%
46.20 :: 1.10%
Average
68.03%
71.68%
70.56%
68. 53%5
71.04%
Average
3,45
2.40
2.85
3.75
2.55
Table 3: Comparison of our best SCONE model to recent statc-of-the-art approaches on Meta-Dataset (for the setting of training on the training sets of all datasets).
EXPERIMENTAL DETAILS
We plan to open source our code upon publication, including all experimental details. In the meantime, we outline these details below for completeness.
Architecture We use ResNet-18 18 the feature extractor for all of our experiments, following the implementation in (Triantafillou et al.] 2020). For the SCONE variants, we add FiLM to all of the batch normalization layers throughout the network.
Image processing For all experiments, we use Meta-Datasetf input pipeline to obtain images, and we follow the image processing performed in (Chen al.) (2000) which yields images of size 128 x 128. We apply standard data augmentation consisting of horizontal flipping and random cropping followed by standardization using a commonly-use mean and standard deviation as in (Chen et al. 2020). For episodic models, data augmentation is applied in both the support and query sets. No data augmentation is used at validation nor test time.
Optimization We use ADAM with exponential learning rate decay and weight decay of le to optimize all models in this work. We tune the initial learning rate, the decay rate, and the number of updates between each learning rate decay separately for each model presented in the paper. The initial learning rate values we considered are 0.0005 and 0.001, with decay factor of 0.8 or 0.9 applied every 1000, 2000, 3000 steps. We ran variant for every combination of those values. We also tune the weight decay applied to the FiLM parameters (for SCONE variants) or the batch normalization parameters (for non- -CO-S variants). We tried the values: le 8, le 6, le - 4.
SCONE hyperparameters For the SCONE variants, aside from the above hyperparameters, we additionally tune the smoothing parameter m described in the main paper that is used for training and for evaluation. We did not tune the MAX-SHOT hyperparameter mentioned in the main paper as we found that our initial choices worked reasonably. Specifically, we set it to 40 for the smaller-scale experiments where the maximum shot was 40, and to 200 for the large-scale experiments. The latter choice was performed heuristically since shots much larger than 200 unlikely under the shot distribution induced by Setaa-Daassetss episode generator. For more information on that shot distribution, refer the reader to the next section.
SCONE smoothing hyperparameter We tuned the value of this hyperparameter that will be used both at training and evaluation. At training time, we considered values in the range 0, 0.2, 0.4, 0.6, 0.9 for Prototypical Network experiments, and we picked the variant that worked best according to the validation performance that was computed without smoothing. Once the model was trained and all of the remaining hyperparamters were tuned, we performed final validation round to tune the evaluation- time smoothing that will be used in the chosen model. We found it beneficial to larger values here, picking the value of 1 - 1e 06 for example for the Prototypical Network ImageNet. In the Meta- Meta-Baseline codebase, we trained with larger values of smoothing (the best we found was le 10) and didn't find it beneficial to additionally smooth at evaluation time.
14

---

Under review as a conference paper at ICLR 2021
Prototypical Networks (ImugeNet only)
Datasct
Standard
BN
EST
Best k-sbot
SCONE w/o
SCONE
ILSVRC-2012
$0.90 = (.12%)))
51.81 + (36%(()
52.17 + (.09%(3)
52.36 = (.8%((3)
52.8%% + 1.09%(3)
52.31 = 1.11%(3)
Omniglot
63.12 1.37%((.55
63.14 1.32%(5.55
66.07 (2.5)
65.94 + (.33%((2.5)
64.71% (2.5%
65.60 + 1.34%
Aircraft
54.30 +: 0,97%(5.5)
53.26 + 0.97%((5.5)
55.64 +: (2.5)
56.03 : (.95%(..5)
55.38% 1 0.96%
55.38 1: (..5%((.5)
Birds
0.97944.55
L01% (2.5)
(.05%
1.115((.55
1
(.15(
DTD
66.62 = (.90%(6)
68.33 = 0.81 (4.5))
68.20 0.77%
69.61 = 0.80%
68.88 + (.78%
69.58 1 0.77%
Quickdraw
59.79 = (3)
59.17 + 0.96%
60.05 (397%(3)
60.68 = (397%(()
60.00% (30
60.81 0.95%(3)
Fungi
36.77 1.07945.5)
38,96 + (1.0%
39.50 +: (2.5%
37.96 + 1.08945555)
39.19% + 1.15% (2.5)
39.66 + (2.5%
VGG Flower
86.61 = 0.87%((5
87.70 + (2)
88.55 = 0.68%)(2
87.45 = 0.86%(()
86.98% + 0.80%(5)
88.03 = 0.73%)(2
Traffic Signs
1065943.5)
+ 1.03%(6)
(.07%((355)
(.05%(3.5)
MSCOCO
43.02 1: (3.5)
43.11 1 1.6%% (3.5)
43.45 (3.5)
43.20 1 1.18%((355
43.43% 1 1.8%%
44.25 1 1.11 (3.5)
Average
$7.80 %
58.12 %
58.92%
59.21%
58.79%
59,38%
Average ranks
4,90
4.10
3.25
3.05
3,00
2.70
Table 4: The same table Table additionally annotated with per-row ranks.
Model selection For each experiment, perform early stopping according to the performance on the validation set. For the models that train on single shot k in the smaller-scale experiments. the validation performance that monitor for early stopping is the average query set accuracy k-shot 5-way episodes drawn from the validation set. For the models in the small-scale experiments that train distribution of shots, use the average validation performance over 5-way episodes whose shot sampled according to the same distribution used for training the respective model. For the larger-scale Meta-Dataset experiments, we draw validation episodes only from the validation set of ImageNet for the experiments that train on ImageNet only, or from the validation sets of all datasets for the experiments that train all datasets. In both cases, the validation episodes are drawn using Meta- Meta-Dataset's episode generator that yields episodes of variable ways and variable shots with class imbalance. all cases, the average validation performance is computed over 600 validation episodes and is monitored every 2K training updates. We apply exponential smoothing to resulting validation "curve" (using the default value of 0.6 in TensorBoard), Then, we choose update step at which the highest peak of that curve is found and we use the checkpoint corresponding to that update step for testing.
DISTRIBUTION OF SHOTS IN META-DATASET EPISODES
For reference, Figure displays histograms of the number of shots produced by Meta-Dataset's episode sampling algorithm. These are computed by sampling 600 episodes per dataset for each of the training, validation and test splits of Meta- Deta--Dt
TABLES WITH MORE DETAILED RANKS
In this section, we include copy of the same tables appearing previously in the paper, but additionally annotated with per-row ranks, to make the rank computation method more transparent. These tables are Table 4 Table 5 and T
15

---

Under review as a conference paper at ICLR 2021
8000
3500
6000
2500
2000
4000
1500
1000
500
0
50
100
150
200
250
50
100
150
200
250
(a) Training episodes from ImageNet
(b) Training episodes from all datasets
7000
6000
3000
5000
4000
2000
3000
1000
2000
1000
0
(c)
Validation episodes from ImageNet
Validation episodes from all datasets
1400
8000
6000
1000
800
4000
400
2000
200
50
100
150
200
250
50
100
150
200
250
(e) Test episodes from ImageNet
(f) Test episodes from all datasets
Figure 7: Histogram of shots appearing in episodes generated using Meta-Dataset's sampling algo- rithm for the different splits.
Datasct
Classifer-Baseline
Control
SCONE
SLS2CC-2012
53.44 +: 0.82% (1.5)
49.83 +: 0.80% (3)
53,69 +: 0.83% (1.5)
Omniglot
81.66 0.73% (3)
89.28 0.51%
90.01
0.49%
Aiteraft
70.65 + 0.62% (3)
81.60 1: 0.49% (1)
78.27 1: 0.54% (2)
(3)
DTD
71.28 +: 0.56% (1.5)
70.47 + 0.58% (3)
71.89 + 0.59 (1.5)
Quickdraw
64.09 1: 0.67%. (3)
72.79 0.59% (1.5)
71.95 0.56 (1.5)
Fungi
50.23 1 0.81% (3)
55.28 1 0.73%
$7.04
0.74%
VGG Flower
89.14 +: 0.44% (3)
90.13 + 0.43% (2)
91.09 = 0.39% (1)
Traffic Signs
70.37
70.33 0.56
MSCOCO
53.92 1: 0.78% (1.5)
47.85 + 0.81% (3)
52.94 1 0.82% (1.5)
Average
68.03%
70.63%
71.68%
Average rank
2.55
2
1.45
Table The same table as Table 2 additionally annotated with per-row ranks.
16

---

Under review as a conference paper at ICLR 2021
Dotaset
Clastfier-Baselinee
Meta-Baseline SCONE
SUR-pt
TaskNorm
Simple CNAPs
ILSVRC-2012
53.44 0,82443.5)
53.69 0.33%(35)
56.40 +: 1.209//))
50.60 1.109(5)
58.60 1.10%((1
Omniglot
81.66 1: 0.73%(5)
90.01 0.49%((255
88.50 1: 0,809(()
90.70 0.60%(2.5)
91.70 0.60%((1)
Aircraft
70.65 = 0.62%(5)
78.27 + (.54%(()
79.50 + 0.804K(3
83.80 + 0.60
82.40 :: 0.70%
Birds
76.99 0.64%H225)
79.62 + (I)
76.40 0.90%((255
0.80%(4.5)
0.80%.(455
DTD
71.28 : 0.56%)(2.5)
71.89 : (.59%((25)
73.10 = 0,70%
62.10 + (,70%((5
67.80 U (.8
Quickdraw
0.67%(5)
(.56%((4)
0.70%)(255)
(1)
Fungi
50.23 + 0.81%)(2
57.04 + (174%(()
48.20 + 0.90%((.5)
48.70 1.0%g
46.90 (.00(
VGG Flower
89.14 0.44%((4.5
91.09 + (2)
90.60 = (.50%(()
89.60 0.60%
90.70 0.50 (2)
Traffic Signs
89.14 1: (344%)(3
70.33 (256%
65.10 0.809K(5)
67.00 0.70%
73,50 0.70%((1
MSCOCO
$3.92 = 0.78%(1.55
$2.94 = 0.82%((1.5
52.10 + 1.005K(3)
43.40 + 1.00%k(5)
46.20 + (.0%%(4)
Averae
68.03%
71.68%
70.56%
68.53%
71.04%
Averge
3.45
2.40
3.75
2.55
Table 6: The same table Table 3 additionally annotated with per-row ranks.
17


Reviews:
[{"review_id": "qc3lD4xX2zu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an implementation method of using different numbers of shots of data for few-shot learning such as to mitigate the negative effect of \"different shots\". It optimized the FiLM parameters using meta gradient descent during episodic meta-training with different-shot learning tasks. It conducted the experiments on quite a set of \"meta-dataset benchmarks\".\n\nPros:\n\n1. The study of varying-shot task learning could be interesting, as the quantity of data in each task really matters for few-shot learning.\n\n2. \"Conditional on the shot number\" is a reasonable idea. \n\nCons:\n\n1. The motivation for conducting such a work is not new, and more importantly, the proposed method is too incremental compared to many existing works. (1) An existing work [Cao et al. ICLR’20] (also cited by this submission) studied the same problem both theoretically and empirically and proposed a similar linear transformation method to handle the varying-shot learning issue (Please kindly refer to section 3.3 in [Cao et al. ICLR’20]). This submission makes an incremental contribution of meta-learning transformation parameters conditional on shot numbers (using an existing method called FiLM). (2) Regardless of using such conditional, the idea of meta-learning feature transformation parameters (i.e. scaling and shifting parameters) has been well-deployed in existing few-shot learning papers [Sun et al. CVPR’19] (not cited in this submission) and [Oreshkin et al., 2018; Requeima et al., 2019; Bateni et al., 2019]. While as I can see from some other papers from the same authors of these related works, this transformation seems to be working well in many recent and similar tasks such as class-incremental learning (different numbers of samples for old or new classes) [Liu et al. CVPR’20] and semi-supervised few-shot learning tasks [Li et al. NeurIPS’19]. (3) Conditional on a single scalar ($k$ in this submission) is also not new in meta-learning, please kindly check [Shu et al. NeurIPS’19]. While I admit that this “conditional” is somehow the main contribution proposed in this submission for the few-shot settings. But I don’t think it is significant.\n\n2. The experiments lack comparisons to the sota few-shot learning methods (in the same setting of using varying-shot tasks in episodic training). This is a very clear weakness of this paper. The paper does not prove that using the proposed conditional meta-training can achieve better performance of varying-shot learning over the sota few-shot learning methods. Table 1 makes me feel that this paper proposes a new BN method.\n\n3. I want to talk more about varying shots. Actually, for different classes (different levels of hardness or diversity among the classes), the model may need different numbers of samples to learn a good representation for each of them (e.g. the common sense is that a model needs more samples for the non-rigid class \"person\" than the rigid class \"spoon\"). Given the high randomness of the class sampled in any meta-learning task, it is not clear how many shots a task really needs (for its all classes). So, I am a bit doubting \"is it necessary to study the problems of varying-shot learning\". \n\n[Sun et al. CVPR’19] Meta-Transfer Learning for Few-Shot Learning\n\n[Liu et al. CVPR’20] Mnemonics Training: Multi-Class Incremental Learning without Forgetting (details about feature transformation are given in the supplementary as the transformation method is not the contribution in this CVPR’20 paper)\n\n[Li et al. NeurIPS’19] Learning to Self-Train for Semi-Supervised Few-Shot Classification (details about feature transformation are given in the supplementary as the transformation method is not the contribution in this NeurIPS’19 paper)\n\n[Shu et al. NeurIPS’19] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (please check fig1(c) and its explanations)", "extra_scores": {}, "extra_text": {}}, {"review_id": "pn_JmBAuKG1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes to use a shot-conditioned model that specializes in pre-trained few-shot learning model to a wide spectrum of shots. The proposed approach is simple but effective, it trains neural networks that conditioned on a support set of a different number of shots K during the episodic fine-tuning stage, with FiLM as the conditioning mechanism.\n\n\nPros:\n1. This paper is clearly written, well-motivated, and thoroughly evaluated. It is an enjoyable reading experience .\n2. The proposed model and training algorithm is simple but effective\n3. Experiments are well designed, which first verifies that shot-conditioned few-shot learners can achieve relatively good performances on different Ks and then perform the large-scale evaluation.\n\nCons:\n1. Smoothing the shot distribution section is overly simplified. It is very hard to understand algorithm 1 given the limited details presented in the method section. I would suggest explaining in detail the smooth-shot procedure in the paper. \n2. Lack of ablation studies. There are two key designs of the few-shot learner presented in this paper, that does not have detailed ablation study results. The first is whether using the convex combination of FiLM parameters to obtain shot distribution s. The second is whether to use smoothing in shot distribution. How much are these two designs contribute to the model's final performances?\n3. In figure 2, it seems that the method trained with 5-shot has very good performances in all three scenarios presented? Would it be possible that such a good K-shot can be always found and therefore we do not necessarily need this K-conditioned few-shot learner? For instance, just always train with 5-shot and evaluate different shots.\n4. Instead of looking at the results of some instances of different K, it would be nice to have a comprehensive evaluation over different Ks. For instance, evaluating over all Ks from 1 to 40 and compute the average accuracy of each different model. To reduce the computation overhead, we can also try to evaluate Ks in {1, 10, 20, 30, 40}.\n5. Comparison of state-of-the-art methods for meta-dataset experiments.\n6. What is UMAP projection? The content of the experiments is not self-contained. How shall we read Figure 3?\n7. Figure 4 is also not straightforward to understand. What is the value on the y-axis? What does it mean? \n \n\nMinor:\n1. Inconsistent formats for average performances in Table 1. It seems that the results of the first two columns is badly formatted?", "extra_scores": {}, "extra_text": {}}, {"review_id": "mzfkA51ASJ_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to understand the role of this episodic fine-tuning phase and discovers that fine-tuning on episodes of a particular shot can specialize the pre-trained model to solving episodes of that shot at the expense of performance on other shots. It proposes a shot-conditional form of episodic fine-tuning.\n\nThe main contribution of this work is the training objective, which varying the shots by introducing a distribution over shots $P_k$. In addition, the model parameters are not separately or independently maintained for different shots. The author proposed a conditioning mechanism using FiLM, where $k$ is a conditional variable or input to the FiLM network ($\\gamma$ and $\\beta$). The idea of shot-specific feature extractors is not new, and it is a common trick in amortized variational inference to reduce the number of model parameters.\n\nIn the experimental section, I did not see other baselines in previous works. The comparisons are mostly against the two methods named standard and L2 BN, which seem to be simple variation of Prototypical Networks. Even the proposed SCONE is a variation of Prototypical Networks, it also reduces the novelty or originality.\n\nI am not familiar with this area or related research papers, but was assigned to review this paper. My evaluation is only based on the my understanding of the contents including in the manuscript. I would like to see the comments from other reviewers in this domain.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5HfOQ7TdPvQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a solution to few-shot meta learning approaches overfitting to the number of shots they are finetuned on, and not generalizing as well as expected to novel shots. In order to mitigate this problem, the paper suggests a parameterization of the meta learner which also conditions on the number of shots the model trains on. In practice, this is done via manipulation of the batch normalization parameters based on the number of shots. With this conditioning, the paper shows that the models perform better across a range of shots that they are evaluated on, compared to various sensible baselines.\n\nStrengths\n+ The paper does tackle an important problem, a meta learner should not be overfitting to the number of shots\n+ The paper has very thorough experiments and numerous visualizations (for e.g. of the learnt batch norm embeddings) to help gain better intuitions on what the method does\n+ The paper is well written and easy to understand\n\nWeaknesses\n\nImportant points for the rebuttal are marked with (*)\n\n(*) The problem setting of changing the number of shots during meta-training is still somewhat artificial since one could still test the model on a different number of shots during the Query set evaluation. I wonder if in light of this issue, conditioning on the number of shots is getting at the root of the issue. One would rather want to have methods like Cao et.al. which aim to make the model invariant (or insensitive) to the number of shots as opposed to explicitly conditioning on them. Would be great if the authors could clarify this issue.\n\n(*) Related to the above issue, a disadvantage of the proposed method over prototypical networks is that ProtoNets could be applied to episodes of any number of shots, even those unseen during training. Similarly, Cao et.al. can also be applied to any number of shots since it does a projection of the feature embedding. However the conditioning approach fails in this case, which is somewhat unrealistic. I understand that there is a smoothing step in the shot distribution which could partially help mitigate that issue, but it does not seem satisfying. How do the authors think this issue could be mitigated, and how should one view this as a potential shortcoming of the approach in light of the goals of the paper?\n\n(*) In general, it seems appropriate for the paper to have a comparison to Cao et.al. who propose a solution to shot-overfitting in the context of prototypical networks. While I understand that the proposed approach is more general than that of Cao et.al. that is not sufficiently demonstrated via. the experiments which still focus on the prototypical networks, which is not really an issue, but then warrants comparison to Cao et.al.\n\n(*) It would also be really useful to include a comparison to prototypical networks that does sum of the features instead of the mean to compute the prototype, which would preserve information on how many shots there exists in the support set. In general, it would also serve as a more stringent test of whether shot conditioning in the way the paper does it is needed when the method is able to deal with or recognize when the number of shots has changed, which the prototypical network approach is not able to do. \n\nIt would also help to guide discussion on how shot conditioning might look like for different algorithms. For prototypical networks we used FiLM, for something like MAML one could imagine unrolling for a different number of steps based on how many shots there are to prevent overfitting. A discussion on this would help improve the paper. \n\n(*) It would be useful to compare against a single model which trains on “k-shots” and is evaluated on a mixture of k-shots (as in meta-dataset) and search explicitly for the best value of “k” from [1, MAX_SHOTS]. Does the proposed approach still outperform that “best” (over k) model? That seems to be an important baseline comparison. \n\n** POST REBUTTAL UPDATE**\n\nI went through the other reviews and the author rebuttals. I thank the authors for throughly addressing all of my concerns and \nrunning some baseline experiments which I think are quite interesting, and add to the completeness of the paper, including a\ncomparison to EST and an optimal K-shot baseline. I think the idea is simple and interesting, and now the paper has enough\nexperimental comparisons to be a valuable addition to the conference. I have updated my score accordingly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a shot-conditional form of episodic fine-tuning approach for few-shot image classification. There were a number of concerns raised, e.g., there lacks of sufficient comparison with SOTA baselines, the justification on the significance of shot-aware approach is not entirely convincing, and incremental contributions in both novelty and improvements. While some of these issues were improved in the rebuttal, the revision remains not satisfied by the reviewers. Overall, I think the paper has some interesting idea, but is still not ready for publication. 
Decision: Reject