Few-Shot Few-Shot Learning and the role of Spatial Attention
Authors: Yann Lifchitz
Yannis Avrithis
Sylvaine Picard
Venue:  (2020)
Keywords: few-shot learning
spatial attention

Abstract: Few-shot learning is often motivated by the ability of humans to learn new tasks from few examples. However, standard few-shot classification benchmarks assume that the representation is learned on a limited amount of base class data, ignoring the amount of prior knowledge that a human may have accumulated before learning new tasks. At the same time, even if a powerful representation is available, it may happen in some domain that base class data are limited or non-existent. This motivates us to study a problem where the representation is obtained from a classifier pre-trained on a large-scale dataset of a different domain, assuming no access to its training process, while the base class data are limited to few examples per class and their role is to adapt the representation to the domain at hand rather than learn from scratch. We adapt the representation in two stages, namely on the few base class data if available and on the even fewer data of new tasks. In doing so, we obtain from the pre-trained classifier a spatial attention map that allows focusing on objects and suppressing background clutter. This is important in the new problem, because when base class data are few, the network cannot learn where to focus implicitly. We also show that a pre-trained network may be easily adapted to novel classes, without meta-learning.

Under review as a conference paper at ICLR 2020
FEW-SHOT FEW-SHOT LEARNING AND THE ROLE OF SPATIAL ATTENTION
Anonymous authors Paper under double-bling review
ABSTRACT
Few-shot learning is often motivated by the ability of humans learn new tasks from few examples. However, standard few-shot classification benchmarks assume that the representation is learned limited amount of base class data, ignoring the amount of prior knowledge that a human may have accumulated before learning new tasks. At the same time, even if a powerful representation is available, it may happen in some domain that base class data are limited or non-existent. This motivates us to study problem where the representation obtained from a classifier pre-trained on large-scale dataset of different domain, assuming no access to its training process, while the base class data are limited few examples per class and their role is to adapt the representation to the domain at hand rather than learn from scratch. We adapt the representation two stages, namely on the few base class data if available and on the even fewer data of new tasks. In doing so, we obtain from the pre-trained classifier a spatial attention map that allows focusing on objects and suppressing background clutter. This is important in the new problem, because when base class data are few, the network cannot learn where to focus implicitly. We also show that a pre-trained network may be easily adapted to novel classes, without meta-learning.
INTRODUCTION
The ever improving performance deep learning models, apart from technical progress, is largely due to the existence of large-scale datasets, fully weakly labeled by humans (Russakovsky et al.) 2015, Kuznetsova et al. 2018 Mahajan al. 2018). At the same time, reducing the need for supervision is becoming increasingly important, e.g. by taking advantage of prior learning (Bengio
ct al. |011 Mallya a et al. 2018 Li Hoiem 2018) or exploiting unlabeled data (Caron et al. 2018
Radosavovic et al. 2018).
An extreme situation is few-shot learning (Koch et al. 2015 Vinyals et al. 2016 Snell et al. 2011 Hariharan & Girshick 2017), where the problem is learn novel (previously unseen) classes using only a very small labeled training set, typically not more than 10 examples per class. Here not only the annotation but even the raw data not available. This problem is often motivated by the ability of humans to learn new tasks from few examples (Lake et al. 201 2015), which has given rise to meta-learning (Santoro et al. 2016 Mishra et al. 2017 Ravi & Larochelle [0017] Bertinetto et al.
2018 Han et al.) 2018), or learning to learn. In this scenario, a training set is treated as a collection of smaller sets where every class has few labeled examples.
However, there is a huge gap between the motivating example of humans learning new tasks and how the few-shot classification task is set up. On one hand, for the sake of simplicity in experiments, the base class datasets where the representation is learned from scratch, contain a few dozen or hundred classes with few hundred examples each. This by no means comparable to datasets available to date (Kuznetsova et al. 2018 Mahajan et al., (2018), let alone the amount of prior knowledge that human may have accumulated before learning a new task. On the other hand, for a given domain of novel classes, e.g. bird species (Welinder et al. 2010), base class data of such size in the same domain may not exist.
In this work, we depart from the standard few-shot classification scenario in two directions. First, allow the representation to be learned from large-scale dataset in a domain different than the base and novel class domain. In particular, model prior knowledge by classifier that is pre-trained on

---

Under review as a conference paper at ICLR 2020
such a dataset, having no access to its training process. We thus maintain the difficulty of domain gap and the simplicity of experiments (by not training from scratch), while allowing a powerful representation. Second, we assume only few or zero examples per base class. Hence, the role of base classes is to adapt the representation to the domain at hand rather than learn from scratch. This scenario can be seen as few-shot version of few-shot learning.
We treat this problem as a two-stage adaptation process, first on the few base class examples if available and second on the even fewer novel class examples. Because of the limited amount of data, it is not appropriate to apply e.g. transfer learning (Bengio et al. 2011) or domain adaptation (Ganin & Lempitsky 2014), either of the two stages. Because the network is pre-trained. and we do not have access to its training process or data, meta-learning is not an option either. We thus resort to few steps of fine-tuning as in the meta-testing stage of Finn et (217))
) and Ravi & Larochelle (0117).
Focusing on image classification, we then investigate the role of spatial attention in the new problem. With large base class datasets, the network can implicitly learn the relevant parts of the images where to focus. In our setup, base class data are few, so our motivation is that spatial attention mechanism may help the classifier in focusing on objects, suppressing background clutter. We observe that although the prior classes of the pre-trained classifier may be irrelevant to a new task, uncertainty over a large number of such classes may express anything unknown like background. This is a class-agnostic property and can apply to new tasks.
In particular, given an input image, we measure the entropy-based certainty of the pre-trained classifier in its prediction on the prior classes every spatial location and we use it to construct spatial attention map. This map can be utilized in a variety of ways, for instance weighted spatial pooling or weighted loss per location; and at different situations like the two adaptation stages or at inference. By exploring different alternatives, we show that a pre-trained network may be easily adapted to novel classes, without meta-learning.
In the following, we provide a detailed problem formulation and related background in section 2 then we describe our spatial attention mechanism in section 3 and its use in few-shot classification in section 4 We provide experimental results in section [S] and we conclude in section 6
PROBLEM, BACKGROUND, RELATED WORK AND CONTRIBUTION
Few-shot classification. We are given set of training examples X := {x,} X, and corre- sponding labels y == (n))== C C. where C' == [c] == {I....,c) ..} is a set of base classes. The objective is to learn a representation on these data, process that we call base training, such that can solve new tasks. A new task comprises set of support examples X' == {x C X and labels y' : (o)r' C (C")"', where n' < n and C' == [c'] is set of novel classes disjoint from C. The most common setting is k examples per novel class, so that n' = k'd', referred to as c'wway, kh-shot classification. The objective now is to learn a classifier on these support data, process that we call adaptation. This classifier should map new query example from X' to prediction in C".
Few-shot few-shot classification. Few-shot classification assumes there is more data in base than novel classes, and a domain shift between the two, in the sense of no class overlap. Here we consider modified problem where n can be small or zero, but there is another set C = [ce] of prior classes with even more data X and labels y with n < n := | and a greater domain shift to C, C Again, the most common setting is examples per base class, so that n kc. We are using classifier that is pre-trained on this data but we do not have direct access to either X y", or its learning process. The objective of base training is now to adapt the representation to the domain of C, C" rather than learn it from scratch; but we still call it base training.
In the remaining of this section we present general background on seot classification that typically applies to base classes C' or novel classes C", but may also apply to both, in which case the symbols e, C' and d', C' may be used interchangeably. Prior classes and the pre-trained network are only considered in the following sections.
The classiffer is a function fa. : X + R (resp. IR'') with learnable parameters 8, W. mapping a new example x E X' to vector of probabilities p = So,w (x) over (resp. e') base (resp.
Classifier.
2

---

Under review as a conference paper at ICLR 2020
novel) classes. The classifier prediction is the class of maximum probability
x (p) == arg max max
where Pj is the j-th element of p. The classifier is built on top of embedding subeediingfuncctin do : X X, this function yields
Brxd Given an example the spatial dimensions and the feature dimensions. For A' comprising 2d images for instance, the feature is w h X d tensor that the activation of the last convolutional layer, where w is the spatial resolution. The embedding is vector in IR in the special case r=
d feature tensor (x(xx), where represents
The embedding parameters 0 may be updated into 0' in the adaptation process, in which case we have an embedding function Φer and classifier for .w. Again 6, 6' may be used interchangeably.
Cosine classifier. A simple form of classifier that was introduced in few-shot learning independently by Qi et al. (2018) and Gidaris & Komodakis (2018) is parametric linear classifier that consists of a fully-connected layer without bias on top of the embedding function dy followed by softmax. If W := (wr)}=1 is the collection of class weights with w, E Rrxd, the classifier is defined by
fx) == a (T(s(00(x), 1,)15-1)
for x E X, where 1 Rd is the softmax function (u)) := (e") _e"
Σ for torue R
a trainable scale parameter and is cosine similarity We minimize the cost function
J(X (x..w(
over 0, W at base training, where ((p, v) == log Py for p Rq, y C' is the cross-entropy loss.
Prototypes. An alternative classifier that more appropriate during few-shot adaptation or at testing is a prototype classifier proposed by (Sell et al. (2017) and followed by QQ et al. (2018) and Gidaris &
Komodakis (2018) too. If S := {i n 3/4 =j j} denotes the indices of support examples labeled in class J. then the prototype of this class is given by the average features
1
Σ doo (x²) LES,
Pj [S
of those examples for j E C²_ Then, denoting by P (py) the collection of prototypes, a query X E X' is classified as sør, (er.p(2 pxx), as defined by (2).
Dense classifier. This is classifier where the loss function applies densely each spatial location of the feature tensor rather than by global pooling or flattening as implied by (2) The classifier can be
of any form but cosine classifier (Qi et al. 2018 Gidaris & Komodakis 2018) is studied by Lifchitz 2019). In particular, the embedding 6s (x) is al.
collection of vectors [d(4) (x)]{= where
(
E Rd is an embedding of spatial location & E [r]. The classifier (2) is then generalized to
fe.w w X + Rrxc now mapping an example to vector of probabilities per location, defined by
fo fe,w(x) (x) = o (T(s(db)) (x), w))l)
(5)
for E X, while the class weights W are shared locations with wy IRd. Cross-entropy applies using the same label y, for each location of example x,, generalizing the cost function (3) to
J(X
(x:),,))
Related work. Prototypical networks (Snell al. 2017) use a prototype classifier. At testing, query is classified to the nearest prototype, while at adaptation, computing a prototype per class
For matrices u, Rr s(u, v) = (u,, v) /[Ija]] [[v]]) with (:-)) and ||-I] heing the Frobenius inner product and norm respectively.
3

---

Under review as a conference paper at ICLR 2020
is the only learning to be done. Base training is based meta-learning : number of fictitious tasks called episodes are generated by randomly sampling a number of classes from C' and then a number of examples in each class from X with their labels from y. These data are assumed to be support examples and queries of novel classes C'. Labels are now available for the queries and the objective is that they classified correctly. In imprinting (Qi al. 2018), cosine classifier (2) and standard cross-entropy (3) are used instead at base training. At adaptation, class prototypes P' are computed { and imprinted in the classifier, that is, W replaced by W == (W, P). The entire embedding function is then fine-tuned based again on (3) to make predictions on n + n' base and novel classes, which requires the entire training data yX.y). Few-shot learning withour forgetting (Gidaris& & Komodakis) 2018) uses model similar to imprinting the main difference being that only the weight parameters W of the base classes are stored and not the entire training data. At base training, a cosine classifier is trained by (3) followed by episodes. At adaptation, prototypes are adapted to W by a class attention mechanism. Model agnostic meta- weta-learnnng (MAML) (Finn itt al, (2017) uses folly-connected layer as classifier. At adaptation, the entire embedding function is fine-tuned on each new task using (3) only on the novel class data, but for few steps such that the used where the loss function mimics the iterative optimization that normally takes place at adaptation. In implanting (Lifchitz et al. 2019). a prototype classifier is trained in episodes, keeping the base embedding function fixed but attaching parallel implant stream of convolutional layers that learns features useful for each new task.
classifier does not overfit. At base training, episodes
Contribution.
The problem we consider is variant of few-shot learning that has not been studied before. It involves sequential adaptation of given network in two stages, each comprising a limited amount of data. There are many ways of exploiting prior learning to reduce the required amount data and supervision like transfer learning (Bengio et al.) 2011 Yosinski et al. 2014), domain adaptation (Ganin & Lempitsky 2014; Rebuffi al. 2017: Mallya et al.) 2018). or incremental learning (Li & Hoiem] 201 8] Rebullli et all.]2016 Yoon et al.
2018). However, none applies to the
few-shot domain where just a handful of examples are given.
Attention has been studied as core component of several few-shot learning and meta meta-learning approaches (Vinyals et al. 2016 Mishra et al. 2018 Ren et al., 12088:/Gdaae & Komodakis 20188. but it always refers to examples (e-g. images) as unit. Spatial atlention on the other hand refers to neurons at different spatial locations. It is ubiquitous in several problems, for instance weakly supervised object detection (Zhou et al., 2016, Hou et al. 2018 Zhu et al., (0177) and non-local convolution (Hu et al. 2018 Wang et al. 2018 Chen et al.[20188 8) but has not been applied to few-shot learning until recently (Lifchitz et al. 2019 Wertheimer & Hariharan| 2019 Li et al. 2019b Liu et al. 2019). By breaking up one example (e.g. image) to several, one at each spatial location, the supervision becomes weak, in multiple-instance learning (Andrews al. 2003). typical e.g. in weakly-supervised image segmentation (Jetley et al., (018). Our spatial attention mechanism is extremely simple, based on the pre-trained network, without training on the base novel class data.
SPATIAL ATTENTION FROM PRE-TRAINING
We assume pre- pre-trained network with an embedding function Φe- : X + followed by global average pooling (GAP) and classifier that is fully connected layer with weights V := (w) pdxe" and biases ba e Re denoted jointly by U == (W*, b²). Without re-training, we remove the last pooling layer and apply the classifier densely as in with temperature T. Then, similarly to (5), the classifier for.0r _u- X - R. maps an example to vector of probabilities per location, where classifier parameters U are shared over locations:
convolution, followed by softmax
for (x) :=):= (x))=)][]]] a ( (W Pg (x) + b b-))
(g)
We now want to apply this classifier to examples in set X (resp. X') of base (resp. novel) classes C (resp. C') in order to provide a spatial attention mechanism embeddings obtained by parameters 0 (resp. 8¹). We formulate the idea on X, C, A in this section but it applies equally to X', C², " In par- o(g)
ticular, given an example x X, we use the vector of probabilities (99) := to spatial location 9 E [r] to compute a scalar weight (x),, expressing the discriminative power of the particular location q of example x.
==f(1) -f U- (x) corresponding
4

---

Under review as a conference paper at ICLR 2020
Figure 1: Examples of images from CUB (top) and minilmageNet (bottom) overlaid with entropy- based spatial attention maps obtained from (8) using only the predicted class probabilites from ResNet-18 18 pre-trained on Places. See section for details on datasets and networks.
Since x belongs to a set of classes C different than C there is no ground truth to be applied to the output of the pre- pre-trained classifier for fu. However, the distribution pl can still be used to evaluate how discriminative the input is. We use the entropy function for this purpose, H(p) EyP} Pj log(p;). ). We map the entropy to [0, 1]. measuring the certainly of the pre-trained classifier in its prediction on the prior classes CO:
( (x) == H (g)) (x)
(8)
log c
for [r], where we ignore dependence on parameters 0 U to simplify notation, since they remain fixed. We use this weight for location q assuming that uncertainty over large number of prior classes expresses anything unknown like background, which can apply to new set of classes. We
then & nooomalli the weights w(x) = [w()) (x)]; IR" as v(xx = wxx)/ [[w(x)],, We call
(x) the spatial attention weights of x.
The weights are applied in different ways depending on the problem. If the embedding 6o(x) is normally vector Rd obtained by GAP on feature tensor (g(x)) E Rrxd as -Ever) ( (x) for X, then GAP replaced by global global weighted average pooling (GwAP):
beg (x) = Σ ( (x) (x).
(9)
vE(r)
for x E X. We recall that this applies equally to 0' the case of novel classes.
Figure shows examples of images with spatial attention maps. Despite the fact that there has been no training involved for the estimation of attention on the particular datasets, the result can still be useful in suppressing background clutter.
4 SPATIAL ATTENTION IN FEW-SHOT CLASSIFICATION
Here we discuss the use of attention maps at inference on novel classes, as well as learning on novel classes, In the latter case, the weights are pre-computed for all training examples since the pre- pre-trained network remains fixed in this process. In summary, we either replace GAP by GWAP (9) in all inputs the embedding network, or use dense classification (
4.1 BASE CLASS TRAINING
Starting from traited embedding network dæs, either solve new tasks novel classes C" directly, in which case 9 = 8", or perform base class training, fine-tuning from 0". Adaptation may involve for instance fine-tuning the last layers or the entire network, applying spatial attention
5

---

Under review as a conference paper at ICLR 2020
mechanism not. Recalling that Og- is still needed for weight estimation (8) the most practical setting is to fine-tune the last layers. in which case @ shares the same backbone network with Φg.. perform few gradient descent steps with low learning rate.
Following MAML (Finn et al., 2017),
We use a dense classifier fo.w X' + gixe (5) with class weights W. Given the few base class examples X and labels y, learn W at the same time fine-tuning by minimizing (6).
4.2 NOVEL CLASS ADAPTATION
Optionally, given the few novel class support examples X' and labels y', we can further adapt the embedding network, while applying our attention mechanism to the loss function. As in section 4.1 6e shares the same backbone with Φe. being derived from it by fine-tuning the last layers. We perform even fewer gradient descent steps with lower learning rate
We use prototype classifier where vector embeddings Φe (x') of support examples x' X' are obtained by GwAP with @ør defined as in (9) and class prototypes P := (py) are obtained per class by averaging embeddings of support examples as defined by 44) and updated whenever O' is updated. The classifier før P : X + Ie is a standard cosine classifier (2) and the loss function is standard cross- cross-entropy J(X, y: 0, P) (3) with embedding Φg obtained by GwAP (9). Attention weights apply to embeddings of all inputs the network, each time focusing on most discriminative parts. In case of no adaptation to the embedding network, we fix 0' = 0. Computing the prototypes P is then the only learning to be done and we can proceed to inference directly.
4.3 NOVEL CLASS INFERENCE
At inference, as in section 4.2 we adopt a prototype classifier where vector embeddings de (x') of support examples x' E X are obtained by GwAP with Φu defined as in (9) and class prototypes
p: (p,) are obtained per class by averaging embeddings of support examples as defined by (4p. Then, given a query E X, we similarly obtain a vector embedding de (x) by GwAP (9) and predict the class π T())rp(x)) of the nearest prototype according to cosine similarity where स given by ① and fer
_P by (2). We thus focus discriminative parts of both support and query examples, suppressing background clutter.
EXPERIMENTS
5.1 EXPERIMENTAL. SETUP
Pretrained Network. We assume that have gathered prior knowledge on unrelated visual tasks. This knowledge is modeled by a deep convolutional network, trained on a large-scale dataset. In our experiments. we choose use a ResNet-18 (He et al. 2016)) pre-trained on the Slaces365-Standatt subset of Places365 (Zhou et al. 2017). We refer to this subset as Places. This subset contains around 1.8 million images across 365 classes. The classes are outdoor and indoor scenes. We select this dataset for its large scale, diversity of content and different nature than other popular datasets (see below). Images are resampled to 224 224 pixels for training. We choose ResNet-18 8 as the architecture of the pre-trained model as it is a powerful network that is also used in other few-shot learning studies (Chen et al., 2019) Dvornik et al.] 2019), which helps in comparisons. We make assumption on the pre-training of the network. We do not access either the pre-training process or the dataset. We rather use a publicly available converged model that has been trained with fully-connected layer classifier, as assumed in section 3
like CUB-200-201
Datasets. We apply our method to two standard datasets in few-shot learning. The first is CUB-200- 2011 (Wah et al. (01IT, referred to as CUB, originally meant for fine-grained classification, and subsequently introduced to few-shot learning by Hilliard et al. (2018). This dataset contains 11,788 images of birds across 200 classes corresponding to different species. We use the split proposed by Ye et al. (2018). where 100 classes are used as base classes and the remaining 100 novel, out of which 50 for validation and 50 for testing. CUB images are cropped using bounding box annotations and resampled to x24x 224.

---

Under review as a conference paper at ICLR 2020
NOVEL:
NOVEL: =
Attention
Adapiaaion
BASE
PLACES
38.80:1:34
39. 39.69:1:224
39.76::024
40.79:10:24
55 05.099:..24
56.9520.23
63.2920024
64.27:1:23
40.50:1:223
41.74:1:.24
41.11:00:4
42.23
57.25
58.89:00255
65.42-0.23
66.78:10.25
36.474-028
57.16+0029
$6.69-0.29
57.32++0.29
74.27400.5
74.95--0.33
75.82-0223
76.324+0:3
ALL
80.68-00.27
80,48:0.27
80.68=0.27
80.56:00227
90.18:00116
90.33=00166
91.22=0015
91,17:015
BASE
RANDOMLY INITIALIZED
31.65:1:0119
31.37:10.19
39.45-10.20
42.70:10.21
k = 5
40.52++0025
40.50+=0.26
52. 02.94:00.25
53.45=+0:25
ALL
71.7840030
71.77
0..60020.188
05.96220.19
Baseline++
67.02-4-050
83. 53.5588-0.554
ProtoNet
71.88:4091
87 0.444
Ensemble
68.77440.71
84.62-40.44
Table 1: Average 5-way K' k'-shot novel class accuracy CUB. We use ResNet-18 either pre-trained on Places or we train it from scratch on k base class examples. ProtoNet (Snell et al., 2017) is as reported by Chen al. (2019). For ensemble (Dvornik al. (019). report the distilled model from an ensemble of 20. Baselines to be compared only to randomly initialized with k = ALL
The second dataset is minilmageNet (Vinyals et al. 2016). a subset of the ImageNet ILSVRC-
12 (Russakovsky et al. 2014) containing 100 classes with 600 images per class. Following the split from Ravi Larochelle (2017), 64 classes are used as base classes and 36 as novel, out of which 16 for validation and 24 for testing. Originally, minilmageNet images have been down-sampled from the ImageNet resolution to 84. In this work, similarily to ohehn et al. (2019); Dvornik et al. (2019). resample 224 224x224 instead, which is consistent with the choice of pre-trained network.
Contrary to CUB, minilmageNet has some non-negligible overlap with Places. Some classes or even objects appear in both datasets. To better satisfy assumption of domain gap, remove the most problematic overlapping classes from minilmageNet. As detailed in in/Appendix A] we remove 3 base classes, validation class and 2 novel classes. We refer this pruned dataset as modified mini/mageNNet. For the sake of comparison and because this overlap can happen in practice, we also experiment on the original miniImageNet, reported in Appendixx B
Evaluation protocol. To adapt to our few-shot version of few-shot learning, we randomly keep only k images per base class. We experiment with & {0, 1,5] and k (0, 20, 50) respectively for the CUB and minil MageNN... For novel classes, we use the standard setting K E [1,5]. We generate a few-shot task on novel classes by selecting support set X'. In particular, we sample classes from the validation or test set and from each class we sample k' images. In all experiments, d' = 5, i.e. 5-way classification. For each task we additionally sample 30 novel class images per class, use as queries for evaluation. We report average accuracy and 95% confidence interval over 5,000 tasks for each experiment. The base class training set X contains examples per class for each base class in C. Each experiment can be seen as a few-shot classification task on few base class examples.
Baselines. We evaluate experiments with the network being either pre- mre-trained on Places or randomly initialized. In both cases, we report measurements for different number k of examples per base class, as well as all examples X. the latter case (randomly initialized), we do not use the option k given
because then there would be no reasonable representation to adapt to perform inference a few-shot task on novel classes. In all cases, we compare to the basclines of using no adaptation and no spatial attention. When learning from scratch, spatial attention is not applied as do not have access to the pre-trained classifier.
the case of random initialization, and using all examples in X. compare to Baseline++ (Chen et al., 2019) and prototypical networks (Snell et al. 2017). as reported in the benchmark by CChn et al. (2019), as well as category traversal (CTM) (Li et al. 2019a) and ensembles (Dvomik al. (2019), all using ResNet-18 18. They only be compared to our randomly initialized baseline when using base training on all data.
Implementation details. At base training, tum with mani-bachhes of size 200. At adaptation, we perform a maximum of 60 iterations over the support examples using Adam optimizer with fixed learning rate. In both cases, the learning rate,
stochastic gradient descent with Nesterov momen-
7

---

Under review as a conference paper at ICLR 2020
NOVEL: K' =1
NOVEL: K' =
Attention Adaptation
BASE
PLACES
k
61.66=0.30
63.36=0.29
62.09:10:0
63.5610.30
78.86::0:22
80:15::::3
80.38:10:2
81:05::::22
k 20
62.95=0.29
03.15:00.2
63.11:10.29
63.13:10.29
78.414021
78.53::0.21
79.67
7::22::0.21
50
65.10-0029
65.1840.29
65,244-029
79.944020
79.99+00.0
80,96+-0.20
ALL
66.20=0029
65.94=0.29
66.23#0229
66.06::0.29
80.37+021
20.24
81,56:20030
81.50+00.30
BASE
RANDOMLY INITIALIZED
20
33.43=0.211
33.35:10.21
43.83:10:21
44.21:1:021
k = 50
41.03-0.24
41.05-+0.24
$4,68++0022
$4.922++0.22
AL1
55.99-0.28
56.1320.28
72.434022
10:00.0021
Table 2: Average 5-way k'-shol novel class accuracy on modified minilmageNet. We use ResNet- 18 either pre-trained on Places or we train it from scratch on base class examples. Baselines only shown in Appendix B on the original minilmageNet.
schedule if any and number of iterations are determined the validation set. The temperature used by (7) for the computation of the entropy is fixed per dataset, again on the validation set. In particular, use T 100 and T = 2.6 respectively for CUB and modified minilmageNet.
5.2 RESULTS
We present results in Tables D and respectively for CUB and modified minilmageNet.
Effect of base training. For aneiigrieed few-shot classification (CUB), base training is extremely important adapting to the new domain, improving the baseline 1-shot accuracy by more than 40% with no adaptation and spatial attention. On object classification in general (modified manelmageNett), it is less important, improving by 4.5%. It is the first time that experiments are conducted on just small subset of the base class training set. It is interesting that 50 examples per class are bringing nearly the same improvement as all examples, i.e. hundreds per class.
Effect of (novel class) adaptation. Fine-tuning the network on k novel class examples per class, even fewer than i: in the case of base classes, comes with the risk of over-fitting. We still show that small further improvement is possible with small learning rate. The improvement is more significant when & low, in which case, more adaptation of the embedding network the novel class domain is needed. In the extreme case of CUB dataset without base training, adaptating on only the 25 images of the 5-way 5-shot tasks brings an improvement of 8.20%
Effeet of spatial attention. Spatial attention allows focusing on the most discriminative parts of the input, which is more beneficial when fewer examples are available. The extreme case is having no base class images and only one image per novel class. In this case, most improvement comes on modified minilmageNet without base training, where spatial attention improves 5-way 5-shot classification accuracy by 1.5% after adaptation. The attention maps appear to be domain independent as they improve CUB accuracy even when images from the bird domain have been seen (k 0).
6 CONCLUSION
In this paper we address the problem of few-shot learning when even base classes images are limited in number. To address it, we use a pre-trained network on large-scale dataset and very simple spatial attention mechanism that does not require any training on the base or novel classes. We consider two few-shot learning datasets: CUB and minilmageNet, with different domain gaps to our prior dataset Places. Our findings indicate that even when the domain gap is large between the dataset used for pre-training and the base/novel class domains, is still possible to get significant benefit from base class training even with a few examples, which is very important as it reduces the need for supervision. The gain from spatial attention is more pronounced in this case.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Stuart Andrews, Joannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instamee instance learning. In NIPS, 2003.
Yoshua Bengio. Guyon, G. Dror, Lemaire, G. Taylor, and Silver. Deep learning representations for unsupervised and transfer learning. in Proc. of ICMI, 2011.
Luca Bertinetto, João F Henriques, Philip HS Torr, and Andrea Vodaldi. Meta-lcarning with differentiable closed-fome solvers. arXiv preprint 180Xiv8805.0000 36, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. ECCV, pp. 132-149, 2018.
Wei-Yo Yu Chen, Cen-Cheng Liu, Zsolt Kira, Yu- Chiang Frank Wang, and Jia Bin Huang. A closer look at few-shot classification, ICLR, 2019.
Yunpeng Chen, Yannis Kalantidis. Jianshu Li, Shuicheng Yan, and Jiashi Feng. A 2-nets: Double attention 2018.
networks. In NIPS, pp.
Nikita Dvomik, Cordelia Schmid, Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification, ICCV, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning adaptation of deep networks. In ICML, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. 2014.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
Chunrui Han, Shiguang Shan, Meina Kan, Shuzhe Wu, and Xilin Chen. Face recognition with contrastive convolution. In ECCV, 2018.
Bharath Hariharan and Ross B. Girshick. Low-shot visual recognition by shrinking and hallucinating features. ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D. Corley, and Nathan O. Hodas. Few-shot learning with metric-agnostic conditional emheddings. CoRR, abs/1802 04376, 2018.
Qibin Hou, Pao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-crasing network for integral object attention. In NIPS, pp. 1399955,,91 2018.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excicee networks. In CVPR, pp. 7132-7141, 2018.
Saumya Jetley, Nicholas A. Lord, Namhoon Lee, and Philip H. S. Torr. Learn to pay attention. In ICLR, 2018.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICMLW, 2015.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, and Tom Duerig. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00983 2018
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedingy of the Annual Meeting the Cognitive Science Society, 2011.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenhaum. Human-level concept learning through probabilistic program induction. Science. 133--366 2015.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task-relevant features for few-shot learning by category traversal. In CVPR, 2019a.
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. In CVPR, 2019b.
Z. Li and D. Hoiem. Learning without forgetting. IEFE Transactions Pattern Analysis and Machine Intelligence, 20112) Dec 2018

---

Under review as a conference paper at ICLR 2020
Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and implanting for few-shot learning. CVPR, 2019.
Ziwei Liu, Zhongqi Miao, Xianhang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, pp. 181-196, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In ECCV, 2018.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Meta-learning with temporal convolutions. arXiv preprint arXiv:100.00314,, 2017.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-leamer. ICLR, 2018.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In CVPR, 2018.
Ilija Radosavovic, Piotr Dollar, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In CVPR, June 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. ICLR, 2017.
Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, and Christoph H. Lampert. iCaRL; Incremental classifier and reprosentation learning. arXiv preprint vrXiv:161.00770 2016.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S Zemel. Incremental few-shot learning with attention attractor networks. arXiv preprint arXiv: 1810.07218, 2018.
Russakovsky, J Deng, H Su, J Krause, S Satheesh, S Ma, Z Huang, A Karpathy, A Khosla, M Bernstein, al. Imagenet large scale visual recogmition challenge. arXiv, 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3)211-252, 2015. doi: 10.1007/s1 11007//11163301111 1633-015-0866...
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with eeemory-aggmente neural networks. In ICML, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel, Prototypical networks for few-shot learning. In NIPS, 2017.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report
California Institute of Technology, 201
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pp. 7794-7803, 2018
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongic, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-00 Californi Institute of Technology, 2010.
Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization in realistic settings. In CVPR, 2019.
Han-Jia Ye. Hexiang Hu, De-Chuan Zhan, and Fei Sha. Learning embedding adaptation for few-shot learning. CoRR, abs/1812.03664 2018.
Jachong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. 2018.
10

---

Under review as a conference paper at ICLR 2020
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable features in deep neural networks? In NIPS, 2014,
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. CVPR, June 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017.
Yi Zhu, Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Soft proposal networks for weakly supervised object localization. In ICCV, Oct 2017.

---

Under review as a conference paper at ICLR 2020
A DETAILS ON REMOVING DATASET OVERLAPS
We study the overlap between Places and minilmageNet by measuring, for each minilmageNet class, what is the most frequent prediction among Places classes by the pre-trained Resnet-18 classifier and what proportion of examples are classified in this class, Ranking minilmageNet classes by that proportion, we check class names on both datasets and manually inspect examples in the top-ranking classes. We chose to remove only the most clearly overlapping classes, that is, identical classes and classes of objects contained in images of a Places class. For instance, most images from bus station indoor class of Places contain bus, we chose to remove the school bus class from man. Table lists the classes removed from minilmageNet this way.
mineilmageee split
milelliaageeet class
Places class with overlap
TRAIN
carousel
carnusel amusement arcade cliff
TRAIN TRAIN
slot
cliff
VALIDATION TEST
cural reel
underwater deep bus station indoor bookstore
school bus
bookshop
Table 3: Classes removed from minilmageNer to form the modified minilmageNer dataset and the
corresponding overlapping Places classes.
B ORIGINAL miniLMAGENET RESULTS
We also run our experiments on the original minilmageNet dataset which partially overlaps our pre-training dataset Places. We use T 2.4 for the temperature in 7 The remaining setup for modified minilmageNet. Results are shown in Table a
NOVEL:
NOVEL:
Attention Adaptation
BASE
PLACES
65.80-4031
67.56200.1
67.96=0.31
$1.90-20.2
83.00=00222
83.45-20222
k 20
66.98:10229
67.63-1:229
67.3220:29
67.80:2.29
81.442021
81.82200211
82,562021
82.92:1021
50
09.11:1029
09.17:1029
09.22:10:9
69. 09.30:1:.29
83.14:10.20
83.25::0.20
83.97:0.19
84.10:10.19
ALT.
69.71:0.29
29:81
69.70 29.701..20
70.00:00.9
83.31
83.25:20.19
84.20.20.19
84.2420.1
RANDOMLY INITIALIZED
= 20
37.754:023
37.74--0.23
49.13-4:0.35
49.67:10.23
k: = 50
42.79-4:223
42.79::0.25
$7.18-20.25
57. 0.2223
ALI.
59.68-10.7
75.42:20.20
75, 75.95.00.00
Buseline++
$1.87+t077
75. 05.68-
ProtoNet
03.68:20665
Ensemble
63.06:1:663
80.63-20.43
CTM
64.12:10.55
80.51
Table 4: Average 5-way k' novel class accuracy minilmageNet.. We use ResNet- 18 either pre-trained on Places or we train from scratch on l base class examples. ProtoNet (Snell et al.) 2017) is reported by [Chnn et al.] (2019). CTM refers to the data-augmented version of Li et al. (2019a). For ensemble (Dvomik et al. 2019). we use the distilled model from an ensemble of 20. Baselines to be compared only to randomly initialized with A ALL.
Compared to the results of modified minilmageeet (Table 2, performances nearly uniformly increased by 3-4% and conclusions remain the same. The increase in performance is due to having more training data, well as putting back casily classified classes in the test dataset. Observe that, unlike CUB (cf. Table T}, CTM (Li et al.) 2019a) and ensembles (Dvornik et al., 2019) perform better than randomly initialized baseline Our objective is not to improve the state of the art of the standard few-shot setup. but rather to study the new problem using network pre-trained on large-scale dataset. In this respect, our simple baseline better facilitates future research.
12


Reviews:
[{"review_id": "rkgOJZaaKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A new task is suggested, similarly to FSL the test is done in an episodic manner of k-shot 5-way, but the number of samples for base classes is also limited. The model is potentially pre-trained on a large scale dataset from another domain. The suggested method is applying spatial attention according to entropy criteria (or certainty) of the original classifier (from a different domain).\n\n\nI think the suggested task is important and more realistic than the usual FSL benchmarks. I would modify it so instead of discarding mini-imagenet classes that are overlapping with Places I would discard the problematic Places classes. This way it will be easier to compare to standard FSL. Also, I don’t understand why for CUB the benchmarks includes k={0,1,5} while for mini-imagenet it is k={0,20,50}, obviously k={0,1,5} are more interesting.\n\nAs for the suggested method, I find it hard to judge since there are no strong baselines to compare against. Also, the ablation study of removing the attention and/or adaptation doesn’t result in a definitive conclusion. \n\n\nUpdate:\nWhile your comments do weaken some of my concerns, I'm afraid it is not enough for changing my previous rating. I think being more careful about the benchmark definition with regards to train/test overlap and comparing to stronger baselines will help improve the paper for future submissions.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xtjozaYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new realistic setting for few-shot learning that we can obtain representations from a pre-trained model trained on a large-scale dataset, but cannot access its training details. Also, there may be a large domain shift between the dataset of the pre-trained model and our dataset. For the pre-trained model, they will not only use its weights but also use it to generate a spatial attention map and help the model focuses on objects of images. Back to the standard few-shot classification problem, they will first adapt the model with base class samples and then adapt to novel classes.\n\nThe proposed new setting is very meaningful since we already have many powerful pre-trained models and why not exploit its usage for few-shot learning problems. However, I doubt the novelty and effectiveness of the attention way used in the paper. The attention module helps the model focuses on the objects not the background, which is absolutely correct. But there are already some relevant studies in the missing reference Large-Scale Long-Tailed Recognition in an Open World, CVPR2019. Also, from the results, the significant improvements come from the weights of the pre-trained model but not the attention used. Is the attention way used in the paper a good way to exploit the pre-trained model for few-shot classification problems?\n\nAlso, I am curious about the dense classification used in the adaptation phase. Will it achieve similar performance with finetuning using just standard loss?\n\nBtw, according to the formatting instructions, the abstract should be limited in one paragraph.\n\n=========================================================\nAfter Rebuttal:\n\nI thank the author for the response.\n\nI do see there are differences in the way of generating attention masks between the proposed work and (Liu et al.). But the improvements from the attention module is not significant, especially when using all base data.\n\nI keep my original scores.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1epAP5jFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a problem “few-shot few-shot learning” that aims to firstly transfer prior knowledge from one domain to the domain where the base training tasks reside, and then train a few-shot learning model on training tasks and apply it to novel test tasks. The two “few-shot” in the name refers to base training tasks and novel test tasks. In their algorithm, they use a model pre-trained on another dataset as the prior knowledge and fine-tune it on training tasks. During the test, they use the weighted average of samples’ representations per class as the prototype of each class, where the weight is large for samples with more discriminative prediction over pre-trained domain’s classes. Afterward, classification is reduced to finding the nearest neighbor among the class prototypes. Some experiments show that the pre-trained model can improve few-shot classification accuracy.\n\nMy major concerns:\n\n1) They try to propose a new problem, but their description shows that the problem is exactly the same as what most “few-shot learning” works aim to solve: use a pre-trained model, train a meta-learner on few-shot training tasks, and apply it to novel test tasks. \n\n2) The algorithm does not have any important contributions comparing to existing ones: they define a prototype per class based on the pre-trained model and apply the nearest neighbor classification. The so-called “prototypical classifier” is actually the nearest neighbor classifier since no prototypical network structure is learned in the proposed method.\n\n3) I would not call the weighted average as “attention” because it is not: the weight in attention is computed by a module with learnable parameters, while the weight in this paper is computed by the entropy of a pre-defined model’s output prediction. \n\n4) The “spatial attention” only makes sense when the pre-trained domain’s classes can describe the main concepts appearing in the images of novel classes. This assumption is too strong since it requires class-level (rather than lower-level) relationships.\n\n5) The base training is not necessary in the algorithm: it is used to only fine-tuning theta and W. As the author said in the beginning of Section 4.1, they can directly solve novel tasks based on the pre-trained model.\n\n6) The experiments show that the pre-trained model is helpful in few-shot learning, which is a known fact.\n\n7) The writing of this paper is very poor: a lot of typos and grammar errors, inconsistency between narratives, abuse of notations, wrong equation reference, even missing punctuations. They make the paper hard to understand.\n\n-------------\n\nUpdate:\n\nThanks for the authors' rebuttal! After reading their rebuttal, I still have main concerns about the novelty of the problem and the writing quality. The proposed method tends to be incremental. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the interesting problem of meta-learning in problem spaces where training "tasks" are scarce.  Two criticisms that seems to shared across reviewers are that (i) it is debatable how "novel" the space of meta learning with "few" tasks is, especially since there aren't established standard for how many training tasks should be available, and (ii) the paper could use more comparisons with baseline methods and ablations to understand the contributions.  As an AC, I down-weight criticism (i) because I don't feel the paper has to be creating a new problem definition; it's acceptable to make advances within an existing space.  However, criticism (ii) seems to remain.  After conferring with reviewers it seems that the rebuttal was not strong enough to significantly alter the reviewer's opinions on this issue, and so the paper does not have enough support to justify acceptance.  The paper certainly addresses interesting issues, and I look forward to seeing a revised/improved version at another venue. 
Decision: Reject