Meta-Learning by Hallucinating Useful Examples
Authors: Yu-Xiong Wang
Yuki Uchiyama
Martial Hebert
Karteek Alahari
Venue:  (2020)
Keywords: few-shot learning
meta-learning

Abstract: Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks, which aim to learn novel concepts from very few examples. The hallucination process, however, is still far from generating effective samples for learning. In this work, we investigate two important requirements for the hallucinator --- (i) precision: the generated examples should lead to good classifier performance, and (ii) collaboration: both the hallucinator and the classification component need to be trained jointly. By integrating these requirements as novel loss functions into a general meta-learning with hallucination framework, our model-agnostic PrecisE Collaborative hAlluciNator (PECAN) facilitates data hallucination to improve the performance of new classification tasks. Extensive experiments demonstrate state-of-the-art performance on competitive miniImageNet and ImageNet based few-shot benchmarks in various scenarios.

Under review as a conference paper at ICLR 2020
META-LEARNING BY HALLUCINATING USEFUL Ex- AMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks, which aim learn novel concepts from very few examples. The hallucination process, however, is still far from generating effective samples for learning. In this work, we investigate two important requirements for the hallucinator - (i) precision: the generated examples should lead to good classifier performance, and (ii) collaboration: both the hallucinator and the classification component need to be trained jointly. By integrating these requirements as novel loss functions into general meta-learning with hallucination framework, model-agnostic PrecisE Collaborative hAlluciNator (PECAN) facilitates data hallucination to improve the performance of new classification tasks. Extensive experiments demonstrate state-of-the-ard performance on competitive minilmageNet and ImageNet based few-shot benchmarks in various scenarios.
INTRODUCTION
Modern deep learning models rely heavily on large amounts of annotated examples (Deng et al. 2009). Their data-hungry nature limits their applicability to real-world scenarios, where the cost of annotating examples is prohibitive, or they involve rare concepts (Zhu et al. 2014 Fink 2011). In contrast, humans can grasp new concept rapidly and make meaningful generalizations, even from single example (Schmidi 2009). To bridge this gap, there has been recent resurgence of interest in few-shot learning that aims to learn novel concepts from very few labeled examples (Fei-Fei et al.
2006 Vinyals et al. 2016 Wang & Hebert 2016, Snell et al. 2017 Finn et al. 2017).
Existing work tries to solve this problem from the perspective of meta-learning (Thuun 1998.
Schmidhuber (987). which is motivated by the human ability to leverage prior experiences when tackling new task. Unlike the standard machine learning paradigm, where a model is trained on set of exemplars, meta-learning is performed test sets (Vinyals et al., 2016). By sampling small training and test sets from a large collection of labeled examples of base classes, meta- meta-learning based few-shot classification approaches learn to extract task-agnostic knowledge, and apply it to new few-shot learning task of novel classes.
set of tasks, each consisting of its own training and
One notable type of task-agnostic (or meta) knowledge comes from the shared mechanism of data augmentation or hallucination across categories (Wang et al., 2018 Gao et al., 2018 |Scwwartz et al. 2018 Zhang al.. 2018a). Hallucinating additional training data by generating images may seem like an easy solution for few-shot learning, but it is often challenging. In fact, the success of this paradigm is usually restricted to certain domains like handwritten characters (Lake et al.) 2013), requires additional supervision (Dixit et al.) 2017 Zhang et al., (201 8b) or sophisticated heuristics (Hariharan Girshick 2017). An alternative generating raw data in the form of visually realistic images to hallucinate examples in learned feature space (Wang et al., 2018 Gao et al., 2018 Schwwrrzz al. 2018 Zhang et al. 2018a| Xian al., 2019). This can be achieved by, for example, integrating "hallucinator" module into a meta-learnin
framework, where it generates hallucinated examples, guided by real examples (Wang et al., 2018). The learner then uses an augmented training set which includes both the real and the hallucinated examples to learn classifiers. While the existing approaches showed that it is possible to adjust the hallucinator to generate examples that are helpful for classification, the generation process is still far from producing effective samples in the few-shot regime. Our key insight is that, to facilitate data hallucination to improve the performance of new classification tasks, two important requirements should be satisfied: (i) precision: the generated

---

Under review as a conference paper at ICLR 2020
ConvNet
Hallucinator
Learner
Collaborative
Classification
Objective
Objective
(a) Precise hallucinator
(b) Collaborative hallucinator
Figure 1: Illustration of two important properties of our precise collaborative hallucinator, which facilitate data hallucination to improve the performance of classification tasks. (a) Precision: a classifier trained on
hallucinated examples should match the performance of a classifier trained on real examples, demonstrated by shown dark colored shapes
the closeness of their decision boundaries. Real examples and their classifier and solid lines, respectively; hallucinated examples and their classifier are shown as light colored shapes and dashed lines, respectively. (b) Collaboration: all the components need to be trained jointly. In addition to the classification objective imposed on the learner, a collaborative objective is introduced on the hallucinator as direct and early supervision. We integrate these properties into the meta-learning with hallucination framework for few-shot learning.
examples should lead to good classifier performance, and (ii) collaboration: all the components including the hallucinator and the learner need to be trained jointly.
In this work, we propose PrecisE Collaborative hAlluciNator (PECAN), which integrates these requirements into a general meta-learning with hallucination framework, as shown in Figure [] Assume that we have a ballucinator to generate additional examples from the original small training set. A precise hallucinator indicates that classifier trained on both the hallucinated and the few real examples should produce superior validation accuracy. This can be achieved by training the hallucinator end-to-end with the learner, and propaggatg a classification loss based ground- truth labels of validation data [Wang et al., 2018). Since this precision is measured using ground-truth labels, term it as hard precision. And more importantly, if the hallucinator perfectly captures the target distribution, a classifier trained on set of hallucinated examples, despite being generated from a small set of real examples, should produce roughly the same validation accuracy as classifier trained on a large set of real examples, when these two sets are of the same sample size (Shmelkov al.) 2018). This indicates similar level of realism and diversity between the generated and the real examples, as shown in Figure Ta Motivated by this observation, we introduce an additional vvecision-inductnn loss function, which explicitly encourages the hallucinator to generate examples so that classifier trained on them makes predictions similar to the one trained on large amount of real examples. Given that this precision is measured based on classifier predictions, we term it as soft precision. This precision, which is complementary to hard precision and effective, shown in our experiment, is lacking in current approaches (Wang et al. 2018).
Satisfying the precision requirement alone is not sufficient, since the classification objective is still directly associated with the learner, and thus the hallucinator continues to rely on the back- poock-prpagated signal to update its parameters. This leads between the hallucinator and the learner: the learner tends to be stronger and makes allowances for errors in the hallucination, whereas the hallucinator becomes "lazy" and does not make its best effort to capture the data distributions, which is empirically observed in our experiments (See Figure 3) To address this issue, our key insight is to enforce direct and early supervision for the hallucinator, and make its contribution to the overall classification transparent, as shown in Figure [b] Hence, we introduce collaborative objective for the hallucinator, which allows us to directly influence the generation process to favor highly discriminative examples right after hullacination, and to strengthen the cooperation between the hallucinator and the learner.
potential undesirable effect of imbalanced training
Our contributions are three-fold. (1) We propose a novel loss that helps produce precise hallucinated examples, by using the classifier trained on real examples as a guidance, and encouraging the classifier trained on hallucinated examples to mimic its behavior. (2) We introduce a collaborative objective for the hallucinator as early supervision, which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner. (3) By integrating these properties, develop a general meta- meta-learning with hallucination framework, which is model-agnostic and can be combined with any meta-learning models to consistently boost their shot learning performance.
2

---

Under review as a conference paper at ICLR 2020
Here mainly focus on few-shot classification tasks, and we show that approach applies to shw-shot regression tasks as well in the appendix A.7
RELATED WORK
As one of the unsolved problems in machine learning and computer vision, few-shot learning is
attracting growing interest in the deep learning era (Miller al., 2000 Fei-Fei et al.
2006 Lake
jet al.
2013: Santoro et al.] 2016;] Wang & Hebert, [2016] Vinyals et al. [016)] Snell et al.
2017: Finn
al. 2017 Hariharan & Girshick 2017] [ et al. 2017; Trianttiiataiilll et al. 2017 Edwards &
Storkey [2117] Mishra et al. 2018 Douze et al.) 2018 Wang et al. [018]] Chen et 2011a Dvornik
et 2019). Successful generalization from few training samples requires appropriate "inductive
biases" or shared knowledge from related tasks (Baxter 1997), which is commonly acquired through
transfer learning and more recently meta-learning Thrun 1998] Schmidhhuerr 1987 Schmidhuber
et al.] 1997, Bengii et al., [992]. By explicitly "learning- to le-le over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated
task-agnostic knowledge to target few-shot learning problems of novel classes. Within this paradigm of approaches, various types of meta-knowledge has been recently explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using
eistance-basees
classifier such as cosine similarity or nearest neighbor (Koch et al. [0155] Vinyals et al. 20166] Snell
al. 2017: (Sung et al. [2018] Ren et al. 2018 Oreshkin et [2..[[01))))) 8); (2) common initialization of
network parameters (Finn et al. 2011 Nichol & Schulman 2018 Finn et al. 2018) or learned update
rules (Andrychowicz et al.) 2016 Ravi & Larochelle] 2017 Munkhdalai & Yu 2017 Li et al. 2017]
Rusu et 20.19 (3) a transferable strategy to estimate model parameters based on few novel class
examples (Bertinetto et al.) 2016 Qiao et al.) 2018 Qi et al. 2018 Gidaris & Komodakis 20188. or
from an imitial small dataset model (Wang & Hebert 2016 Wang et al.) 2017).
Complementary to these discriminative approaches, our work focuses on synthesizing samples to deal with data scarcity. There has been progress in this direction of data hallucination. either in
pixel or feature spaces (Salakbutdinov et al. 2012 George et al. [017]] Lake al. 2013]
2015
Wong & Yuille 2015; Rezende et al.) [2014] Glodfelloo et al. [2014] Radford et al. 2016 Dixit
et al. 2017| 2017/Hartharr Hariharan Girshick| 2017 Wang et al. 2018 Gao et al. 2018|Schwarrs Schwartz et al.
2018
Zhang et al. (0188). However, it is still challenging for modern generative models to capture the entirety of data distribution (Salimans et al., (016) and produce useful examples that maximally boost the recognition performance (Wang et a [018)), especially in the small sample-size regime. In the context of generative adversarial networks (GANs), Shmelkov et al. (2018) show that images synthesized by state-of-ther ofte--of-theartt approaches, despite their impressive visual quality, are insufficient to tackle recognition tasks, and encourage the use of quantitative measures based on classification results to evaluate GAN models. Rather than using classification results as performance measure, we go step further in this paper by leveraging classification objectives to guide the generation process.
Other related work such as Wang et al. (2018) proposed a general data hallucination framework based on meta-learning, which is a special case of our approach. A GAN-like hallucinator takes a seed example and a random noise vector as input to generate new sample. This hallucinator trained jointly with the classifier in an end-to-end manner. Delta-encoder (Schwartz et al. 2018) is a variant of Wang et al. (018)), where instead of using noise vectors, modifies an ancoder to extract transferable intra-class deformations, i.e., "deltas", and applies them to novel samples to generate new instances. Unlike the above approaches that directly use the produced samples to train the classifier, MetaGAN (Zhang al.) 2018a) trains the classifier in an adversarial manner to augment the classifier with the ability to discriminate between real and synthesized data. Another variant (Gao et al., 2018) explicitly preserves covariance information enable better augmentation. Our work investigates critical yet unexplored properties in this paradigm that the data hallucinator should satisfy. These properties general and can be flexibly incorporated into existing meta-learning approaches and hallucination methods, providing significant gains irrespective of these choices.
META-LEA WITH HALLUCINATION
We begin by presenting the general meta-learning mechanism (Vinyals et al., 2016, Snell et al. 2017: Finn et al. 2017) and meta-learning with hallucination framework for the task of
3

---

Under review as a conference paper at ICLR 2020
Strain
Sample
Strain
gaig
(p.
Sample
SG
Noise
AGG
,
roal
A roal
Stest
Figure 2: Meta-leaming with our precise collaborative hallucinator. In each episode, given initial (sampled) training set Strain we sample its subset Strain. With real seed examples sampled from Strain and noise vector 2, we obtain a set of hallucinated examples Stain through the generator G. Suain and SC are combined to create an augmented training set Strin Conditioning on Stran (Suaiu or Strii).. learner classification or h Icarns a new embedding space and outputs class probabilities p (p or pl"m)) for a set of real test examples Steet- The classification objective Liesrner is a combination of the hard precision and ground-truth labels y) and the soft predsion-induucig loss Warnnr (calculated based prsa and ) The collaborative objective Lnal shares the same formulation of Lierrnrr (i.e.. combination of fhal and (nal), ). but is directly enforced before the embedding layers early supervision for the hallucinator. The hallucinator and the learner trained end-to based on the combination of Cleamer and Chat. Dotted red arrows indicate the flow gradients during back back precppppppggtion...
network d (A
gela learner (classification loss calculated based gare
few-shot image classification. Let be the space of images. We are given two disjoint sets of classes: base class set Crase and unseen novel class set Cnovel- The corresponding base dataset Dbase {[I, 3/.) E I, y4 E Crase } contains a large number of labeled examples per class, while { (I, 5/s) I E I, z/: E ] consists of only small number n of la- beled examples per class. The goal is to learn classifier hgle parametrized by o on Dase that can genssagenrrlizee (Bart & Ullman (205)) to Cnovel even when n as few as one.
the novel dataset 1) novel
Meta-learning aims to achieve such generalization through episodic meta-training that explicitly mimies the few-shot learning scenario on Drase (Vinyals et al. 2016). Specifically, in each episode of the meta-training stage, the meta-learner simulates a few-shot classification task out of D This task constructed by first randomly sampling subset of m classes from Cosser and then randomly sampling a small "training" set Strain (also called the support set) and a small "test" set Steat. (also called the query set). The learner, i.e., the classifier h outputs estimated conditional probabilities p for each example (z, y) in Stest based on Serain That is, p(r) hol (z, Serain). The meta- meta-learner back- pook-propsgaats the gradient of the total classification loss gels- = (...)) loss( (Ir, Strain), v) in Stest to update the learner parameters On. During the meta-testing stage, the resulting hggh is used address the few-shot classification task on Dnovel, which predicts class probabilities of unlabeled test examples conditioned on the given small labeled training set Strain of Cuovel-
Our meta-learning with hallucination framework introduces an additional "hallucinator" module Ge with parameters 0g to augment the small training set Strain- To facilitate training, we follow recent work (Hariharan & Girshick 2017 Wang et al. 2018) and first pre-train a deep convolutional network on Drase using standard cross-entropy loss. We use it to extract the feature representation x E X for an input image I. Meta-lcarning is then performed over the pre-trained features {z,}. As shown in the shaded region in Figure 2 given initial Strain. the hallucinator Ge generates additional examples for each class. Our framework applies to various types of hallucinators, and here consider powerful GAN-like hallucinator in Wang et al. (2018). Each hallucinated example is of the form (Go, (z, 2). y). where (z, y) is a sampled seed example from Strain and 2 is a sampled noise vector. The set of generated examples SC is added to Serain create augmented training set Sang, In the next section, we show how to meta-train Gig. on Crases so that it can hallucinate new examples to augment Strain of Cnovel during meta-testing.
4 PRECISE COLLABORATIVE HALLUCINATOR
We now present PrecisE Collaborative hAlluciNator (PECAN) shown in Figure 2 which exploits two important criteria for useful hallucination: precision and collaboration. As important constraints and guidance, these criteria facilitate hallucination improve the classification performance.
4

---

Under review as a conference paper at ICLR 2020
Basic hallucinator with hard precision. At first, precise hallucinator indicates that classifier trained on Strai should produce superior validation accuracy. We achieve this by training the hallucinator end-to-end with the learner (Wang et al.) 2018). As shown in the shaded region in Figure 2 during each episode of meta-training, the learner module high uses Stram to produce conditional probabilities hggh, (zr, Strinn for each example (ar, v) in the test set Stest. The meta-learner then back- propagates the gradient of the total classification loss gels Σ (a..ess.... (oss (x., Strai)) y) to
update both the learner parameters 9,, and the hallucinator parameters fg.
Soft precision-inducing hallucinator. One of the important characteristics of an optimal generative model is that the generated examples should be indistinguishable from real ones (Goodfellow et al., 2014). We argue that, in terms of our recognition task oriented hallucinator, this means that the classifier trained on hallucinated examples needs to be similar to the classifier trained on real examples. As shown in Figure 2 given an initial relatively large training set Strain which contains n²" examples for each of the m classes, we randomly sample n (n < n²) examples per class, and obtain a subset Strain- From Strain, the hallucinator Go generates n* examples per class S This produces two training sets: Sirain with real examples and SG with hallucinated examples, where both contain the same number of examples. Importantly, note that S is hallucinated from the subset Strain instead of the initial large set Straiu and because n < n", we rule out the trivial identity hallucinator or memorization. We train two additional classification networks: hreal based on Sirain and h based on S both of which have the same architecture as hols, When evaluated on the same test set Stest composed of real examples, comparable performance between hreal and hG shows that the hallucinated samples are sufficiently precise, and as diverse as the real training set. Otherwise, when the hallucinator is imperfect, the accuracy of h will be lower than that of h r
This similarity of classification accuracy essentially measures the difference between the learned (i.e., hallucinated) and the target (i.c., real) distributions, which could serve as an additional supervisory signal for training better hallucinator. Since quantifying the similarity of accuracy directly would be difficult (Hinton et al. 2013), we instead introduce loss function that acts on the network predictions. For an example (Ir, 3) in Stest, the two networks produce conditional probabilities
real (x))(s)) = h1pa Straia) and (((x)) = ngg (Ir, S
respectively. While only the largest entry in p (1)) or (xx) is used to make predictions associated with the ground-truth label y. other entries still carry rich information about the recognition task and the network, as observed in (Hinton et al., [2015] Dvornik et al., 2019). We thus leverage the
probabilities great and for in the absence of the ground-truth label and measure their similarity using the negative cosine distance:
(Freaal po) cos(p po),
where greal and eo are obtained by removing the logit for y in preal and D and re-normalizing the remaining logits using softmax with a learnable temperature. We treat the classification networks hels, real and a the new learner h and use shared parameters for them. Their difference thus lies in different conditional training sets. We obtain the soft precision-inducing loss gree by summing the loss 22 in Stest and then combine it with the hard precision (i.e., the classification) loss as the classification objective. p is now encouraged to not only make the right prediction according to the ground- truth label, but also make similar second-best, third-best, etc., choice predictions as real
Collaboration between hallucinator and learner. We now consider the interaction between the hallucinator G and the learner h. While hallucination is conducted in the pre-trained feature space X, the final classification is performed in new embedding space & learned by the leamer. Since the classification objective is directly imposed on the learner h, the hallucinator G continues to rely on the
signal to update its parameters. We may end up with a good embedding space &b but a poor hallucinator G in the original space X. This undesired effect implies a potential imbalance between the hallucinator and the learner a stronger learner that is able to make allowances for errors in hallucination, but a "lazy" hallucinator that does not make its best effort to capture the data distributions. Indeed, as is empirically validated in the experimental section (see Figure 3), despite being able to match the class distributions in the embedding space o the hallucinated examples are initially pulled away from the class distributions in the feature space X.
To mitigate this issue, we introduce a simple collaborative objective to the hallucinator, which provides an additional constraint regularization on the hallucination process. This collaborative
5

---

Under review as a conference paper at ICLR 2020
objective is the same the above classification objective (i.e., combination of the classification loss and the precision- -inducinn loss), but enforces direct and early supervision for the hallucinator in the pre-trained feature space X. By doing so, we directly influence the update process of the hallucinator parameters, and generate much more discriminative examples right after hallucination than would be the case if we had to rely on gradual pack-propagation from the learner alone. Our objective thus strengthens the cooperation between the hallucinator and the learner for the final classification performance, which can be viewed as a source of deep supervision that introduces auxiliary losses to intermediate layers when training deep neural networks (Simonyan & Zisserman, 2015 Lee et al. 2015). The overall objective combines the classification objective Liearner (on the learner) and the collaborative objective Liai (on the hallucinator), each of which consists of a classification loss øels (hard precision as cross-entropy with respect to ground-truth) and soft precision-indueiie loss ppre;
L(Og, (1)),0n)= Clearner + ACnst gols llearner
+
A
epre learner + A2 + λ epre
(3)
where X, Å1, A2, and Ag are scalar hyper-parameters.
Our hallucinator is general and applies to different types of h (i.e., meta-learning algorithms). Here we focus on the widely used and powerful prototypical networks (PN) (Snell et al.] 2017)) prototype matching networks (PMN) (Vinyals al. 2016 Wang et al. 2018), and cosine classifiers (Cos) (Gidaris& & Komodakis 2018 Chen et al., 201 9a). Without loss of generality, we take PN as an example to explain the overall meta-training and meta- teeta-testing process. PN learns an embedding space o and uses a can-maraettric nearest centroid classifier to assign class probabilities for test example based on its distances from class means in $. As before, in each meta-training episode, after sampling S Strain Strain- and Stest and hallucinating Surain in the pre-trained feature space X, we perform nearest centroid classification and produce the collaborative objective Lhal on Stest- We then feed the examples to the PN learner, obtain their embedded features in L perform nearest centroid classification, and produce the classification objective Liearner on Steet- The final loss is back- precppropaggted to update both the PN learner parameters 0% and the hallucinator parameters Og. Figure 2 shows schematic of the entire process. During meta-testing, we use the resulting Gog to hallucinate new examples augment Strain of Cnovel, and combine the predicted class probabilities in X' and t as the final predictions.
5 EXPERIMENTAL EVALUATION
We explore the use of our meta-learning with hallucination framework for few-shot visual classi- fication tasks. We focus the evaluation the ImageNet based few-shot benehmark (Hariharan & Girshick 2017 Wang et al. 2018 This is one of the largest datasets by far used for few-shot classi- fication and captures more realistic scenarios than others based on handwritten characters (Lake) jet al. 2013) or low-resolution images (Vinyals et al. 2016). The benchmark divides the 1,000 Ima- geNet categories (Russakovsky et al.) 2015) into 389 base classes Chase. with thousands of training images per class, and 611 novel classes Croval, with a small number n of training images per class. Following Haariharnn & Girshick (2017), use Clase to train a convolutional network (ConvNet) based feature extractor and to conduct meta-training. Meta-testing is performed on Cuovel. and the per- formance is evaluated on a held-out test set, i.c., the original validation set of ImageNet. In addition, to avoid over-fitting, both Ciase and Cuovel are further split into two disjoint subsets. 193 of the base and
classes Cgy.. and 300 of the novel classes Caovel are used for cross-validating the remaining 196 base classes C and 311 novel classes Ch are used for the final evaluation. Here we focus on hallucinating novel instances and thus evaluate the performance primarily on the novel classes Chnn Cf which is also consistent with most of the contemporary work (Vinyals et al. 2016 Snell et al. 2017 Finn al. (017). We report the mean top-1 I and top-5 accuracies for 311-way n 1,2, 5, 10, 20-shot classification, with each of them averaged over 5 trials.
In addition to this challenging version of ImageNet, we also evaluate on the widely used minilmageNet (Vinyals et al., (016) dataset to show the generality of our approach. minilmageNet is subset of 100 classes selected randomly from ImageNet with 600 images sampled from each class. Following the data split in Ravi & Larochelle (2017). we use 64 base, 16 validation, and 20 novel classes. We evaluate in the standard 5-way, 1-shot and 5-way, 5-shot settings (Vinyals et al. 2016).

---

Under review as a conference paper at ICLR 2020
Top-1 accuracy
Top-5 accuracy
Method way dlassfication)
n=1
PMN wr POCAN (Ours)
21.3
29.0
39.1
45.3
49.6
47.0
59.1
70.5
75.5
78.7
Meta-kearning motioed
PMN wrd Wang
21.0
69.0
PMN wia
20.3
27.7
57.9
43.7
47.5
44.1
56.4
68.8
74.1
77.2
PMN Wang
19.6
37.5
55.7
68.4
Ph wr
PECAN (Ouas)
205
27.7
57.5
43.9
48.4
40.5
57.5
69.3
74.9
7
Meta learning medhod
PN wr Wang ut all 2018
199
264
41.9
45.9
76.5
Ph wt aus
18.7
25.9
34,9
39.6
42.3
40.2
55.0
66.7
71.6
74.3
PN Snell
2017
17.7
25.3
34.4
39.3
42.1
39,3
54.4
66.3
71.2
73.9
Ca-Cls w/ PDCAN (Ouns)
18.
25.4
33.7
38.
39.2
43.1
53.6
64.8
69.9
71.4
Meta-kearning metioed
CosCCi w/
184
25.1
53.5
37.5
38.6
42.1
52.8
64.0
69.0
70.7
CosCCs Gidaris & Komodakis Che8
Chen etal. 2019
17.7
21.4
25.7
25.4
30,0
41.6
49.2
56.3
60.4
62.7
Othics Mett-kcarning
Cos & Am. Gedrrs & Konodakki 2018
46,0
57.5
69.2
74.8
78.
Bavelines
MN Virvah et al 2016
19.8
25.8
34.8
41.1
46.5
43.6
54.0
66.0
72.5
76.9
MAML
Fina
201
Gro al. 2018
39,2
64.2
76.8
LogRog Wang ct
2018
168
24.9
35.6
42.2
48.0
38.4
51.1
64.8
71.6
76.6
Non meta learaing buselines
LogReg ut Asaloges
Harbana & Giestick, 2017
17.1
23.5
32.5
39.2
48.0
40.7
50.8
02.0
69.3
76.5
Gauasion hallucinatur Wang at 2015
16.7
24.2
33.4
38.2
29.1
SN Koch 2015
38,9
64.6
76.4
Table 1: Top-1 and top-S accuracies (%) on the novel classes for the ImageNet based ri-shot classification benchmark. We use ResNet-10 as the feature extractor. PN: prototypical networks, PMN: prototype matching networks, Cos-Cls: cosine classifiers. Methods with 'w/ G' use a meta-learned hallucinator. Standard deviations for all numbers are of the order of 0.2%. Our PECAN achieves the best performance. Importantly, PECAN is model agnostic and can be combined with different meta-learning models to improve their performance.
Top-1 accuracy
Top-5 accuracy
Method
Tearser
Learner
Sal
hal
n=l
2
5
10
20
n=l
2
5
10
20
PN w/ G+ PECAN
19.9
26.4
35.7
41.9
45.9
45.0
55.9
67.3
73,0
76.5
6.5
10.6
19.5
27.9
35.1
30.5
41.0
55,3
63,8
68.7
V
20.1
26.8
36.7
42.6
47.0
45.3
56.0
67.5
73.3
76.8
20.1
26.6
36.0
42.8
47.6
45.8
56.2
67.6
74.0
77.6
6.8
10.8
19.7
27.9
35.2
31.5
41.4
55.4
63.8
68.7
20.8
27.7
37.5
43.9
48.4
46.5
57.5
69.3
74,9
78.1
Table 2: Ablation on precision and collaboration requirements. igelar; hard precision based on classification loss, .gpres soft precision- induucin loss, 'fnat": collaborative objective imposed on the hallucinator. Different components are complementary to each other.
5.1 RESULTS ON IMAGENET
Implementation details. We mainly use ResNet- architecture (He et al. 2016) as the feature extractor. following Harihaaan & Girshick (2017): Wang et al. (2018). Additionally, we provide results using a deeper ResNet-50 architecture in Section A.3 We extract and record the features, and perform meta-learning by using these pre-computed features. We consider three widely-used,
powerful meta-learning approaches: prototypical networks (PN) (Snell et al.) 2017). prototype matching networks (PMN) (Vinyals et al. 2016 Wang et al. 2018), and cosine classifiers (Cos-Cls) used in Gidaris & Komodakis (2018); Chen et al. (2019a). More implementation details are included in Section
Baselines. First compare with the state-of the-art meta-learning with hallucination method (Wang ctt al., 2018). which is a special case of our approach learned with only the hard precision loss. While Wang et al. (2018) focused on 'PN w/ G' and "PMN w/ G' here consider an additional type of classier with hallucination, 'Cos-Cls w/ G', to show the generality of our work. In addition, we compare with a variety of baselines, including (1) these meta-learning approaches with standard data augmentation techniques (Chen et al. 201 ; (2) data hallucination approaches which are not meta-learned: logistic regression with analogies hallucination (Hariharan & Girshick 2017) and
Gaussian hallucinator (Wang et al. 2018); (3) other recent meta-learning approaches: matching networks (MN) (Vinyals et al.) 2016). aodel-agnostic meta-learnnng (MAML) (Finn al., 2017). and 'cosine classifier & attentive weight generators (Cos & Att)' (Gidaris & Komodakis [01 8); (4) classical few-shot learning approaches: Siamese networks (SN) (Koch et 22.15015); and (5) simple
baselines which are not meta-learned: logistic regression (Hariharan & Girshick 2017). For fair comparison, all these basclines and our approach use the same pre-trained ConvNet backbone.
Comparisons with the state of the art. Table shows that our PECAN consistently outperforms all the baselines by large margins across different scenarios. For this challenging 31 1-way classification, our improvements are of the order of 1% to 2%, while standard deviations for accuracy are of the order of 0. 0.2%. For example, in the case of top-5 accuracy, our 'PN w/ G + PECAN' outperforms 'PN w/ G by 1.5 points for 1 and 1.6 points for 20. Similar trends be observed for 'PMN w/
7

---

Under review as a conference paper at ICLR 2020
G PECAN' and "Cos-Cl Cls w/ G PECAN', and also in the top-1 accuracy regime. This indicates that our approach is general and can work with different meta-learners.
Ablation studies. We conduct series of ablations to evaluate the contribution of each component and different design choices. We use the prototypical network (PN) here due to its fast training speed.
Accuracy (%)
Method
sum1
Matching Networks. VViyaal et
2016
43.56-
0.84
0..31
MAMI. (Conv-4) (Fine al. 2017
48.70+
1.84
63.11+0.92
Prototypical Networks Snell
49.42.1
68.20.10.66
Top-5 accuracy
Reprle
Nichol & Schulman
2018
49.9710.32
6..99
Met--SDD Li ct 2017
50.474 1.87
64.03=0.94
Similarity measure
n=1
2
10
20
MetaGAN
Zhng et al.
2018a
52.71:10.44
68.6310.67
cos(preal real
44.9
55.8
66.8
72.5
76.0
Basclinc++
CCeen ot al
2019a
53.97:00.79
16.16=0.63
Cos & AlL.
(iidaris Komodakis
2018
55.45: 0.89 70.13 20.68
CB(yroal )0)
44.2
53.9
64.5
70.4
73.7
SNATI Mishra et
2018
55.71+ 0.99
68.88+0.92
Relation Networks: [Sung 2018
57.024
71.07.0.69
CE (
44.0 53.9
65.0
70.7
74.1
TADAM Oreshkin 2015
58.50:0.30
76.70=0.30
(p
44.9
55.6
66.9
72.8
76.2
Delta- Encoder (Schwartz
2018
58.7
73.6
Net Chen et
2019
59.14.1 0.86
74.63.0.74
45.0
56.0
67.4
73.3
76.6
LEO |Rosu ct al. 2019
61.76:20.88
77,59=0.12
(KL
44.9
55.6
67.0
72.7
76.2
MetaOptNe: (Loc et al.
2019
62.6410661
78.63=0.46
SKL(
45.0 55.9 67.1
72.8
76.2
SalNet Zhangg et al. 2019
574450088
72.01=0.67
MOGNN-Tansductioon (Kim
2019
76.37
coossfrral, (O)
45.8 56.2 67.6 74.0
77.6
(ResNet-10) Tinn al.
2017
54.8910.89
66.621.083
Table 3: Ablation on choice of similarity measure in the soft precision-inducing induinnn loss. p nd and of; class probabilities of A real and R. respectively. greal and pe: class probabilities in the absence of the ground- truth labels. 'CE': cross- coss-py loss as in knowledge distillation (Hinton al. 20155). 'JS': Sensen-Shannon divergene, 'sKL': symmetric KL-divergence. Our sim- ilarity measure achieves the best performance.
MAML G Wang èi al 2018
56.37-063
68.91=0.57
MAML G + PECAN (Ours)
58.3910.37
71.3620.44
PMN w/ Wang et al 2018
62.28:2053
78.28=0.62
PMN w/ G PECAN (Ours)
63.93::0.00 80.58=0.29
Table 4: Test accuracies (%) on the novel classes for the minilmageNet dataset. 't' indicates 95% confi- dence intervals over tasks. Our PECAN significantly outperforms the state-of-the-art approaches.
Variants of PECAN. PECAN leverages two requirements for the meta-learned hallucinator: precision and collaboration. learner is the basic hallucinator with only the hard precision based on the classification loss. Table shows that each requirement by itself yields performance superior to the basic hallucinator. The soft precision-inducing loss gere consistently helps when combined with the hard precision ge 'feearner + 2pre outperforms 'eils "fearner" and 'gel + epa outperforms 'gels The collaboration objective integrates fearner and Laal to boost the performance: 'feaamer + Gila outperforms
.grls "Learner" Each component is thus essential and complementary to each other, enabling our full PECAN to outperform its variants.
Choice of similarity measure in soft precision-inducine loss. Our precision-inducing loss measures the similarity between classifier predictions preal and p " We used negative cosine distance between the probabilities preal and 5 in the absence of ground-truth labels. Table 3 compares with other types of similarity: variant of negative cosine distance, cross-entropy as in knowledge distillation (Hinton al. 2015), -ennennShannon divergence, and symmetric KL-divergence (Dvornik al. 2019). Our similarity achieves the best performance, and removing the true-class probability consistently helps.
Impact of collaborative objective. Our collaborative objective introduces additional direct and early supervision to train the hallucinator. Table 2 shows quantitatively its contribution to the overall accuracy. Here, we further qualitatively understand its impact though t-SNE visualizations (van der Maaten & Hinton 2008) of the hallucinated examples for novel classes. For ease of analysis, we do not use the precision-indueet inducing loss. Without the collaborative objective, despite being able match the class distributions in the embedding space & (Figure (3b), the hallucinated examples are initially pulled away from the class distributions in the pre-trained feature space X' (Figure (aa). indicating a "lazy" hallucinator. In contrast, the collaborative objective enforces the hallucinator to generate more discriminative examples right after hallucination (Figure Be). leading to improved performance.
Qualitative visualizations. To better understand the hallucination process, Figure 4 shows some examples of classification results for our PECAN and the state-of-the-an baseline (Wang et al. 2018).
8

---

Under review as a conference paper at ICLR 2020
(a) pre-trained space X' w/o CO (b) In embedding space & w/o CO (c) In pre-trained space X w/ CO Figure 3: t-SNE visualizations of hallucinated examples for investigating the impact of collaborative objective (CO) without precision. Seeds shown stars, real examples as crosses, hallucinations triangles. PECAN without CO: (a) in the pre-traine eraitra ResNet-10 feature space (b) in new embedding space learned by PN; PECAN with CO: (c) in the pre-trained ResNet-10 feature space X. Best viewed in color with zoom.
Figure 4: Visual comparisons of top-1 classification results on two representative novel classes between our
PECAN and the state-of-the- art meta meta-lcarned hallucinator (Wang et al. 2018). Top row: bullmastiff; bottom row: American chameleon. Left 3 columns: test images that are correctly classified by both approaches; middle 3 columns: target test images that are misclassified by Wang al. (008 as other classes (the names of the predicted classes by Wang al. (2018) are overlaid the images).
correctly classified by PECAN; right columns: test images from other classes that are misclassified by Wang al.) (2018) as target class, but correctly classified by PECAN. Our approach is able to model a large range of visual variations and diversity. e.g.. bullmastiffss in different poses, and chameleons in different viewpoints and background, whereas Wang et al. (201) is confused by visually similar classes.
5.2 RESULTS ON miRilMAGENET
To show the generality of our approach, we further evaluate on minilmageNet. We use a ResNet- 10 architecture and focus incorporating our hallucinator into the metric-leammig--assee meta-learning approach, prototype matching networks (PMN) (Wang et al.) (2018), and the optimization-ban meta-learning approach, MAML (Finn al. 2017). For MAMI, in each meta-training episode, we sample a batch of few-shot classification tasks. For each of the tasks, we sample training sets Sirin and Strain, sample a test test Sustt hallucinate SG Strain and obtain an augmented training set Saug Strinn In the MAML inner loop, for each task, conditioning on its Staag (S or Strain) we adapt the parameters of cls (ht or reea)) using few gradient updates. For each task, the adapted cl (h?? or heall) is evaluated the corresponding Susst- In the MAML outer loop, average the classification objective on Stest across the batch of tasks. In a similar way, we compute the collaborative objective in the pre-trained feature space. The final loss is used update the initial MAMI. model and the hallucinator. From Table 4 our PECAN significantly outperforms all these state-of- state-of-the-at competitors, including other hallucination based approaches such as MetaGAN (Zhang al. 2018a), delta-encoder (Schwartz et al. 2018). IDeMe-Net (Chen et al. 2019b). and SalNet (Zhang et al. 2019). Our superior performance over MetaGAN, a GAN-based approach hallucinate data, shows that directly matching the classification performance is more desirable than matching the data distribution between hallucinated and real examples for recognition tasks. Our generic framework can be combined with more recent meta-learning methods, such as LEO (Rusu et al. 2019) and MetaOptNet-SVe (Lee et al. 2019). for further improvement.
CONCLUSION
We have presented an approach to few-shot classification that uses precise collaborative hallucinator to generate additional examples. Our hallucinator integrates two important requirements that facilitate data hallucination in way that most improves the classification performance, and is trained end-to- end through meta-learning. The extensive experiments demonstrate our state-of-the-an performance on the challenging ImageNet and miniImaeeeet based few-shot benchmark in various scenarios.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-398999 2016.
Evgeniy Bart and Shimon Ullman. Cross-generalization: Learning novel classes from a single example by feature replacement. In CVPR, 2005.
Jonathan Baxter. A Bayesian/infoommtiinn theoretic model of learning to learn via multiple task sampling. Machine Learning, 28(1):7-39, 1997.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, 1992.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In Advances in Neural Information Processing Systems, pp. 523-531, 2016.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In ICLR, 2019a.
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta-networks for one-shot learning. In CVPR, 2019b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Mandar Dixit, Roland Kwitt, Mare Niethammer, and Nuno Vasconcelos. AGA: AttributeGuided Augmentation. In CVPR, 2017.
Matthijs Douze, Arthur Szlam, Bharath Hariharan, and Hervé Jégou. Low-shot learning with large-scale diffusion. In CVPR, 2018.
Nikita Dvoriik, Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. arXiv preprint 1903:19903.1341 2019.
Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 2006.
Michael Fink. Acquiring a new class from few examples: Learning recurrent domain structures in humans and machines. PhD thesis, The Hebrew University of Jerusalem, 201
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning, In NeurIPS. 2018.
Hang Gao, Zheng Shou, Alireza Zarcian, Hanwang Zhang, and Shih-Fu Chang. Low-shot learning via covariance-prees prreeerving adversarial augmentation networks. In NeurlPS, 2018.
Dileep George, Wolfgang Lehrach, Ken Kansky, Miguel Lázaro-Gredilla. Christopher Laan, Bhaskara Marthi, Xinghua Lou, Zhaoshi Meng, Yi Liu, Huayan Wang, Alex Lavin, and D. Scott Phoenix. A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAS. Science, 2017.
Spyros Gidaris and Nikos Komodakis, Dynamic few-shot visual learning without forgetting. In CVPR, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint Wiiv:1503.0253:3 2015.
Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang Yoo. Edge-labeling graph neural network for few-shot learning. In CVPR, 2019.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhudtinov. Siamese neural networks for one-shot image recognition. In JCML Deep Learning Workshop, 2015.
Brenden M Lake, Ruslan Salakhutdinov, and Josh B Tenenbaum. One-shot learning by inverting a compositional causal process. In NIPS, 2013.
Brenden Lake, Ruslan Salakhutdinov, and Joshua Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
Chen-Yu Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply-supervised nets. In A/STATS, 2015.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-SGD: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 2017.
Erik G Miller, Nicholas E Matsakis, and Paul A Viola. Learning from one example through shared densities transforms. In CVPR, 2000.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learning. In ICLR, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In JCML, 2017.
Alex Nichol and John Schulman. Reptile: A scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In CVPR, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. CVPR, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICIR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211-25) 2015.
11

---

Under review as a conference paper at ICLR 2020
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski. Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Ruslan Salakhutdinov, Josh Tenenbaum, and Antonio Torralba. One-shot learning with a hierarchical nonparametric Bayesian model. Unsupervised and Transfer Learning Challenges in Machine Learning, 2012.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory memory-augmented neural networks. In ICML, 2016.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The mela-melu-... hook. Diploma thesis, Institul f. Informatik, Tech. Univ. Munich, 1987.
Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive Levin search, and incremental seff-imprrvement. Machine Learning. 28(1): 105-130, 1997.
Lauren A Schmidt. Meaning and compositionality as statistical induction of categories and con- straints. PhD thesis, Massachusetts Institute of Technology, 2009.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. In NeurlPS, 2018.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my GAN? In ECCV, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
Sebastian Thrun Lifelong learning algorithms. Learning learn, 8:181-209, 1998.
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. In NIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 9:2579-2605. 2008.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukeuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In ECCV, 2016.
Yu-Xiong Wang. Deva Ramanan, and Martial Hebert. Learning model the tail. In NIPS, 2017.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018.
Alex Wong and Alan I. Yuille. One shot learning via compositions of meaningful patches. In /CCV, 2015.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-VAEGAN-D2. A feature generating framework for any-shot learning. In CVPR, 2019.
12

---

Under review as a conference paper at ICLR 2020
Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via valiency-guidee hallucina- tion of samples. In CVPR, 2019.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. MetaGAN: An adversarial approach to few-shot learning. In NeurlPS, 2018a.
Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Josh Tenenbaum, Bill Freeman, and Jiajun Wu. Learning to reconstruct shapes from unseen classes. In NeurlPS, 2018b.
Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In CVPR, 2014.
A
APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
The embedding architecture of PN is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0.01, and a Euclidean distance similar to Snell et al. (2017) The embedding architecture of PMN is composed of a one-layer bi-directional LSTM and attention LSTM, and a cosine distance as in (Vinyals et al. 2016 Wang et al. (2018). The embedding architecture of Cos-Cls is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0.01 and an additional one-layer MLP without nonlinearity, and cosine distance with learnable temperature similar to Gidaris & Komodakis (2018). The initial value of the temperature 100. The classification weight vector is estimated by averaging the feature vectors of the training examples for each class. The hallucinator G is threc-layer MLP with Ieaky ReLU nonlinearity of slope 0.01, with its parameters initialized to block diagonal identity matrices (Wang et al. 2018). The dimensionality of the hidden layers is 512 for
ResNet- 10 features and 2,048 for ResNet-50 features. featurre..
Top-5 accuracy
Novel classes
Top-5 accuracy (%)
85
Precision inducing
v=
2
5
10
20
|| mual m ")l
45.3 56.0 67.6 73.2 76.6
75
- cos(greo1,pp (Ours) 45.8 56.2 67.6 74.0 77.6
65
Table A.1: Additional analysis of the soft loss in the case PN on the ImageNet based r--hoo classification bench- mark. and A &. 5nmml and p.: class probabilities of A real in the absence of the ground-truth -trth la- bels, respectively. m real and m. class means of real and hallucinated examples, respectively. Our generic similarity generalizes significantly better than the specific similarity applicable PN for novel calsses.
PN G PECAN
PN w/ G
5 10 20
of examples novel class Figure A.1: Top-5 accuracy (%) the novel classes for the ImageNet based n-shot classification benchmark. We use ResNet-50 as the feature extractor. PN: prototypical networks. Methods with "w/ G' use meta-lcarned hallucinator. With a deeper network, all accuracies are higher, and our PECAN significantly outperforms the basclines.
Both the baselines (PN/PMN/Cos- Cls and PN/PMIN/Cos- Cls with hallucination) and our approach metan-tainee on ImageNet for 60,000 episodes by SGD with an initial learning rate 0.05 for PN/PMN and 0.005 for Cos-Cls, decayed by a factor of 10 every 20,000 episodes. In each episode of the meta-training stage, we sample all the base classes following Wang al. (2018), which found that it is advantageous to use more classes rather than fewer. We sample n 20 examples per class from the base dataset Draser leading to an initial training set Strain Consistent with Wang al.) (2018), to make single hallucinator robust to different sample sizes, we randomly sample different sized examples per class from Sigiin from seed example is sampled from Strain. and fed into the hallucinator G with different noise vectors to generate 20 examples per class as SC S Hallucinated examples are sampled from SC S and added to Strain until there are exactly 20 examples per class in Smang For the test set Stest- have 5 random examples per class for prototypical networks (PN) (Snell et al.) (017)) and cosine classifiersCoos-Ci (Gidaris & Komodakis 2018), and random example per class for prototype matching networks (PMN) VVnyaals et al.) 2016 Wang et al. 2018).
15 examples per class, to obtain Strein. One random
For the soft ceerisionninducinn loss, we use softmax with temperature to produce conditional class probabilities greal and 80. This temperature is a learnable parameter. is shared between greal and PC but is not shared between Liearner and Lhat-
13

---

Under review as a conference paper at ICLR 2020
Novel classes
Novel classes
80
21=0.005
80
22=0.001
70
21=0.01
21=0.02
70
22=0.002
60
21=0.05
60
X2=0.005
Top-5 accuracy
50
21=0.11
Top-5 accursio
11=00.2
40
A2=0.01
#1=00.5
A2=0.02
30
#11=1.0
30
21=2.00
20
22=0.05
21=5.0
22=0.1
10
21=10.0
10
21=25.0
22=0.2
n=20
(a) Å
(b) X
Novel classes
80
23=0.001
Top-5 accuracy (%)
70
60
23=0.01
23=0.002
23=0.005
B1-1
50
73=0.02
Top- soccuma
23=0,,5
40
23=0.1
30
23=0.2
23=0.5
23=1
10
23=2
23=5
n=
n=
23=10
# hallucinated examples, / novell class
(c) Aa
(d) Number of hallucinated examples class
Figure A.2: Top-5 accuracy (%) of additional hyper-parameter analysis the novel classes for the ImageNet based r--hoot classification benchmark. This analysis conducted on 'NN w/ PECAN" with ResNet- 10. (a)()(c) provide sensitivity analysis of the hyper-parameters λı, A2, and X3 in the overall objective. The performance of our PECAN is stable over wide range of hyper- parameaeeeterr values. (d) shows how performance gradually increases and then saturates more examples hallucinated for 1-shot classification.
The hyper-parameters obtained by cross-validation on ImageNet are: A==0.5, =0=0.0 and Ag=0.1 for PN with ResNet-10 10; =1=0.2, Ag=0.0001, and A3=0.05 for PN with ResNet-50; A1=0.4, X2=2.0, and A3=0.1 for PMN with ResNet-100 10; and =055, =,=00000001, and A3=0.005 for the cosine classifier with ResNet-10.
During the meta-testing stage, following Wang et 2018 we use n=1, 2, 5, 10 20 examples per class from the novel dataset Duovel, and then hallucinate a fixed number of additional examples for each novel class. By rross-validation, the number hallucinated examples per class is set to for PN, Cos-Cls and Cos-Cls w/ G PECAN with ResNet-10 10, 8 for Cos-Cls w/ G with ResNet-10, 5 for PN with ResNet-50, and 20 for PMN with ResNet- 10. We combine the classifier prediction results in the pre-trained feature space and the learned embedding space using scalar pareaeparrmeter.. By cross-validation, this hyper-parameter is set to 0.05 for PN with ResNet- 10, 0.07 for PN with ResNet-50, for PMN with ResNet-10, and 0.00002 for the cosine classifier with ResNet-10. For the test set Steat following Wang et al. (2018) we have 50 real examples per class, and we average the top-1 top-5 accuracy of them over the novel classes.
A.2 PERFORMANCE ON BASE CLASSES
While we significantly improve the classification performance on novel classes, our approach remains accurate on base classes. For example, for prototypical networks (PN) (Snell et al.) 2017). both our 'PN w/ G + PECAN' and the baseline 'PN w/ G' achieve the same top-5 accuracy 92.4% on ImageNet.
A.3 IMPACT OF DEEPER REPRESENTATION MODELS
Figure A.I shows the results on ImageNet using features from a ResNet-50 architecture. As expected, deeper networks result in better performance for all the approaches, but PECAN hallucination
14

---

Under review as a conference paper at ICLR 2020
strategy still provides large gains across the board over the state-of-the-an meta- learnee hallucinator in (Wang et al., 2018).
A.4 ANALYSIS OF HYPER-PARAMETRR SENSITIVITY
Hyper-parameters in the overall objective. We conduct sensitivity experiments for the hyper- parameters X1 X2, and A3. which trade off different loss components in the overall objective of our PECAN. We vary one of the three cross-validated values. Figures A.2a A.2b And the top-5 accuracy of 'PN w/ G + PECAN on the novel classes for the ImageNet based n-shot classification benchmark. We can see that the top-5 accuracy is stable over wide range of hyper-parameter values, for example when the value of A becomes 50 times larger or 100 times smaller than \ used in the main paper. Across the board, our PECAN consistently and significantly outperforms the baselines shown the main paper.
while fixing the remaining two to their
Number of hallucinated examples. We also show how the top-5 accuracy changes for n 1-shot classification with respect to the number hallucinated images in Figure A.2d, We can see that when the number of hallucinated examples is changed from 0 to 10, the performance of our PECAN gradually improves, and then saturates and drops slightly with more than 10 images generated.
A.5 ADDITIONAL ANALYSIS OF SOFT PRECISION-INDUCING LOSS
Our soft precision-inducing loss measures the similarity between classifier predictions greal and p² This is general similarity measure which applies various types of classifiers. including parametric and non- non-parametric classifiers. For prototypical networks (PN) (Snell et al. (017). a non-parametric nearest centroid classifier is used to assign class probabilities for test example based on its distances from class means. Hence, in this special case, to measure the similarity between classifier predictions, we can directly calculate the distance between the mean of real examples reaa and the mean of hallucinated examples m Table A.1 compares our generic similarity with this specific similarity for PN on the ImageNet based n-shot classification benchmark. The result shows that our similarity generalizes significantly better for novel classes.
A.6 ADDITIONAL VISUALIZATIONS OF CLASSIFICATION RESULTS
Similar to Figure 5 in the main paper, here we provide more examples of classification results for our PECAN and the state-of-the-ard meta-learned hallucinator (Wang et al. 2018) in Figure
A.3 A.3
A.7 EXTENSION TO FEW-SHOT REGRESSION
In the main paper, we focus on few-shot classification tasks. However, our approach general and applies to few-shot regression tasks well. When extending our approach to address regression tasks, we need to slightly modify the design of the hallucinator G. Specifically, in classification tasks, the hallucinator G takes input seed example (I., v) and outputs hallucinated example (x', .y) (G =), y). where the class label y' = y. indicating that the hallucinated example belongs the same class as the seed example. In contrast, in regression tasks, y becomes a continuous quantity. We thus cannot enforce the constraint y' = y. To address this issue, we modify the hallucinator G to directly generate the tuple (x',4)) (/)= G (Ir y, z). This is achieved by the concatenation of a and y as input to G.
We incorporate our hallucinator with MAML Finn et 2017) and evaluate on the n 5-shot sinusoidal regression task proposed in Finn et al. (2017). Each task regresses from the input to the output of sine curve, and different tasks differ in the amplitude and phase of the sinusoid. The amplitude and the phase are uniformly distributed within 10.1, 5.0] and [0. x1), respectively. During meta-training and meta-testing, the input datapoints æ of each task are uniformly sampled from [-5.0, 5.0]. The prediction loss is measured by the mean squared error (MSE) between the prediction and true value. The regressor is a feedforward neural network with 2 hidden layers of size 40 with ReLU nonlinearity. We use MAML (Finn et al. 2017) with one gradient update based on n 5 examples with a fixed step size a = 0.01, and use Adam as the meta-optimizer (Kingma & Welling 2014).
15

---

Under review as a conference paper at ICLR 2020
ALAIOA
Sk
Splmy Coonle
Salty Soletee
ed brcca
sourrrl monkey
bornet
projectile
chocolate SOurCe
hot pot
Figure A.3: Additional visual comparisons of top-1 classification results on four representative novel classes between our PECAN and the state-of-the-arl meta-learned hallucinator (Wang et al. (2018). From top bottom Dungeness crab, king penguin, container ship, and polpie. Left 3 columns: test images that correctly classified by both approaches; middle columns: target test images that are misclassified by Wang al. (2018) as other classes (the names of the predicted classes by Wang et al. (2018) are overlaid on the images), but correctly classified by PECAN; right 3 columns: test images from other classes that are misclassified by Wang al. (2018) the target class, but correctly classified by PECAN. Our approach is able model large range of visual variations and diversity, whereas Wang al. (2018) is confused by visually similar classes.
Method
mean squared error (MSE) n=5
MAML (Finn et al. 2017)
0.84
MAML G (Wang al. 2018
0.67
MAML w/ G + PECAN (Ours)
0.52
Table A.2: Comparison on the n 5-shot sinusoidal regression task in Finn et al. (017)) Our PECAN general and outperforms the baselines for the regression task as well.
We compare with two baselines: (1) standard MAML, and (2) 'MAML w/ G', which is MAML with the hallucinator in (Wang al. 2018). While Wang al. (2018) focus classification tasks, here we extend the use of its hallucinator in the similar way as we discussed before. Table A.2 shows that our PECAN consistently outperforms the basclines for the regression task as well, indicating the generality of our approach.
A.8 VISUALIZZATIONS OF HALLUCINATED EXAMPLES
Our hallucination is performed the pre-trained feature space, and visualizing them directly is not intuitive. In addition to t-SNE visualization of hallucinated samples in Figure 3 here in Figure A.4 we include an additional visualization of hallucinated examples in the pixel space, using the nearest neighbor real image in the feature space, corresponding to each hallucinated feature sample.
16

---

Under review as a conference paper at ICLR 2020
Figure A.4: Hallucinated example visualization for 5 novel classes. Left: t-SNE visualization of hallucinated examples in the feature space. Right: The single image framed in blue is the sampled seed example. All other images represent the hallucinated examples which are visualized by using their nearest neighbor real image in the feature space.
17


Reviews:
[{"review_id": "rJl3L37nYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a general meta-learning with hallucination framework called PECAN. It is model-agnostic and can be combined with any meta-learning models to consistent boost their few-shot learning performance. \n\nThere are two key points for the proposed model. On the one hand, the authors introduce a novel precision-inducing loss which encourages the hallucinator to generate examples so that a classifier trained on them makes predictions similar to the one trained on a large amount of real examples. On the other hand, the authors introduce a collaborative objective for the hallucinator as early supervision, which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner.\n\nOn the whole, the paper is well-written, and the proposed idea is novel and interesting.\n\nI have some following major concerns about the paper:\n(1) In Figure 2, the authors first sample the training set S^*_{train}, which contains n^* examples for each of the m classes, and then they randomly sample n examples per class, and obtain a subset S_{train}. Why not generate the S_{train} directly and then measure your precision-inducing loss over the real set S_{train} and S^G_{train}? I hope the authors explain it in their paper.\n(2) For Function 2 in the paper, why compute the cosine distance on the probability vectors that are obtained by removing the logit for ground-truth label in original probability distributions? Could we compute the distance on the probability vectors that contains the logit for ground-truth label? I hope the authors explain it in detail.\n(3) As far as I know, there are some latest work on few-shot learning in 2019, especially the work “Few-shot Learning via Saliency-guided Hallucination of Samples” and “Edge-Labeling Graph Neural Network for Few-shot Learning”. I hope the authors can compare with these two methods to further demonstrate the effectiveness of the proposed model.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJlzWIKX5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method that builds upon the work of Wang et al. It meta-learns to hallucinate additional samples for few-shot learning for classification tasks. Their two main insights of this paper are to propose a soft-precision term which compares the classifiers' predictions for all classes other than the ground truth class for both a few-shot training set and the hallucinated set and b) to introduce the idea of applying direct early supervision in the feature space in which the hallucination is conducted in addition to in the classifier embedding space. This allows for stronger supervision and prevents the hallucinated samples from not being representative of the classes. The authors show small, but consistent improvement in performance on two benchmarks: ImageNet and miniImageNet with two different network architectures versus various state-of-the-art meta-learning algorithms with and without hallucination. The authors have adequately cited and reviewed the existing literature. They have also conducted many experiments (both in the main paper and in the supplementary material) to show the superior performance of their approach versus the existing ones. Furthermore their ablation studies both for the type of soft precision loss and for their various individual losses are quite nice and thorough. \n\nOverall the contribution of this paper is incremental over Wang et al and is mainly in the introduction of their new loss terms to regularize the hallucination process. This is clearly evident from Table 2 (comparing rows 1 and 3), where much of the performance gain is attained by including the l_learner^cls term versus the collaborative loss term (comparing row 1 and row 4).\n\nFurthermore, I would like the author to answer the following two questions:\n\n1. The authors claim that their method is general applicable to all meta-learning methods and can be combined with them. Yet, the meta learning methods that they apply it to: prototypical networks, prototypical matching networks and cosine classifiers are all metric-learning-based meta-learning techniques. I would like the authors to outline the procedure (and preferably also show experiments) for applying their proposed technique to meta-learning based techniques that do not involve learning a metric-embedding space and instead learn the learning procedure via nested optimization, e.g. MAML and its variants.\n\n2. In Table 4, I would like to see the results of PNM w/G or in other words the results of (Wang et al, 2018)'s method in comparison to the authors' proposed method.\n\n3. The authors make no attempt to solve the problem of hallucinating examples for regression tasks. This is fine as it is perhaps outside he scope of their current work. However, I would like the authors to fully qualify their claims everywhere in the paper and restrict the contribution of their work to classification tasks only.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkeCv6fj_B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors address few-shot learning via a precise collaborative hallucinator. In particular, they follow the framework of (Wang et al., 2018), and introduce two kinds of training regularization. The soft precision-inducing loss follows the spirit of adversarial learning, by using knowledge distillation. Additionally, a collaborative objective is introduced as middle supervision to enhance the learning capacity of hallucinator. \n\nHere are some comments:\n1. The novelty is relatively limited. The idea of hallucinating has been mainly introduced in (Wang et al., 2018). The soft precision-inducing loss is a straightforward extension of knowledge distillation (Hinton et al., 2015). \n\n2. It is not quite clear about how to use the collaborative objective on the hallucinator. Fig.2 and the text in ' Collaboration between hallucinator and learner ' (page 5) are not quite informative. Especially, how to perform the soft precision-inducing loss (l_hal^pre) for hallucinator?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a new approach to meta-learning with generating new useful examples.

The reviewers liked the paper but overall felt that the paper is not ready for publication as it stands.

Rejection is recommended. 
Decision: Reject