Year: 2019

Abstract:
Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks, which aim to learn novel concepts from very few examples. The hallucination process, however, is still far from generating effective samples for learning. In this work, we investigate two important requirements for the hallucinator — (i) precision: the generated examples should lead to good classifier performance, and (ii) collaboration: both the hallucinator and the classification component need to be trained jointly. By integrating these requirements as novel loss functions into a general meta-learning with hallucination framework, our model-agnostic PrecisE Collaborative hAlluciNator (PECAN) facilitates data hallucination to improve the performance of new classification tasks. Extensive experiments demonstrate state-of-the-art performance on competitive miniImageNet and ImageNet based few-shot benchmarks in various scenarios.

heading: 1 INTRODUCTION
text: Modern deep learning models rely heavily on large amounts of annotated examples (Deng et al., 2009). Their data-hungry nature limits their applicability to real-world scenarios, where the cost of annotating examples is prohibitive, or they involve rare concepts (Zhu et al., 2014; Fink, 2011). In contrast, humans can grasp a new concept rapidly and make meaningful generalizations, even from a single example (Schmidt, 2009). To bridge this gap, there has been a recent resurgence of interest in few-shot learning that aims to learn novel concepts from very few labeled examples (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang & Hebert, 2016; Snell et al., 2017; Finn et al., 2017).
Existing work tries to solve this problem from the perspective of meta-learning (Thrun, 1998; Schmidhuber, 1987), which is motivated by the human ability to leverage prior experiences when tackling a new task. Unlike the standard machine learning paradigm, where a model is trained on a set of exemplars, meta-learning is performed on a set of tasks, each consisting of its own training and test sets (Vinyals et al., 2016). By sampling small training and test sets from a large collection of labeled examples of base classes, meta-learning based few-shot classification approaches learn to extract task-agnostic knowledge, and apply it to a new few-shot learning task of novel classes.
One notable type of task-agnostic (or meta) knowledge comes from the shared mechanism of data augmentation or hallucination across categories (Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018; Zhang et al., 2018a). Hallucinating additional training data by generating images may seem like an easy solution for few-shot learning, but it is often challenging. In fact, the success of this paradigm is usually restricted to certain domains like handwritten characters (Lake et al., 2013), or requires additional supervision (Dixit et al., 2017; Zhang et al., 2018b) or sophisticated heuristics (Hariharan & Girshick, 2017). An alternative to generating raw data in the form of visually realistic images is to hallucinate examples in a learned feature space (Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018; Zhang et al., 2018a; Xian et al., 2019). This can be achieved by, for example, integrating a “hallucinator” module into a meta-learning framework, where it generates hallucinated examples, guided by real examples (Wang et al., 2018). The learner then uses an augmented training set which includes both the real and the hallucinated examples to learn classifiers. While the existing approaches showed that it is possible to adjust the hallucinator to generate examples that are helpful for classification, the generation process is still far from producing effective samples in the few-shot regime. Our key insight is that, to facilitate data hallucination to improve the performance of new classification tasks, two important requirements should be satisfied: (i) precision: the generated
examples should lead to good classifier performance, and (ii) collaboration: all the components including the hallucinator and the learner need to be trained jointly.
In this work, we propose PrecisE Collaborative hAlluciNator (PECAN), which integrates these requirements into a general meta-learning with hallucination framework, as shown in Figure 1. Assume that we have a hallucinator to generate additional examples from the original small training set. A precise hallucinator indicates that a classifier trained on both the hallucinated and the few real examples should produce superior validation accuracy. This can be achieved by training the hallucinator end-to-end with the learner, and back-propagating a classification loss based on groundtruth labels of validation data (Wang et al., 2018). Since this precision is measured using ground-truth labels, we term it as hard precision. And more importantly, if the hallucinator perfectly captures the target distribution, a classifier trained on a set of hallucinated examples, despite being generated from a small set of real examples, should produce roughly the same validation accuracy as a classifier trained on a large set of real examples, when these two sets are of the same sample size (Shmelkov et al., 2018). This indicates similar level of realism and diversity between the generated and the real examples, as shown in Figure 1a. Motivated by this observation, we introduce an additional precision-inducing loss function, which explicitly encourages the hallucinator to generate examples so that a classifier trained on them makes predictions similar to the one trained on a large amount of real examples. Given that this precision is measured based on classifier predictions, we term it as soft precision. This precision, which is complementary to hard precision and effective, as shown in our experiment, is lacking in current approaches (Wang et al., 2018).
Satisfying the precision requirement alone is not sufficient, since the classification objective is still directly associated with the learner, and thus the hallucinator continues to rely on the back-propagated signal to update its parameters. This leads to a potential undesirable effect of imbalanced training between the hallucinator and the learner: the learner tends to be stronger and makes allowances for errors in the hallucination, whereas the hallucinator becomes “lazy” and does not make its best effort to capture the data distributions, which is empirically observed in our experiments (See Figure 3). To address this issue, our key insight is to enforce direct and early supervision for the hallucinator, and make its contribution to the overall classification transparent, as shown in Figure 1b. Hence, we introduce a collaborative objective for the hallucinator, which allows us to directly influence the generation process to favor highly discriminative examples right after hallucination, and to strengthen the cooperation between the hallucinator and the learner.
Our contributions are three-fold. (1) We propose a novel loss that helps produce precise hallucinated examples, by using the classifier trained on real examples as a guidance, and encouraging the classifier trained on hallucinated examples to mimic its behavior. (2) We introduce a collaborative objective for the hallucinator as early supervision, which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner. (3) By integrating these properties, we develop a general meta-learning with hallucination framework, which is model-agnostic and can be combined with any meta-learning models to consistently boost their few-shot learning performance.
Here we mainly focus on few-shot classification tasks, and we show that our approach applies to few-shot regression tasks as well in the appendix A.7.

heading: 2 RELATED WORK
text: As one of the unsolved problems in machine learning and computer vision, few-shot learning is attracting growing interest in the deep learning era (Miller et al., 2000; Fei-Fei et al., 2006; Lake et al., 2015; Santoro et al., 2016; Wang & Hebert, 2016; Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Hariharan & Girshick, 2017; George et al., 2017; Triantafillou et al., 2017; Edwards & Storkey, 2017; Mishra et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019a; Dvornik et al., 2019). Successful generalization from few training samples requires appropriate “inductive biases” or shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning and more recently meta-learning (Thrun, 1998; Schmidhuber, 1987; Schmidhuber et al., 1997; Bengio et al., 1992). By explicitly “learning-to-learn” over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task-agnostic knowledge to target few-shot learning problems of novel classes. Within this paradigm of approaches, various types of meta-knowledge has been recently explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using a distance-based classifier such as cosine similarity or nearest neighbor (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol & Schulman, 2018; Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Li et al., 2017; Rusu et al., 2019); (3) a transferable strategy to estimate model parameters based on few novel class examples (Bertinetto et al., 2016; Qiao et al., 2018; Qi et al., 2018; Gidaris & Komodakis, 2018), or from an initial small dataset model (Wang & Hebert, 2016; Wang et al., 2017).
Complementary to these discriminative approaches, our work focuses on synthesizing samples to deal with data scarcity. There has been progress in this direction of data hallucination, either in pixel or feature spaces (Salakhutdinov et al., 2012; George et al., 2017; Lake et al., 2013; 2015; Wong & Yuille, 2015; Rezende et al., 2014; Goodfellow et al., 2014; Radford et al., 2016; Dixit et al., 2017; Hariharan & Girshick, 2017; Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018; Zhang et al., 2018a). However, it is still challenging for modern generative models to capture the entirety of data distribution (Salimans et al., 2016) and produce useful examples that maximally boost the recognition performance (Wang et al., 2018), especially in the small sample-size regime. In the context of generative adversarial networks (GANs), Shmelkov et al. (2018) show that images synthesized by state-of-the-art approaches, despite their impressive visual quality, are insufficient to tackle recognition tasks, and encourage the use of quantitative measures based on classification results to evaluate GAN models. Rather than using classification results as a performance measure, we go a step further in this paper by leveraging classification objectives to guide the generation process.
Other related work such as Wang et al. (2018) proposed a general data hallucination framework based on meta-learning, which is a special case of our approach. A GAN-like hallucinator takes a seed example and a random noise vector as input to generate a new sample. This hallucinator is trained jointly with the classifier in an end-to-end manner. Delta-encoder (Schwartz et al., 2018) is a variant of Wang et al. (2018), where instead of using noise vectors, it modifies an auto-encoder to extract transferable intra-class deformations, i.e., “deltas”, and applies them to novel samples to generate new instances. Unlike the above approaches that directly use the produced samples to train the classifier, MetaGAN (Zhang et al., 2018a) trains the classifier in an adversarial manner to augment the classifier with the ability to discriminate between real and synthesized data. Another variant (Gao et al., 2018) explicitly preserves covariance information to enable better augmentation. Our work investigates critical yet unexplored properties in this paradigm that the data hallucinator should satisfy. These properties are general and can be flexibly incorporated into existing meta-learning approaches and hallucination methods, providing significant gains irrespective of these choices.

heading: 3 META-LEARNING WITH HALLUCINATION
text: We begin by presenting the general meta-learning mechanism (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017) and our meta-learning with hallucination framework for the task of
few-shot image classification. Let I be the space of images. We are given two disjoint sets of classes: a base class set Cbase and an unseen novel class set Cnovel. The corresponding base dataset Dbase = {(Ii, yi) , Ii ∈ I, yi ∈ Cbase} contains a large number of labeled examples per class, while the novel dataset Dnovel = {(Ii, yi) , Ii ∈ I, yi ∈ Cnovel} consists of only a small number n of labeled examples per class. The goal is to learn a classifier hclsθh parametrized by θh on Dbase that can cross-generalize (Bart & Ullman, 2005) to Cnovel even when n is as few as one. Meta-learning aims to achieve such generalization through episodic meta-training that explicitly mimics the few-shot learning scenario on Dbase (Vinyals et al., 2016). Specifically, in each episode of the meta-training stage, the meta-learner simulates a few-shot classification task out of Dbase. This task is constructed by first randomly sampling a subset of m classes from Cbase, and then randomly sampling a small “training” set Strain (also called the support set) and a small “test” set Stest (also called the query set). The learner, i.e., the classifier hclsθh , outputs estimated conditional probabilities p for each example (x, y) in Stest based on Strain. That is, p(x) = hclsθh (x, Strain). The meta-learner back-propagates the gradient of the total classification loss `cls = ∑ (x,y)∈Stest loss(h cls θh (x, Strain), y) in Stest to update the learner parameters θh. During the meta-testing stage, the resulting hclsθh is used to address the few-shot classification task on Dnovel, which predicts class probabilities of unlabeled test examples conditioned on the given small labeled training set Strain of Cnovel. Our meta-learning with hallucination framework introduces an additional “hallucinator” module GθG with parameters θG to augment the small training set Strain. To facilitate training, we follow recent work (Hariharan & Girshick, 2017; Wang et al., 2018) and first pre-train a deep convolutional network on Dbase using a standard cross-entropy loss. We use it to extract the feature representation x ∈ X for an input image I. Meta-learning is then performed over the pre-trained features {xi}. As shown in the shaded region in Figure 2, given an initial Strain, the hallucinator GθG generates additional examples for each class. Our framework applies to various types of hallucinators, and here we consider a powerful GAN-like hallucinator in Wang et al. (2018). Each hallucinated example is of the form (GθG(x, z), y), where (x, y) is a sampled seed example from Strain, and z is a sampled noise vector. The set of generated examples SGtrain is added to Strain to create an augmented training set Saugtrain. In the next section, we show how to meta-train GθG on Cbase, so that it can hallucinate new examples to augment Strain of Cnovel during meta-testing.

heading: 4 PRECISE COLLABORATIVE HALLUCINATOR
text: We now present our PrecisE Collaborative hAlluciNator (PECAN) shown in Figure 2, which exploits two important criteria for useful hallucination: precision and collaboration. As important constraints and guidance, these criteria facilitate hallucination to improve the classification performance.
Basic hallucinator with hard precision. At first, a precise hallucinator indicates that a classifier trained on Saugtrain should produce superior validation accuracy. We achieve this by training the hallucinator end-to-end with the learner (Wang et al., 2018). As shown in the shaded region in Figure 2, during each episode of meta-training, the learner module hclsθh uses S aug train to produce conditional probabilities hclsθh (x, S aug train) for each example (x, y) in the test set Stest. The meta-learner then back-
propagates the gradient of the total classification loss `cls = ∑
(x,y)∈Stest loss(h cls θh (x, Saugtrain), y) to update both the learner parameters θh and the hallucinator parameters θG.
Soft precision-inducing hallucinator. One of the important characteristics of an optimal generative model is that the generated examples should be indistinguishable from real ones (Goodfellow et al., 2014). We argue that, in terms of our recognition task oriented hallucinator, this means that the classifier trained on hallucinated examples needs to be similar to the classifier trained on real examples. As shown in Figure 2, given an initial relatively large training set S∗train, which contains n
∗ examples for each of the m classes, we randomly sample n (n n∗) examples per class, and obtain a subset Strain. From Strain, the hallucinator GθG generates n
∗ examples per class as SGtrain. This produces two training sets: S∗train with real examples and S G train with hallucinated examples, where both contain the same number of examples. Importantly, note that SGtrain is hallucinated from the subset Strain instead of the initial large set S∗train, and because n n∗, we rule out the trivial identity hallucinator or memorization. We train two additional classification networks: hreal based on S∗train and hG based on SGtrain, both of which have the same architecture as h
cls. When evaluated on the same test set Stest composed of real examples, a comparable performance between hreal and hG shows that the hallucinated samples are sufficiently precise, and as diverse as the real training set. Otherwise, when the hallucinator is imperfect, the accuracy of hG will be lower than that of hreal.
This similarity of classification accuracy essentially measures the difference between the learned (i.e., hallucinated) and the target (i.e., real) distributions, which could serve as an additional supervisory signal for training a better hallucinator. Since quantifying the similarity of accuracy directly would be difficult (Hinton et al., 2015), we instead introduce a loss function that acts on the network predictions. For an example (x, y) in Stest, the two networks produce conditional probabilities
preal(x) = hrealθh (x, S ∗ train) and p G(x) = hGθh(x, S G train), (1)
respectively. While only the largest entry in preal(x) or pG(x) is used to make predictions associated with the ground-truth label y, other entries still carry rich information about the recognition task and the network, as observed in (Hinton et al., 2015; Dvornik et al., 2019). We thus leverage the probabilities p̂real and p̂G in the absence of the ground-truth label and measure their similarity using the negative cosine distance:
ψ(p̂real, p̂G) = − cos(p̂real, p̂G), (2) where p̂real and p̂G are obtained by removing the logit for y in preal and pG, and re-normalizing the remaining logits using softmax with a learnable temperature. We treat the classification networks hcls, hreal, and hG as the new learner h and use shared parameters for them. Their difference thus lies in different conditional training sets. We obtain the soft precision-inducing loss `pre by summing the loss (2) in Stest and then combine it with the hard precision (i.e., the classification) loss as the classification objective. pG is now encouraged to not only make the right prediction according to the ground-truth label, but also make similar second-best, third-best, etc., choice predictions as preal.
Collaboration between hallucinator and learner. We now consider the interaction between the hallucinator G and the learner h. While hallucination is conducted in the pre-trained feature space X , the final classification is performed in a new embedding space Φ learned by the learner. Since the classification objective is directly imposed on the learner h, the hallucinator G continues to rely on the back-propagated signal to update its parameters. We may end up with a good embedding space Φ but a poor hallucinator G in the original space X . This undesired effect implies a potential imbalance between the hallucinator and the learner — a stronger learner that is able to make allowances for errors in hallucination, but a “lazy” hallucinator that does not make its best effort to capture the data distributions. Indeed, as is empirically validated in the experimental section (see Figure 3), despite being able to match the class distributions in the embedding space Φ, the hallucinated examples are initially pulled away from the class distributions in the feature space X . To mitigate this issue, we introduce a simple collaborative objective to the hallucinator, which provides an additional constraint or regularization on the hallucination process. This collaborative
objective is the same as the above classification objective (i.e., a combination of the classification loss and the precision-inducing loss), but enforces direct and early supervision for the hallucinator in the pre-trained feature space X . By doing so, we directly influence the update process of the hallucinator parameters, and generate much more discriminative examples right after hallucination than would be the case if we had to rely on gradual back-propagation from the learner alone. Our objective thus strengthens the cooperation between the hallucinator and the learner for the final classification performance, which can be viewed as a source of deep supervision that introduces auxiliary losses to intermediate layers when training deep neural networks (Simonyan & Zisserman, 2015; Lee et al., 2015). The overall objective combines the classification objective Llearner (on the learner) and the collaborative objective Lhal (on the hallucinator), each of which consists of a classification loss `cls (hard precision as cross-entropy with respect to ground-truth) and a soft precision-inducing loss `pre:
L(θG, θh) = Llearner + λLhal = `clslearner + λ1` pre learner + λ2` cls hal + λ3` pre hal , (3)
where λ, λ1, λ2, and λ3 are scalar hyper-parameters.
Our hallucinator is general and applies to different types of h (i.e., meta-learning algorithms). Here we focus on the widely used and powerful prototypical networks (PN) (Snell et al., 2017), prototype matching networks (PMN) (Vinyals et al., 2016; Wang et al., 2018), and cosine classifiers (Cos) (Gidaris & Komodakis, 2018; Chen et al., 2019a). Without loss of generality, we take PN as an example to explain the overall meta-training and meta-testing process. PN learns an embedding space Φ and uses a non-parametric nearest centroid classifier to assign class probabilities for a test example based on its distances from class means in Φ. As before, in each meta-training episode, after sampling S∗train, Strain, and Stest and hallucinating S G train in the pre-trained feature space X , we perform nearest centroid classification and produce the collaborative objective Lhal on Stest. We then feed the examples to the PN learner, obtain their embedded features in Φ, perform nearest centroid classification, and produce the classification objective Llearner on Stest. The final loss is back-propagated to update both the PN learner parameters θh and the hallucinator parameters θG. Figure 2 shows a schematic of the entire process. During meta-testing, we use the resulting GθG to hallucinate new examples to augment Strain of Cnovel, and we combine the predicted class probabilities in X and Φ as the final predictions.

heading: 5 EXPERIMENTAL EVALUATION
text: We explore the use of our meta-learning with hallucination framework for few-shot visual classification tasks. We focus the evaluation on the ImageNet based few-shot benchmark (Hariharan & Girshick, 2017; Wang et al., 2018). This is one of the largest datasets by far used for few-shot classification and it captures more realistic scenarios than others based on handwritten characters (Lake et al., 2015) or low-resolution images (Vinyals et al., 2016). The benchmark divides the 1,000 ImageNet categories (Russakovsky et al., 2015) into 389 base classes Cbase, with thousands of training images per class, and 611 novel classes Cnovel, with a small number n of training images per class. Following Hariharan & Girshick (2017), we use Cbase to train a convolutional network (ConvNet) based feature extractor and to conduct meta-training. Meta-testing is performed on Cnovel, and the performance is evaluated on a held-out test set, i.e., the original validation set of ImageNet. In addition, to avoid over-fitting, both Cbase and Cnovel are further split into two disjoint subsets. 193 of the base classes Ccvbase and 300 of the novel classes Ccvnovel are used for cross-validating hyper-parameters, and the remaining 196 base classes Cfinbase and 311 novel classes Cfinnovel are used for the final evaluation. Here we focus on hallucinating novel instances and thus evaluate the performance primarily on the novel classes Cfinnovel, which is also consistent with most of the contemporary work (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017). We report the mean top-1 and top-5 accuracies for 311-way, n = 1, 2, 5, 10, 20-shot classification, with each of them averaged over 5 trials.
In addition to this challenging version of ImageNet, we also evaluate on the widely used miniImageNet (Vinyals et al., 2016) dataset to show the generality of our approach. miniImageNet is a subset of 100 classes selected randomly from ImageNet with 600 images sampled from each class. Following the data split in Ravi & Larochelle (2017), we use 64 base, 16 validation, and 20 novel classes. We evaluate in the standard 5-way, 1-shot and 5-way, 5-shot settings (Vinyals et al., 2016).

heading: 5.1 RESULTS ON IMAGENET
text: Implementation details. We mainly use a ResNet-10 architecture (He et al., 2016) as the feature extractor, following Hariharan & Girshick (2017); Wang et al. (2018). Additionally, we provide results using a deeper ResNet-50 architecture in Section A.3. We extract and record the features, and perform meta-learning by using these pre-computed features. We consider three widely-used, powerful meta-learning approaches: prototypical networks (PN) (Snell et al., 2017), prototype matching networks (PMN) (Vinyals et al., 2016; Wang et al., 2018), and cosine classifiers (Cos-Cls) used in Gidaris & Komodakis (2018); Chen et al. (2019a). More implementation details are included in Section A.1.
Baselines. First we compare with the state-of-the-art meta-learning with hallucination method (Wang et al., 2018), which is a special case of our approach learned with only the hard precision loss. While Wang et al. (2018) focused on ‘PN w/ G’ and ‘PMN w/ G’, here we consider an additional type of classier with hallucination, ‘Cos-Cls w/ G’, to show the generality of our work. In addition, we compare with a variety of baselines, including (1) these meta-learning approaches with standard data augmentation techniques (Chen et al., 2019a); (2) data hallucination approaches which are not meta-learned: logistic regression with analogies hallucination (Hariharan & Girshick, 2017) and Gaussian hallucinator (Wang et al., 2018); (3) other recent meta-learning approaches: matching networks (MN) (Vinyals et al., 2016), model-agnostic meta-learning (MAML) (Finn et al., 2017), and ‘cosine classifier & attentive weight generators (Cos & Att)’ (Gidaris & Komodakis, 2018); (4) classical few-shot learning approaches: Siamese networks (SN) (Koch et al., 2015); and (5) simple baselines which are not meta-learned: logistic regression (Hariharan & Girshick, 2017). For fair comparison, all these baselines and our approach use the same pre-trained ConvNet backbone.
Comparisons with the state of the art. Table 1 shows that our PECAN consistently outperforms all the baselines by large margins across different scenarios. For this challenging 311-way classification, our improvements are of the order of 1% to 2%, while standard deviations for accuracy are of the order of 0.2%. For example, in the case of top-5 accuracy, our ‘PN w/ G + PECAN’ outperforms ‘PN w/ G’ by 1.5 points for n = 1 and 1.6 points for n = 20. Similar trends can be observed for ‘PMN w/
G + PECAN’ and ‘Cos-Cls w/ G + PECAN’, and also in the top-1 accuracy regime. This indicates that our approach is general and can work with different meta-learners.
Ablation studies. We conduct a series of ablations to evaluate the contribution of each component and different design choices. We use the prototypical network (PN) here due to its fast training speed.
Variants of PECAN. PECAN leverages two requirements for the meta-learned hallucinator: precision and collaboration. ‘`clslearner’ is the basic hallucinator with only the hard precision based on the classification loss. Table 2 shows that each requirement by itself yields performance superior to the basic hallucinator. The soft precision-inducing loss `pre consistently helps when combined with the hard precision `cls: ‘`clslearner + ` pre learner’ outperforms ‘` cls learner’ and ‘` cls hal + ` pre hal’ outperforms ‘` cls hal’. The collaboration objective integrates `learner and `hal to boost the performance: ‘`clslearner + ` cls hal’ outperforms ‘`clslearner’. Each component is thus essential and complementary to each other, enabling our full PECAN to outperform its variants.
Choice of similarity measure in soft precision-inducing loss. Our precision-inducing loss measures the similarity between classifier predictions preal and pG. We used negative cosine distance between the probabilities p̂real and p̂G in the absence of ground-truth labels. Table 3 compares with other types of similarity: variant of negative cosine distance, cross-entropy as in knowledge distillation (Hinton et al., 2015), Jensen-Shannon divergence, and symmetric KL-divergence (Dvornik et al., 2019). Our similarity achieves the best performance, and removing the true-class probability consistently helps.
Impact of collaborative objective. Our collaborative objective introduces additional direct and early supervision to train the hallucinator. Table 2 shows quantitatively its contribution to the overall accuracy. Here, we further qualitatively understand its impact though t-SNE visualizations (van der Maaten & Hinton, 2008) of the hallucinated examples for novel classes. For ease of analysis, we do not use the precision-inducing loss. Without the collaborative objective, despite being able to match the class distributions in the embedding space Φ (Figure 3b), the hallucinated examples are initially pulled away from the class distributions in the pre-trained feature space X (Figure 3a), indicating a “lazy” hallucinator. In contrast, the collaborative objective enforces the hallucinator to generate more discriminative examples right after hallucination (Figure 3c), leading to improved performance.
Qualitative visualizations. To better understand the hallucination process, Figure 4 shows some examples of classification results for our PECAN and the state-of-the-art baseline (Wang et al., 2018).
5.2 RESULTS ON miniIMAGENET
To show the generality of our approach, we further evaluate on miniImageNet. We use a ResNet-10 architecture and focus on incorporating our hallucinator into the metric-learning-based meta-learning approach, prototype matching networks (PMN) (Wang et al., 2018), and the optimization-based meta-learning approach, MAML (Finn et al., 2017). For MAMl, in each meta-training episode, we sample a batch of few-shot classification tasks. For each of the tasks, we sample training sets S∗train and Strain, sample a test test Stest, hallucinate SGtrain, and obtain an augmented training set S aug train. In the MAML inner loop, for each task, conditioning on its Saugtrain (S G train, or S ∗ train), we adapt the parameters of hcls (hG, or hreal) using few gradient updates. For each task, the adapted hcls (hG, or hreal) is evaluated on the corresponding Stest. In the MAML outer loop, we average the classification objective on Stest across the batch of tasks. In a similar way, we compute the collaborative objective in the pre-trained feature space. The final loss is used to update the initial MAML model and the hallucinator. From Table 4, our PECAN significantly outperforms all these state-of-the-art competitors, including other hallucination based approaches such as MetaGAN (Zhang et al., 2018a), delta-encoder (Schwartz et al., 2018), IDeMe-Net (Chen et al., 2019b), and SalNet (Zhang et al., 2019). Our superior performance over MetaGAN, a GAN-based approach to hallucinate data, shows that directly matching the classification performance is more desirable than matching the data distribution between hallucinated and real examples for recognition tasks. Our generic framework can be combined with more recent meta-learning methods, such as LEO (Rusu et al., 2019) and MetaOptNet-SVM (Lee et al., 2019), for further improvement.

heading: 6 CONCLUSION
text: We have presented an approach to few-shot classification that uses a precise collaborative hallucinator to generate additional examples. Our hallucinator integrates two important requirements that facilitate data hallucination in a way that most improves the classification performance, and is trained end-toend through meta-learning. The extensive experiments demonstrate our state-of-the-art performance on the challenging ImageNet and miniImageNet based few-shot benchmark in various scenarios.

heading: A APPENDIX

heading: A.1 ADDITIONAL IMPLEMENTATION DETAILS
text: The embedding architecture of PN is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0.01, and a Euclidean distance similar to Snell et al. (2017). The embedding architecture of PMN is composed of a one-layer bi-directional LSTM and attention LSTM, and a cosine distance as in (Vinyals et al., 2016; Wang et al., 2018). The embedding architecture of Cos-Cls is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0.01 and an additional one-layer MLP without nonlinearity, and a cosine distance with a learnable temperature similar to Gidaris & Komodakis (2018). The initial value of the temperature is 100. The classification weight vector is estimated by averaging the feature vectors of the training examples for each class. The hallucinator G is a three-layer MLP with leaky ReLU nonlinearity of slope 0.01, with its parameters initialized to block diagonal identity matrices (Wang et al., 2018). The dimensionality of the hidden layers is 512 for ResNet-10 features and 2,048 for ResNet-50 features.
Both the baselines (PN/PMN/Cos-Cls and PN/PMN/Cos-Cls with hallucination) and our approach are meta-trained on ImageNet for 60,000 episodes by SGD with an initial learning rate 0.05 for PN/PMN and 0.005 for Cos-Cls, decayed by a factor of 10 every 20,000 episodes. In each episode of the meta-training stage, we sample all the base classes following Wang et al. (2018), which found that it is advantageous to use more classes rather than fewer. We sample n∗=20 examples per class from the base dataset Dbase, leading to an initial training set S∗train. Consistent with Wang et al. (2018), to make a single hallucinator robust to different sample sizes, we randomly sample different sized examples per class from S∗train, from 1 to 15 examples per class, to obtain Strain. One random seed example is sampled from Strain, and fed into the hallucinator G with different noise vectors to generate 20 examples per class as SGtrain. Hallucinated examples are sampled from S G train and added to Strain until there are exactly 20 examples per class in S aug train. For the test set Stest, we have 5 random examples per class for prototypical networks (PN) (Snell et al., 2017) and cosine classifiers(Cos-Cls) (Gidaris & Komodakis, 2018), and 1 random example per class for prototype matching networks (PMN) (Vinyals et al., 2016; Wang et al., 2018).
For the soft precision-inducing loss, we use softmax with temperature to produce conditional class probabilities p̂real and p̂G. This temperature is a learnable parameter. It is shared between p̂real and p̂G, but is not shared between Llearner and Lhal.
The hyper-parameters obtained by cross-validation on ImageNet are: λ1=0.5, λ2=0.02, and λ3=0.1 for PN with ResNet-10; λ1=0.2, λ2=0.0001, and λ3=0.05 for PN with ResNet-50; λ1=0.4, λ2=2.0, and λ3=0.1 for PMN with ResNet-10; and λ1=0.5, λ2=0.000001, and λ3=0.005 for the cosine classifier with ResNet-10.
During the meta-testing stage, following Wang et al. (2018) we use n=1, 2, 5, 10 or 20 examples per class from the novel dataset Dnovel, and then hallucinate a fixed number of additional examples for each novel class. By cross-validation, the number of hallucinated examples per class is set to 10 for PN, Cos-Cls and Cos-Cls w/ G + PECAN with ResNet-10, 8 for Cos-Cls w/ G with ResNet-10, 5 for PN with ResNet-50, and 20 for PMN with ResNet-10. We combine the classifier prediction results in the pre-trained feature space and the learned embedding space using a scalar hyper-parameter. By cross-validation, this hyper-parameter is set to 0.05 for PN with ResNet-10, 0.07 for PN with ResNet-50, 1 for PMN with ResNet-10, and 0.00002 for the cosine classifier with ResNet-10. For the test set Stest, following Wang et al. (2018) we have 50 real examples per class, and we average the top-1 or top-5 accuracy of them over the novel classes.

heading: A.2 PERFORMANCE ON BASE CLASSES
text: While we significantly improve the classification performance on novel classes, our approach remains accurate on base classes. For example, for prototypical networks (PN) (Snell et al., 2017), both our ‘PN w/ G + PECAN’ and the baseline ‘PN w/ G’ achieve the same top-5 accuracy 92.4% on ImageNet.
A.3 IMPACT OF DEEPER REPRESENTATION MODELS
Figure A.1 shows the results on ImageNet using features from a ResNet-50 architecture. As expected, deeper networks result in better performance for all the approaches, but our PECAN hallucination
strategy still provides large gains across the board over the state-of-the-art meta-learned hallucinator in (Wang et al., 2018).

heading: A.4 ANALYSIS OF HYPER-PARAMETER SENSITIVITY
text: Hyper-parameters in the overall objective. We conduct sensitivity experiments for the hyperparameters λ1, λ2, and λ3, which trade off different loss components in the overall objective of our PECAN. We vary one of the three hyper-parameters while fixing the remaining two to their cross-validated values. Figures A.2a, A.2b, and A.2c show the top-5 accuracy of ‘PN w/ G + PECAN’ on the novel classes for the ImageNet based n-shot classification benchmark. We can see that the top-5 accuracy is stable over a wide range of hyper-parameter values, for example when the value of λ1 becomes 50 times larger or 100 times smaller than λ1 used in the main paper. Across the board, our PECAN consistently and significantly outperforms the baselines shown in the main paper.
Number of hallucinated examples. We also show how the top-5 accuracy changes for n = 1-shot classification with respect to the number of hallucinated images in Figure A.2d. We can see that when the number of hallucinated examples is changed from 0 to 10, the performance of our PECAN gradually improves, and then saturates and drops slightly with more than 10 images generated.

heading: A.5 ADDITIONAL ANALYSIS OF SOFT PRECISION-INDUCING LOSS
text: Our soft precision-inducing loss measures the similarity between classifier predictions preal and pG. This is a general similarity measure which applies to various types of classifiers, including parametric and non-parametric classifiers. For prototypical networks (PN) (Snell et al., 2017), a non-parametric nearest centroid classifier is used to assign class probabilities for a test example based on its distances from class means. Hence, in this special case, to measure the similarity between classifier predictions, we can directly calculate the distance between the mean of real examples mreal and the mean of hallucinated examples mG. Table A.1 compares our generic similarity with this specific similarity for PN on the ImageNet based n-shot classification benchmark. The result shows that our similarity generalizes significantly better for novel classes.

heading: A.6 ADDITIONAL VISUALIZATIONS OF CLASSIFICATION RESULTS
text: Similar to Figure 5 in the main paper, here we provide more examples of classification results for our PECAN and the state-of-the-art meta-learned hallucinator (Wang et al., 2018) in Figure A.3.

heading: A.7 EXTENSION TO FEW-SHOT REGRESSION
text: In the main paper, we focus on few-shot classification tasks. However, our approach is general and applies to few-shot regression tasks as well. When extending our approach to address regression tasks, we need to slightly modify the design of the hallucinator G. Specifically, in classification tasks, the hallucinator G takes as input a seed example (x, y) and outputs a hallucinated example (x′, y′) = (G(x, z), y), where the class label y′ = y, indicating that the hallucinated example belongs to the same class as the seed example. In contrast, in regression tasks, y becomes a continuous quantity. We thus cannot enforce the constraint y′ = y. To address this issue, we modify the hallucinator G to directly generate the tuple (x′, y′) = G (x, y, z). This is achieved by the concatenation of x and y as input to G.
We incorporate our hallucinator with MAML (Finn et al., 2017) and evaluate on the n = 5-shot sinusoidal regression task proposed in Finn et al. (2017). Each task regresses from the input to the output of a sine curve, and different tasks differ in the amplitude and phase of the sinusoid. The amplitude and the phase are uniformly distributed within [0.1, 5.0] and [0, π], respectively. During meta-training and meta-testing, the input datapoints x of each task are uniformly sampled from [−5.0, 5.0]. The prediction loss is measured by the mean squared error (MSE) between the prediction and true value. The regressor is a feedforward neural network with 2 hidden layers of size 40 with ReLU nonlinearity. We use MAML (Finn et al., 2017) with one gradient update based on n = 5 examples with a fixed step size α = 0.01, and use Adam as the meta-optimizer (Kingma & Welling, 2014).
We compare with two baselines: (1) standard MAML, and (2) ‘MAML w/ G’, which is MAML with the hallucinator in (Wang et al., 2018). While Wang et al. (2018) focus on classification tasks, here we extend the use of its hallucinator in the similar way as we discussed before. Table A.2 shows that our PECAN consistently outperforms the baselines for the regression task as well, indicating the generality of our approach.
A.8 VISUALIZATIONS OF HALLUCINATED EXAMPLES
Our hallucination is performed in the pre-trained feature space, and visualizing them directly is not intuitive. In addition to t-SNE visualization of hallucinated samples in Figure 3, here in Figure A.4 we include an additional visualization of hallucinated examples in the pixel space, using the nearest neighbor real image in the feature space, corresponding to each hallucinated feature sample.

Review summaries:
In this paper, the authors address few-shot learning via a precise collaborative hallucinator. In particular, they follow the framework of (Wang et al., 2018), and introduce two kinds of training regularization. The soft precision-inducing loss follows the spirit of adversarial learning, by using knowledge distillation. Additionally, a collaborative objective is introduced as middle supervision to enhance the learning capacity of hallucinator. 

This paper describes a method that builds upon the work of Wang et al. It meta-learns to hallucinate additional samples for few-shot learning for classification tasks. Their two main insights of this paper are to propose a soft-precision term which compares the classifiers' predictions for all classes other than the ground truth class for both a few-shot training set and the hallucinated set and b) to introduce the idea of applying direct early supervision in the feature space in which the hallucination is conducted in addition to in the classifier embedding space. This allows for stronger supervision and prevents the hallucinated samples from not being representative of the classes. The authors show small, but consistent improvement in performance on two benchmarks: ImageNet and miniImageNet with two different network architectures versus various state-of-the-art meta-learning algorithms with and without hallucination. The authors have adequately cited and reviewed the existing literature. They have also conducted many experiments (both in the main paper and in the supplementary material) to show the superior performance of their approach versus the existing ones. Furthermore their ablation studies both for the type of soft precision loss and for their various individual losses are quite nice and thorough.