Meta-Learning by Hallucinating Useful Examples

1 INTRODUCTION . Modern deep learning models rely heavily on large amounts of annotated examples ( Deng et al. , 2009 ) . Their data-hungry nature limits their applicability to real-world scenarios , where the cost of annotating examples is prohibitive , or they involve rare concepts ( Zhu et al. , 2014 ; Fink , 2011 ) . In contrast , humans can grasp a new concept rapidly and make meaningful generalizations , even from a single example ( Schmidt , 2009 ) . To bridge this gap , there has been a recent resurgence of interest in few-shot learning that aims to learn novel concepts from very few labeled examples ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ) . Existing work tries to solve this problem from the perspective of meta-learning ( Thrun , 1998 ; Schmidhuber , 1987 ) , which is motivated by the human ability to leverage prior experiences when tackling a new task . Unlike the standard machine learning paradigm , where a model is trained on a set of exemplars , meta-learning is performed on a set of tasks , each consisting of its own training and test sets ( Vinyals et al. , 2016 ) . By sampling small training and test sets from a large collection of labeled examples of base classes , meta-learning based few-shot classification approaches learn to extract task-agnostic knowledge , and apply it to a new few-shot learning task of novel classes . One notable type of task-agnostic ( or meta ) knowledge comes from the shared mechanism of data augmentation or hallucination across categories ( Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ) . Hallucinating additional training data by generating images may seem like an easy solution for few-shot learning , but it is often challenging . In fact , the success of this paradigm is usually restricted to certain domains like handwritten characters ( Lake et al. , 2013 ) , or requires additional supervision ( Dixit et al. , 2017 ; Zhang et al. , 2018b ) or sophisticated heuristics ( Hariharan & Girshick , 2017 ) . An alternative to generating raw data in the form of visually realistic images is to hallucinate examples in a learned feature space ( Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ; Xian et al. , 2019 ) . This can be achieved by , for example , integrating a “ hallucinator ” module into a meta-learning framework , where it generates hallucinated examples , guided by real examples ( Wang et al. , 2018 ) . The learner then uses an augmented training set which includes both the real and the hallucinated examples to learn classifiers . While the existing approaches showed that it is possible to adjust the hallucinator to generate examples that are helpful for classification , the generation process is still far from producing effective samples in the few-shot regime . Our key insight is that , to facilitate data hallucination to improve the performance of new classification tasks , two important requirements should be satisfied : ( i ) precision : the generated examples should lead to good classifier performance , and ( ii ) collaboration : all the components including the hallucinator and the learner need to be trained jointly . In this work , we propose PrecisE Collaborative hAlluciNator ( PECAN ) , which integrates these requirements into a general meta-learning with hallucination framework , as shown in Figure 1 . Assume that we have a hallucinator to generate additional examples from the original small training set . A precise hallucinator indicates that a classifier trained on both the hallucinated and the few real examples should produce superior validation accuracy . This can be achieved by training the hallucinator end-to-end with the learner , and back-propagating a classification loss based on groundtruth labels of validation data ( Wang et al. , 2018 ) . Since this precision is measured using ground-truth labels , we term it as hard precision . And more importantly , if the hallucinator perfectly captures the target distribution , a classifier trained on a set of hallucinated examples , despite being generated from a small set of real examples , should produce roughly the same validation accuracy as a classifier trained on a large set of real examples , when these two sets are of the same sample size ( Shmelkov et al. , 2018 ) . This indicates similar level of realism and diversity between the generated and the real examples , as shown in Figure 1a . Motivated by this observation , we introduce an additional precision-inducing loss function , which explicitly encourages the hallucinator to generate examples so that a classifier trained on them makes predictions similar to the one trained on a large amount of real examples . Given that this precision is measured based on classifier predictions , we term it as soft precision . This precision , which is complementary to hard precision and effective , as shown in our experiment , is lacking in current approaches ( Wang et al. , 2018 ) . Satisfying the precision requirement alone is not sufficient , since the classification objective is still directly associated with the learner , and thus the hallucinator continues to rely on the back-propagated signal to update its parameters . This leads to a potential undesirable effect of imbalanced training between the hallucinator and the learner : the learner tends to be stronger and makes allowances for errors in the hallucination , whereas the hallucinator becomes “ lazy ” and does not make its best effort to capture the data distributions , which is empirically observed in our experiments ( See Figure 3 ) . To address this issue , our key insight is to enforce direct and early supervision for the hallucinator , and make its contribution to the overall classification transparent , as shown in Figure 1b . Hence , we introduce a collaborative objective for the hallucinator , which allows us to directly influence the generation process to favor highly discriminative examples right after hallucination , and to strengthen the cooperation between the hallucinator and the learner . Our contributions are three-fold . ( 1 ) We propose a novel loss that helps produce precise hallucinated examples , by using the classifier trained on real examples as a guidance , and encouraging the classifier trained on hallucinated examples to mimic its behavior . ( 2 ) We introduce a collaborative objective for the hallucinator as early supervision , which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner . ( 3 ) By integrating these properties , we develop a general meta-learning with hallucination framework , which is model-agnostic and can be combined with any meta-learning models to consistently boost their few-shot learning performance . Here we mainly focus on few-shot classification tasks , and we show that our approach applies to few-shot regression tasks as well in the appendix A.7 . 2 RELATED WORK . As one of the unsolved problems in machine learning and computer vision , few-shot learning is attracting growing interest in the deep learning era ( Miller et al. , 2000 ; Fei-Fei et al. , 2006 ; Lake et al. , 2015 ; Santoro et al. , 2016 ; Wang & Hebert , 2016 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Hariharan & Girshick , 2017 ; George et al. , 2017 ; Triantafillou et al. , 2017 ; Edwards & Storkey , 2017 ; Mishra et al. , 2018 ; Douze et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2019a ; Dvornik et al. , 2019 ) . Successful generalization from few training samples requires appropriate “ inductive biases ” or shared knowledge from related tasks ( Baxter , 1997 ) , which is commonly acquired through transfer learning and more recently meta-learning ( Thrun , 1998 ; Schmidhuber , 1987 ; Schmidhuber et al. , 1997 ; Bengio et al. , 1992 ) . By explicitly “ learning-to-learn ” over a series of few-shot learning tasks ( i.e. , episodes ) , which are simulated from base classes , meta-learning exploits accumulated task-agnostic knowledge to target few-shot learning problems of novel classes . Within this paradigm of approaches , various types of meta-knowledge has been recently explored , including ( 1 ) a generic feature embedding or metric space , in which images are easy to classify using a distance-based classifier such as cosine similarity or nearest neighbor ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Ren et al. , 2018 ; Oreshkin et al. , 2018 ) ; ( 2 ) a common initialization of network parameters ( Finn et al. , 2017 ; Nichol & Schulman , 2018 ; Finn et al. , 2018 ) or learned update rules ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ; Munkhdalai & Yu , 2017 ; Li et al. , 2017 ; Rusu et al. , 2019 ) ; ( 3 ) a transferable strategy to estimate model parameters based on few novel class examples ( Bertinetto et al. , 2016 ; Qiao et al. , 2018 ; Qi et al. , 2018 ; Gidaris & Komodakis , 2018 ) , or from an initial small dataset model ( Wang & Hebert , 2016 ; Wang et al. , 2017 ) . Complementary to these discriminative approaches , our work focuses on synthesizing samples to deal with data scarcity . There has been progress in this direction of data hallucination , either in pixel or feature spaces ( Salakhutdinov et al. , 2012 ; George et al. , 2017 ; Lake et al. , 2013 ; 2015 ; Wong & Yuille , 2015 ; Rezende et al. , 2014 ; Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Dixit et al. , 2017 ; Hariharan & Girshick , 2017 ; Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ) . However , it is still challenging for modern generative models to capture the entirety of data distribution ( Salimans et al. , 2016 ) and produce useful examples that maximally boost the recognition performance ( Wang et al. , 2018 ) , especially in the small sample-size regime . In the context of generative adversarial networks ( GANs ) , Shmelkov et al . ( 2018 ) show that images synthesized by state-of-the-art approaches , despite their impressive visual quality , are insufficient to tackle recognition tasks , and encourage the use of quantitative measures based on classification results to evaluate GAN models . Rather than using classification results as a performance measure , we go a step further in this paper by leveraging classification objectives to guide the generation process . Other related work such as Wang et al . ( 2018 ) proposed a general data hallucination framework based on meta-learning , which is a special case of our approach . A GAN-like hallucinator takes a seed example and a random noise vector as input to generate a new sample . This hallucinator is trained jointly with the classifier in an end-to-end manner . Delta-encoder ( Schwartz et al. , 2018 ) is a variant of Wang et al . ( 2018 ) , where instead of using noise vectors , it modifies an auto-encoder to extract transferable intra-class deformations , i.e. , “ deltas ” , and applies them to novel samples to generate new instances . Unlike the above approaches that directly use the produced samples to train the classifier , MetaGAN ( Zhang et al. , 2018a ) trains the classifier in an adversarial manner to augment the classifier with the ability to discriminate between real and synthesized data . Another variant ( Gao et al. , 2018 ) explicitly preserves covariance information to enable better augmentation . Our work investigates critical yet unexplored properties in this paradigm that the data hallucinator should satisfy . These properties are general and can be flexibly incorporated into existing meta-learning approaches and hallucination methods , providing significant gains irrespective of these choices .

Summary: This paper describes a method that builds upon the work of Wang et al. It meta-learns to hallucinate additional samples for few-shot learning for classification tasks. Their two main insights of this paper are to propose a soft-precision term which compares the classifiers' predictions for all classes other than the ground truth class for both a few-shot training set and the hallucinated set and b) to introduce the idea of applying direct early supervision in the feature space in which the hallucination is conducted in addition to in the classifier embedding space. This allows for stronger supervision and prevents the hallucinated samples from not being representative of the classes. The authors show small, but consistent improvement in performance on two benchmarks: ImageNet and miniImageNet with two different network architectures versus various state-of-the-art meta-learning algorithms with and without hallucination. The authors have adequately cited and reviewed the existing literature. They have also conducted many experiments (both in the main paper and in the supplementary material) to show the superior performance of their approach versus the existing ones. Furthermore their ablation studies both for the type of soft precision loss and for their various individual losses are quite nice and thorough.