Meta-Learning by Hallucinating Useful Examples

1 INTRODUCTION . Modern deep learning models rely heavily on large amounts of annotated examples ( Deng et al. , 2009 ) . Their data-hungry nature limits their applicability to real-world scenarios , where the cost of annotating examples is prohibitive , or they involve rare concepts ( Zhu et al. , 2014 ; Fink , 2011 ) . In contrast , humans can grasp a new concept rapidly and make meaningful generalizations , even from a single example ( Schmidt , 2009 ) . To bridge this gap , there has been a recent resurgence of interest in few-shot learning that aims to learn novel concepts from very few labeled examples ( Fei-Fei et al. , 2006 ; Vinyals et al. , 2016 ; Wang & Hebert , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ) . Existing work tries to solve this problem from the perspective of meta-learning ( Thrun , 1998 ; Schmidhuber , 1987 ) , which is motivated by the human ability to leverage prior experiences when tackling a new task . Unlike the standard machine learning paradigm , where a model is trained on a set of exemplars , meta-learning is performed on a set of tasks , each consisting of its own training and test sets ( Vinyals et al. , 2016 ) . By sampling small training and test sets from a large collection of labeled examples of base classes , meta-learning based few-shot classification approaches learn to extract task-agnostic knowledge , and apply it to a new few-shot learning task of novel classes . One notable type of task-agnostic ( or meta ) knowledge comes from the shared mechanism of data augmentation or hallucination across categories ( Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ) . Hallucinating additional training data by generating images may seem like an easy solution for few-shot learning , but it is often challenging . In fact , the success of this paradigm is usually restricted to certain domains like handwritten characters ( Lake et al. , 2013 ) , or requires additional supervision ( Dixit et al. , 2017 ; Zhang et al. , 2018b ) or sophisticated heuristics ( Hariharan & Girshick , 2017 ) . An alternative to generating raw data in the form of visually realistic images is to hallucinate examples in a learned feature space ( Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ; Xian et al. , 2019 ) . This can be achieved by , for example , integrating a “ hallucinator ” module into a meta-learning framework , where it generates hallucinated examples , guided by real examples ( Wang et al. , 2018 ) . The learner then uses an augmented training set which includes both the real and the hallucinated examples to learn classifiers . While the existing approaches showed that it is possible to adjust the hallucinator to generate examples that are helpful for classification , the generation process is still far from producing effective samples in the few-shot regime . Our key insight is that , to facilitate data hallucination to improve the performance of new classification tasks , two important requirements should be satisfied : ( i ) precision : the generated examples should lead to good classifier performance , and ( ii ) collaboration : all the components including the hallucinator and the learner need to be trained jointly . In this work , we propose PrecisE Collaborative hAlluciNator ( PECAN ) , which integrates these requirements into a general meta-learning with hallucination framework , as shown in Figure 1 . Assume that we have a hallucinator to generate additional examples from the original small training set . A precise hallucinator indicates that a classifier trained on both the hallucinated and the few real examples should produce superior validation accuracy . This can be achieved by training the hallucinator end-to-end with the learner , and back-propagating a classification loss based on groundtruth labels of validation data ( Wang et al. , 2018 ) . Since this precision is measured using ground-truth labels , we term it as hard precision . And more importantly , if the hallucinator perfectly captures the target distribution , a classifier trained on a set of hallucinated examples , despite being generated from a small set of real examples , should produce roughly the same validation accuracy as a classifier trained on a large set of real examples , when these two sets are of the same sample size ( Shmelkov et al. , 2018 ) . This indicates similar level of realism and diversity between the generated and the real examples , as shown in Figure 1a . Motivated by this observation , we introduce an additional precision-inducing loss function , which explicitly encourages the hallucinator to generate examples so that a classifier trained on them makes predictions similar to the one trained on a large amount of real examples . Given that this precision is measured based on classifier predictions , we term it as soft precision . This precision , which is complementary to hard precision and effective , as shown in our experiment , is lacking in current approaches ( Wang et al. , 2018 ) . Satisfying the precision requirement alone is not sufficient , since the classification objective is still directly associated with the learner , and thus the hallucinator continues to rely on the back-propagated signal to update its parameters . This leads to a potential undesirable effect of imbalanced training between the hallucinator and the learner : the learner tends to be stronger and makes allowances for errors in the hallucination , whereas the hallucinator becomes “ lazy ” and does not make its best effort to capture the data distributions , which is empirically observed in our experiments ( See Figure 3 ) . To address this issue , our key insight is to enforce direct and early supervision for the hallucinator , and make its contribution to the overall classification transparent , as shown in Figure 1b . Hence , we introduce a collaborative objective for the hallucinator , which allows us to directly influence the generation process to favor highly discriminative examples right after hallucination , and to strengthen the cooperation between the hallucinator and the learner . Our contributions are three-fold . ( 1 ) We propose a novel loss that helps produce precise hallucinated examples , by using the classifier trained on real examples as a guidance , and encouraging the classifier trained on hallucinated examples to mimic its behavior . ( 2 ) We introduce a collaborative objective for the hallucinator as early supervision , which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner . ( 3 ) By integrating these properties , we develop a general meta-learning with hallucination framework , which is model-agnostic and can be combined with any meta-learning models to consistently boost their few-shot learning performance . Here we mainly focus on few-shot classification tasks , and we show that our approach applies to few-shot regression tasks as well in the appendix A.7 . 2 RELATED WORK . As one of the unsolved problems in machine learning and computer vision , few-shot learning is attracting growing interest in the deep learning era ( Miller et al. , 2000 ; Fei-Fei et al. , 2006 ; Lake et al. , 2015 ; Santoro et al. , 2016 ; Wang & Hebert , 2016 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Hariharan & Girshick , 2017 ; George et al. , 2017 ; Triantafillou et al. , 2017 ; Edwards & Storkey , 2017 ; Mishra et al. , 2018 ; Douze et al. , 2018 ; Wang et al. , 2018 ; Chen et al. , 2019a ; Dvornik et al. , 2019 ) . Successful generalization from few training samples requires appropriate “ inductive biases ” or shared knowledge from related tasks ( Baxter , 1997 ) , which is commonly acquired through transfer learning and more recently meta-learning ( Thrun , 1998 ; Schmidhuber , 1987 ; Schmidhuber et al. , 1997 ; Bengio et al. , 1992 ) . By explicitly “ learning-to-learn ” over a series of few-shot learning tasks ( i.e. , episodes ) , which are simulated from base classes , meta-learning exploits accumulated task-agnostic knowledge to target few-shot learning problems of novel classes . Within this paradigm of approaches , various types of meta-knowledge has been recently explored , including ( 1 ) a generic feature embedding or metric space , in which images are easy to classify using a distance-based classifier such as cosine similarity or nearest neighbor ( Koch et al. , 2015 ; Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Ren et al. , 2018 ; Oreshkin et al. , 2018 ) ; ( 2 ) a common initialization of network parameters ( Finn et al. , 2017 ; Nichol & Schulman , 2018 ; Finn et al. , 2018 ) or learned update rules ( Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ; Munkhdalai & Yu , 2017 ; Li et al. , 2017 ; Rusu et al. , 2019 ) ; ( 3 ) a transferable strategy to estimate model parameters based on few novel class examples ( Bertinetto et al. , 2016 ; Qiao et al. , 2018 ; Qi et al. , 2018 ; Gidaris & Komodakis , 2018 ) , or from an initial small dataset model ( Wang & Hebert , 2016 ; Wang et al. , 2017 ) . Complementary to these discriminative approaches , our work focuses on synthesizing samples to deal with data scarcity . There has been progress in this direction of data hallucination , either in pixel or feature spaces ( Salakhutdinov et al. , 2012 ; George et al. , 2017 ; Lake et al. , 2013 ; 2015 ; Wong & Yuille , 2015 ; Rezende et al. , 2014 ; Goodfellow et al. , 2014 ; Radford et al. , 2016 ; Dixit et al. , 2017 ; Hariharan & Girshick , 2017 ; Wang et al. , 2018 ; Gao et al. , 2018 ; Schwartz et al. , 2018 ; Zhang et al. , 2018a ) . However , it is still challenging for modern generative models to capture the entirety of data distribution ( Salimans et al. , 2016 ) and produce useful examples that maximally boost the recognition performance ( Wang et al. , 2018 ) , especially in the small sample-size regime . In the context of generative adversarial networks ( GANs ) , Shmelkov et al . ( 2018 ) show that images synthesized by state-of-the-art approaches , despite their impressive visual quality , are insufficient to tackle recognition tasks , and encourage the use of quantitative measures based on classification results to evaluate GAN models . Rather than using classification results as a performance measure , we go a step further in this paper by leveraging classification objectives to guide the generation process . Other related work such as Wang et al . ( 2018 ) proposed a general data hallucination framework based on meta-learning , which is a special case of our approach . A GAN-like hallucinator takes a seed example and a random noise vector as input to generate a new sample . This hallucinator is trained jointly with the classifier in an end-to-end manner . Delta-encoder ( Schwartz et al. , 2018 ) is a variant of Wang et al . ( 2018 ) , where instead of using noise vectors , it modifies an auto-encoder to extract transferable intra-class deformations , i.e. , “ deltas ” , and applies them to novel samples to generate new instances . Unlike the above approaches that directly use the produced samples to train the classifier , MetaGAN ( Zhang et al. , 2018a ) trains the classifier in an adversarial manner to augment the classifier with the ability to discriminate between real and synthesized data . Another variant ( Gao et al. , 2018 ) explicitly preserves covariance information to enable better augmentation . Our work investigates critical yet unexplored properties in this paradigm that the data hallucinator should satisfy . These properties are general and can be flexibly incorporated into existing meta-learning approaches and hallucination methods , providing significant gains irrespective of these choices .

Summary: In this paper, the authors address few-shot learning via a precise collaborative hallucinator. In particular, they follow the framework of (Wang et al., 2018), and introduce two kinds of training regularization. The soft precision-inducing loss follows the spirit of adversarial learning, by using knowledge distillation. Additionally, a collaborative objective is introduced as middle supervision to enhance the learning capacity of hallucinator.