Augmentation-Interpolative AutoEncoders for Unsupervised Few-Shot Image Generation

1 INTRODUCTION . Modern generative models can synthesize high-quality ( Karras et al. , 2019 ; Razavi et al. , 2019 ; Zhang et al. , 2018a ) , diverse ( Ghosh et al. , 2018 ; Mao et al. , 2019 ; Razavi et al. , 2019 ) , and highresolution ( Brock et al. , 2018 ; Karras et al. , 2017 ; 2019 ) images of any class , but only given a large training dataset for these classes ( Creswell et al. , 2017 ) . This requirement of a large dataset is impractical in many scenarios . For example , an artist might want to use image generation to help create concept art of futuristic vehicles . Smartphone users may wish to animate a collection of selfies , or researchers training an image classifier might wish to generate augmented data for rare classes . These and other applications will require generative models capable of synthesizing images from a large , ever-growing set of object classes . We can not rely on having hundreds of labeled images for all of them . Furthermore , most of them will likely be unknown at the time of training . We therefore need generative models that can train on one set of image classes , and then generalize to a new class using only a small quantity of new images : few-shot image generation . Unfortunately , we find that the latest and greatest generative models can not even represent novel classes in their latent space , let alone generate them on demand ( Figure 1 ) . Perhaps because of this generalization challenge , recent attempts at few-shot image generation rely on undesirable assumptions and compromises . They need impractically large labeled datasets of hundreds of classes ( Edwards & Storkey , 2016 ) , involve substantial computation at test time ( Clouâtre & Demers , 2019 ) , or are highly domain-specific , generalizing only across very similar classes ( Jitkrittum et al. , 2019 ) . In this paper , we introduce a strong , efficient , unsupervised baseline for few-shot image generation that avoids all the above compromises . We leverage the finding that although the latent spaces of powerful generative models , such as VAEs and GANs , do not generalize to new classes , the representations learned by autoencoders ( AEs ) generalize extremely well . The AEs can then be converted into generative models by training them to interpolate between seed images ( Sainburg et al. , 2018 ; Berthelot et al. , 2018 ; Beckham et al. , 2019 ) . These Interpolative AutoEncoders ( IntAEs ) would seem a natural fit for few-shot image generation . Unfortunately , we also find that although IntAEs can reproduce images from novel classes , the ability to interpolate between them breaks down upon leaving the training domain . To remedy this , we introduce a new training method based on data augmentation , which produces smooth , meaningful interpolations in novel domains . We demonstrate on three different settings ( handwritten characters , faces and general objects ) that our Augmentation-Interpolative Autoencoder ( AugIntAE ) achieves simple , robust , highly general , and completely unsupervised few-shot image generation . 2 RELATED WORK . 2.1 GENERATIVE MODELING . AEs were originally intended for learned non-linear data compression , which could then be used for downstream tasks ; the generator network was discarded ( Kramer , 1991 ; Hinton & Salakhutdinov , 2006 ; Masci et al. , 2011 ) . VAEs do the opposite : by training the latent space toward a prior distribution , the encoder network can be discarded at test time instead . New images are sampled directly from the prior ( Kingma & Welling , 2013 ) . Subsequent models discard the encoder network entirely . GANs sample from a noise distribution and learn to generate images which fool a concurrently-trained real/fake image discriminator ( Goodfellow et al. , 2014 ) . Bojanowski et al . ( 2017 ) and Hoshen et al . ( 2019 ) treat latent codes as learnable parameters directly , and train separate sampling procedures for synthesizing the novel images . Recent work has also seen a return to AEs as conditional generators , by training reconstruction networks to interpolate smoothly between pairs or sets of seed images . This is accomplished by combining the reconstruction loss on seed images with an adversarial loss on the seed and interpolated images . Different forms of adversarial loss ( Sainburg et al. , 2018 ; Berthelot et al. , 2018 ) and interpolation ( Beckham et al. , 2019 ) have been proposed . While all of these approaches generate new images , it is unclear if any of them can generalize to novel domains . Some results suggest the opposite : a VAE sufficiently powerful to model the training data becomes incapable of producing anything else ( Bozkurt et al. , 2018 ) . 2.2 FEW-SHOT IMAGE GENERATION . Current attempts at few-shot image generation span a wide range of approaches and models . Neural Statistician , an early attempt , is similar to the AE in that it is built for few-shot classification , and largely discards the generative capability ( Edwards & Storkey , 2016 ) . Generation-oriented iterations exist , but likewise depend on a large , varied , labelled dataset for training ( Hewitt et al. , 2018 ) . Other approaches based on few-shot classification include generative matching networks ( Bartunov & Vetrov , 2018 ) and adversarial meta-learning ( Clouâtre & Demers , 2019 ) . These models also depend on heavy supervision , and are fairly complicated , involving multiple networks and training procedures working in tandem - making them potentially difficult to train in practice reliably . Separate work has approached few-shot image generation from the side of generative modeling . Wang et al . ( 2018 ) , Noguchi & Harada ( 2019 ) and Wu et al . ( 2018 ) investigate the ability of GANs to handle domain adaptation via fine-tuning - thus requiring substantial computation , and more novel class examples than are available in the few-shot setting . Zhao et al . ( 2020 ) train GANs directly from few examples , though still more than are at hand for few-shot learning , and can be considered orthogonal work , as AugIntAE can serve as a useful pre-trained initialization . Antoniou et al . ( 2017 ) and Liu et al . ( 2019 ) use adversarial training to produce feed-forward few-shot generators . However , both models still depend on varied , labelled training data , and risk exhibiting the same problems as standard GANs : mode collapse and hyperparameter sensitivity ( Arora et al. , 2017 ; 2018 ) . Jitkrittum et al . ( 2019 ) introduce an algorithm for class-conditioning an unconditioned generative model . New images are produced by matching latent space batch statistics to real images from a single , possibly novel class . Nguyen et al . ( 2017 ) learn individual latent codes from a pretrained discriminator , while Wang et al . ( 2020 ) train a latent sampling network . These approaches have little to no evaluation on novel classes , and to what degree they generalize depends entirely on the pretrained image generator . They may also require substantial test-time computation . In contrast , AugIntAEs are lightweight , train robustly , and generalize broadly from completely unlabelled data . 3 PROBLEM SETUP . Let X be a large , unlabelled collection of images depicting objects from a set of classes C. Let X ′ be a very small set of images - as few as two - belonging to a novel class c′ 6∈ C. Our goal is to train a network on X which , given X ′ , generates images clearly belonging to c′ . We refer to this as the network ’ s ability to generalize to new domains ( note that this usage is distinct from “ generalizing ” to novel images in the same domain , a much simpler task ) . We can not directly adapt the network to X ′ using SGD , as X ′ contains too few images to prevent overfitting . This is an extremely difficult problem , since it is unclear if a neural network trained to model the data distribution in X can even represent images from a different distribution , let alone sample from it . Therefore , we first examine whether existing generative models can faithfully encode novel class images in latent space . We train a VAE and a WGAN-GP ( Gulrajani et al. , 2017 ) on MNIST handwritten digits ( LeCun et al. , 1998 ) , as well as an encoder network that inverts the WGAN-GP as in Bau et al . ( 2019 ) ( details in appendix ) . We then evaluate the ability of each generative model to recover particular images . Using the built-in VAE encoder and the WGAN-GP inversion network , we find that while both models can reconstruct training images ( Fig . 2 , top ) , the same approach fails on images from novel classes - in this case , EMNIST handwritten letters ( Cohen et al. , 2017 ) . The outputs do not much resemble the inputs ; crucial semantic information is lost ( Fig . 2 , bottom ) . To discount the possibility of sub-optimal encoders , we simulate an oracle encoder , refining the latent code parameters for each image directly via SGD . These reconstructions are not much better . Fig . 1 demonstrates a similar failure in a large , state-of-the-art pretrained GAN . This confirms prior findings ( Bozkurt et al. , 2018 ) that current generative approaches by default can not even represent images from novel classes . Generating new novel class images is simply out of the question . Why do sophisticated generative models fail to generalize ? We argue this is largely by design . Generative models such as VAEs and GANs are trained to minimize the divergence between a prior distribution and a learned posterior distribution , where one or both are approximated by repeated sampling . VAEs push the learned latent posterior toward a Gaussian prior , while GANs map samples from the prior to a posterior distribution in image space . In both cases , latent vectors are repeatedly sampled and sent through the generator . Thus , by the time the generator is trained to convergence , and the posterior approaches the prior ( or vice-versa ) , every region of the latent space feasible under the prior will have been mapped at some point to a training image - or , in the case of GANs , to an image indistinguishable from a training image . This means that a properly trained VAE or GAN can not construct or reconstruct new object classes . If it could , then it would have been able to sample such images during training - which would mean it had not been properly trained at all . 4 AUGMENTATION-INTERPOLATIVE AUTOENCODERS . AutoEncoders : As discussed above , minimizing the divergence between prior and posterior training distributions ensures good image synthesis , but poor generalization . The opposite could also hold : AEs do not enforce any latent distribution on the data posterior , and so might generalize well . More formally , given a network E that maps an image x to latent vector z , and a generator G mapping z back to image space , we refer to the function composition G ( E ( · ) ) as the autoencoder . E and G are trained jointly over X to minimize the pixel reconstruction error between x ∈ X and G ( E ( x ) ) . The question of generalization becomes , to what degree does a trained AE maintain close proximity between x′ and G ( E ( x′ ) ) for x′ which lies far from the data manifold of X ? By this measure , we find that our conjecture holds : AEs generalize surprisingly well . Examples are given in Fig . 3 , demonstrating near-perfect generalization performance over three pairs of classdisjoint datasets : MNIST digits to EMNIST letters , Omniglot training alphabets to Omniglot testing alphabets ( Lake et al. , 2015 ) , and CIFAR-10 to CIFAR-100 ( Krizhevsky et al. , 2009 ) . We also quantitatively evaluate generalization ( in terms of reconstruction error ) between all the above pairs , as well as between Omniglot and MNIST/EMNIST , which includes domain shifts , e.g. , stroke width ( Table 1 ) . Reconstruction quality is high across the board , especially given that the MNIST and CIFAR-10 networks learn only ten distinct classes ! AEs exhibit very little overfitting to the training domain , learning a general mapping despite heavy class constraints . It is possible that this generalization is a result of AEs simply learning an identity function . Fortunately , this is not the case : AEs learn clear image priors . We find that our trained AEs are much more effective at encoding real images than noise ( see Fig . 3 , right ) . We also find that low-frequency noise is encoded more faithfully than high-frequency noise - an analysis is provided in appendix . The learned AE mapping , while general , is also nontrivial . Interpolative AutoEncoders : The fact that AEs generalize suggests they are capable of acting as few-shot image generators for novel classes , given a method for sampling appropriate latent z vectors . One possibility is to interpolate between data points in latent space : every sampled point is a weighted sum of two or more real points . This allows us to produce novel combinations of seed images without changing the semantic content . Unfortunately , it is a known fact that AEs do not interpolate well , as shown in Fig . 5 , row 2 ( Berthelot et al. , 2018 ) . Prior work ( Sainburg et al. , 2018 ; Berthelot et al. , 2018 ; Beckham et al. , 2019 ) addresses this by applying an adversarial loss to the interpolated images , which works well in the training domain . However , we find the Interpolative AutoEncoder ( IntAE ) approach overly restrictive for our purposes : it constrains all interpolations between arbitrary image pairs to the training domain . For example , on MNIST , an IntAE must produce recognizable digits when interpolating between a 3 and a 4 , a semantically unintuitive result . This makes the learning process harder and causes the model to learn interpolations that do not generalize . When it interpolates between letters ( Fig . 5 , row 3 , left ) , we find that it does produce undesirable artifacts - that look like numbers ! Augmentation-Interpolative AutoEncoders : We introduce a novel training procedure for IntAEs to remove such artifacts while maintaining generalizability . Instead of optimizing interpolated images using only a GAN loss , we train the network to directly recover known , semantically interpolated images from the corresponding interpolated latent code . This accomplishes two things simultaneously : first , we only interpolate between images where the interpolation makes semantic sense , since we must know the interpolated image in advance . This simplifies the learning problem significantly . Second , the model is no longer constrained to the training manifold when interpolating arbitrary training image pairs . The network can now learn simpler , more direct interpolations that work well on both training and novel domains . Formally , suppose we have a triplet of imagesA = f ( ρ1 ) , B = f ( ρ2 ) and C = f ( αρ1+ ( 1−α ) ρ2 ) , where f is some unknown image generating process , ρ is a semantic variable , and α ∈ [ 0 , 1 ] . Using this triplet , we train the interpolative AE to reconstruct C by decoding the interpolated latent code of A and B . Formally , we train the encoder E and the generator G by minimizing : Lrecon = ||C −G ( αE ( A ) + ( 1− α ) E ( B ) ) ||1 ( 1 ) In practice , finding appropriate image tripletsA , B , C in a dataset of independent images is difficult . Instead , we synthesizeA , B , C using affine spatial transformations , and color jitter for 3-channel images . Given training image x , we randomly sample two sets of augmentation parameters ( translation , rotation , hue , etc. ) . Applying each of these transformations independently to x yields A and B ( for example , a 10o rotation and a −5o rotation ) . We then sample a weight α ∈ [ 0 , 1 ] and compute a weighted average of the two transformations , which we apply to x to produce C ( in our example , if α = 13 , C represents a 5 o rotation ) . The Augmentation-Interpolative AutoEncoder ( AugIntAE ) is then trained to recover C from the α-weighted interpolation between the latent embeddings for A and B . This corresponds to Equation 1 . We can also augment the model with the original IntAE losses : a reconstruction loss on A and B , and a GAN loss Ladv on the interpolated C. In practice , we found that the former did not noticeably affect performance , while the latter was helpful in reducing the blurriness of output images . Subsequent models include Ladv . The full procedure is displayed in Fig . 4 . At first glance , learning the space of affine and color transformations does not appear particularly helpful for an IntAE . Very few visual relationships in the real world can be captured by these transformations alone . However , we find that these learned interpolations act as a powerful regularizer on the latent space , allowing AugIntAE to smoothly capture far more interesting and difficult transformations as well , such as shape , lighting , and even 3D pose . Few-shot generation : Once the AugIntAE is trained , we can sample novel images given only a set of seeds . Simply select a random pair of images , find their latent space embeddings , sample α ∈ [ 0 , 1 ] , and generate the image from the α-weighted mixture of embeddings . More sophisticated sampling techniques are possible , but left to future work .

Summary: The paper investigates the generative model which generalizes to new domain with limited samples.  Authors firstly explore the current hot generative models:  VAEs and GANs, and experimentally find that both VAEs and GANs fail to learn a model which generalizes  well to novel domain.  Interestingly, AutoEncoders exhibits effective performance of the generalizability to new domain. With the encouraging insight, authors further approach Augmentation-Interpolative AutoEncoders. Specially,  the paper firstly augments the training sample to get the  input pair, and extracts the latent feature by the sharing Encoder. A weighted sum of both features is conducted to form the mixed feature, which further is taken as input for the decoder to synthesize the output sample. Differently   the paper performs the reconstruction loss between the output and the mixed input which sum the input pair with the same from to the one of the latent space.