Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourages convex latent distributions

We present a neural network architecture based upon the Autoencoder ( AE ) and Generative Adversarial Network ( GAN ) that promotes a convex latent distribution by training adversarially on latent space interpolations . By using an AE as both the generator and discriminator of a GAN , we pass a pixel-wise error function across the discriminator , yielding an AE which produces sharp samples that match both high- and low-level features of the original images . Samples generated from interpolations between data in latent space remain within the distribution of real data as trained by the discriminator , and therefore preserve realistic resemblances to the network inputs . 1 INTRODUCTION . Generative modeling has the potential to become an important tool for exploring the parallels between perceptual , physical , and physiological representations in fields such as psychology , linguistics , and neuroscience ( e.g . Sainburg et al . 2018 ; Thielk et al . 2018 ; Zuidema et al . 2018 ) . The ability to infer abstract and low-dimensional representations of data and to sample from these distributions allows one to quantitatively explore and vary complex stimuli in ways which typically require hand-designed feature tuning , for example varying formant frequencies of vowel phonemes , or the fundamental frequency of syllables of birdsong . Several classes of unsupervised neural networks such as the Autoencoder ( AE ; Hinton & Salakhutdinov 2006 ; Kingma & Welling 2013 ) , Generative Adversarial Network ( GAN ; Goodfellow et al . 2014 ) , autoregressive models ( Hochreiter & Schmidhuber , 1997 ; Graves , 2013 ; Oord et al. , 2016 ; Van Den Oord et al. , 2016 ) , and flow-based generative models ( Kingma & Dhariwal , 2018 ; Dinh et al. , 2014 ; Kingma et al. , 2016 ; Dinh et al. , 2016 ) are at present popularly used for learning latent representations that can be used to generate novel data samples . Unsupervised neural network approaches are ideal for data generation and exploration because they do not rely on hand-engineered features and thus can be applied to many different types of data . However , unsupervised neural networks often lack constraints that can be useful or important for psychophysical experimentation , such as pairwise relationships between data in neural network projections , or how well morphs between stimuli fit into the true data distribution . We propose a novel AE that hybridizes features of an AE and a GAN . Our network is trained explicitly to control for the structure of latent representations and promotes convexity in latent space by adversarially constraining interpolations between data samples in latent space to produce realistic samples1 . 1Pairwise interpolations in between latent samples may only cover a subset of the convex hull of the latent distribution , as described in Figure 1 . 1.1 BACKGROUND ON GENERATIVE ADVERSARIAL NETWORKS ( GANS ) AND AUTOENCODERS ( AES ) . An AE is a form of neural network which takes as input xi ( e.g . an image ) , and is trained to generate a reproduction of the input2 G ( xi ) , by minimizing some error function between the input xi and output G ( xi ) ( e.g . pixel-wise error ) . This translation is usually performed after compressing the representation xi into a low-dimensional representation zi . This low-dimensional representation is called a latent representation , and the layer corresponding to zi in the neural network is often called the latent layer . The first half of the network , which translates from xi to zi , is called the encoder ; the second half of the network , which translates from zi to xi , is called the decoder . The combination of these two networks make the AE capable of both dimensionality reduction ( X → Z ) ; Hinton & Salakhutdinov 2006 ) , and generativity ( Z → X ) . Importantly , AE architectures are generative3 , however they are not generative models ( Bishop , 2006 ) because they do not model the joint probability of the observable and target variables P ( X , Z ) . Variants such as the Variational Autoencoder ( VAEs ; Kingma & Welling 2013 ) , which model P ( X , Z ) are generative models . AE latent spaces , therefore , can not be sampled probabilistically , without modeling the joint probability as in VAEs . The AE architecture that we propose here does not model the joint probability of X and Z and thus is not a generative model , although the latent space of our network could be modeled probabilistically ( e.g . with a VAE ) . GAN architectures are comprised of two networks , a generator , and a discriminator . The generator takes as input a latent sample , zi , drawn randomly from a distribution ( e.g . uniform or normal ) , and is trained to produce a sampleGd ( zi ) in the data domainX . The discriminator takes as input both xi and Gd ( zi ) , and is trained to differentiate between real xi , and generated Gd ( zi ) samples , typically by outputting either a 0 or 1 in a single-neuron output layer . The generator is trained to oppose the discriminator by ’ tricking ’ it into categorizing Gd ( z ) samples as x samples . Intuitively , this results in the generator producing Gd ( Z ) samples indistinguishable ( at least to the discriminator ) from those drawn from the distribution x . Thus the discriminator acts as a ’ critic ’ of the samples produced by a generator that is attempting to reproduce the distribution x . Because GANs sample directly from a predefined latent distribution , GANs are generative models , explicitly representing the joint probability , P ( X , Z ) . One common use for both GANs and AEs has been exploiting the semantically rich low-dimensional manifold , Z , on which data are either projected onto or sampled from ( White , 2016 ; Hinton & Salakhutdinov , 2006 ) . Operations performed in Z carry rich semantic features of data , and interpolations between points in Z produce semantically smooth interpolations in the original data space X ( e.g . Radford et al . 2015 ; White 2016 ) . However , samples generated by latent representations of both AEs and GANs are limited by the constraints provided by the algorithm . A significant amount of work has been done over the past several years in developing variants of AEs and GANs which add additional constraints and functionality to GAN and AE architectures , for example improving stability of GANs ( e.g . Berthelot et al . 2017 ; Radford et al . 2015 ; Salimans et al . 2016 ) , disentangling latent representations ( e.g . Higgins et al . 2016 ; Chen et al . 2016 ; Bouchacourt et al . 2017 ) , adding generative capacity to AEs ( e.g . Kingma & Welling 2013 ; Kingma et al . 2016 ; Makhzani et al . 2015 ) , and adding bidirectional inference to GANs ( e.g . Larsen et al . 2015 ; Mescheder et al . 2017 ; Berthelot et al . 2017 ; Dumoulin et al . 2016 ; Ulyanov et al . 2017 ; Makhzani 2018 ) . In this work , we describe several limitations of GANs and Autoencoders , specifically as they relate to stimuli generation for psychophysical research , and propose a novel architecture , GAIA , that utilizes aspects of both the AE and GAN to negate shortcomings of each architecture independently . Our method provides a novel approach to increasing the stability of network training , increasing the convexity of latent space representations , preserving of high-dimensional structure in latent space representations , and bidirectionality from X → Z and Z → X . 1.2 CONVEXITY OF LATENT SPACE . Generative latent-spaces enable the powerful capacity for smooth interpolations between real-world signals in a high-dimensional space . Linear interpolations in a low-dimensional latent space often 2We denote G ( xi ) as being equivalent to Gd ( Ge ( xi ) ) , or xi being passed through the encoder and decoder of the generator , G 3having the power or function of generating , originating , producing , or reproducing ( Webster , 2018 ) produce comprehensible representations when projected back into high-dimensional space ( e.g . Engel et al . 2017 ; Dosovitskiy et al . 2015 ) . In the latent spaces of many network architectures such as AEs , however , linear interpolations are not necessarily justified , because the space between endpoints on an interpolation in Z is not explicitly trained to fall within the data distribution when translated back into X . A convex set of points is defined as a set in which the line-segment connecting any pair of points will fall within the rest of the set ( Klee , 1971 ) . For example , in Figure 1A , the purple distribution represents data projected into a two-dimensional latent space , and the surrounding whitespace represents regions of latent space that do not correspond to the data distribution . This distribution would be non-convex because a line connecting two points in the distribution ( e.g . the black points in Figure 1A ) could contain points outside the data distribution ( the red region ) . In an AE , if the red region of the interpolation were sampled , projections back into the high-dimensional space may not necessarily correspond to realistic exemplars of x , because that region of Z does not belong to the true data distribution . One approach to overcoming non-convexity in a latent space is to force the latent representation of the dataset into a pre-defined distribution ( e.g . a normal distribution ) , as is performed by VAEs . By constraining the latent space of a VAE to fit a normal distribution , the latent space representations are encouraged to belong to a convex set . This method , however , pre-imposes a distribution in latent space that may be a suboptimal representation of the high-dimensional dataset . Standard GAN latent distributions are sampled directly , similarly allowing arbitrary convex distributions to be explicitly chosen for latent spaces . In both cases , hard-coding the distributional structure of the latent space may not respect the high-dimensional structure of the original distribution . 1.3 PIXEL-WISE ERROR AND BIDIRECTIONALITY . AEs that perform dimensionality reduction ( in particular VAEs ) can produce blurry images due to their pixel-wise loss functions ( Goodfellow et al. , 2014 ; Larsen et al. , 2015 ) , which minimize loss by smoothing the sharp contrasts ( e.g . edges ) present in real data . GANs do not suffer from this blurring problem , because they are not trained to reproduce input data . Instead , GANs are trained to generate data that could plausibly belong to the true distribution X . Thus , smoothing over uncertainty tends to be discouraged by the discriminator because it can use smoothed edges as a distinguishing feature between data sampled from X and G ( X ) . Producing data that fits into the distribution of x , rather than reproducing individual instances of xi comes at a cost , however . While AEs learn both the translation from X to Z and Z to X , GANs only learn the latter ( Z → X ) . In other words , the pixel-wise loss function of the AE produces smoothed data but is bidirectional , while the discriminator-based loss function of the GAN produces sharp images and is unidirectional . 2 GENERATIVE ADVERSARIAL INTERPOLATIVE AUTOENCODING ( GAIA ) . Our model , GAIA ( Figure 1 left ) , is bidirectional but is trained on both a GAN loss function and a pixel-wise loss function , where the pixel-wise loss function is passed across the discriminator of the GAN to ensure that features such as blurriness are discriminated against . In full , GAIA is trained as a GAN in which both the generator and the discriminator are AEs . The discriminator is trained to minimize the pixel-wise loss ( ` 1 ) between real data ( xi ) and their AE reproduction in the discriminator ( D ( xi ) ) while maximizing the AE loss between samples generated ( G ( xi ) ) by the generator and their reproduction in the discriminator ( D ( G ( xi ) ) ) : ‖xi −D ( xi ) ‖1 − ‖xi −D ( G ( xi ) ) ‖1 The generator is trained on the inverse , to minimize the pixel-wise loss between input ( xi ) and output ( D ( G ( xi ) ) ) such that the discriminator reproduces the generated samples to be as close to the original data as possible : ‖xi −D ( G ( xi ) ) ‖1 Using an AE as a generator has been previously been used in the VAE-GAN ( Larsen et al. , 2015 ) , and decreases blurring from the pixel-wise loss in AEs at the expense of exact-reproduction of data . Similarly , using an AE as a discriminator has been previously used in BEGAN ( Berthelot et al. , 2017 ) , which improves stability in GANs but remains unidirectional4 . In GAIA , we combine these two architectures , allowing the generator to be trained on a pixel-wise loss that is passed across the discriminator , explicitly reproducing data as in an AE , while producing sharper samples characteristic of a GAN . In addition , linear interpolations are taken between the latent-space representations : β ← N ( µ , σ2 ) zint . = zgeniβ + zgenj ( 1− β ) Where interpolations are Euclidean interpolations between pairs of points in Z , sampled from a 1-dimensional Gaussian distribution5 centered around the midpoint between zgeni and zgenj . The midpoints are then passed through the decoder of the discriminator , which are treated as generated samples by the GAN loss function : ‖Gd ( zint. ) −D ( Gd ( zint . ) ) ‖1 The discriminator is trained to maximize this loss , and the generator is trained to minimize this loss . 4Although it is possible to find the regions of Z most closely corresponding to xi 5σ = 0.25 . We sample along the midpoint using a Gaussian rather than uniformly because we found that interpolations near to original samples required less training than interpolations to produce realistic interpolations . In full , the loss of the discriminator , as in BEGAN , is to minimize pixel-wise loss of real data , and maximize pixel-wise loss of generated data ( including interpolations ) : LDisc =‖xi −D ( xi ) ‖1− ‖xi −D ( G ( xi ) ) ‖1− ‖Gd ( zint. ) −D ( Gd ( zint . ) ) ‖1 The loss of the generator is to minimize the error across the descriminator for the input data in the generator ( D ( G ( xi ) ) ) , along with the minimizing the error of the interpolations generated by the generator ( D ( Gd ( zint. ) ) ) . LGen =‖xi −D ( G ( xi ) ) ‖1+ ‖Gd ( zint. ) −D ( Gd ( zint . ) ) ‖1 In summary , the generator and discriminator are both AEs . As a result , reconstructions of x have the potential to resemble the input data ( G ( x ) ) at a pixel level , a feature non-existent in other GAN based inference methods ( Figure 5 ) . We also train the network on interpolations in the generator , to explicitly train the generator to produce interpolations ( Gd ( zint . ) ) which deceive the discriminator and are closer to the distribution in X than interpolations from an unconstrained AE .

Summary: This paper proposes an autoencoder architecture and training procedure for producing high-quality reconstructions and realistic interpolations. A "generator" autoencoder is trained to fool a "discriminator" autoencoder. The generator tries to minimize its own reconstruction error and minimize the reconstruction error of the discriminator when fed with interpolated latent vectors of real datapoints. The discriminator autoencoder has three losses, corresponding to minimizing reconstruction error on real datapoints and maximizing reconstruction error on the generator's output on both real datapoints and interpolated outputs. The authors also propose a loss which encourages the distances between real datapoints and their corresponding latent vectors to be similar, as well as a heuristic procedure for stabilizing GAN training. Qualitative results are shown on CelebA.