PAIRWISE AUGMENTED GANS WITH ADVERSARIAL RECONSTRUCTION LOSS

1 INTRODUCTION . Deep generative models are a powerful tool to sample complex high dimensional objects from a low dimensional manifold . The dominant approaches for learning such generative models are variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) and generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . VAEs allow not only to generate samples from the data distribution , but also to encode the objects into the latent space . However , VAE-like models require a careful likelihood choice . Misspecifying one may lead to undesirable effects in samples and reconstructions ( e.g. , blurry images ) . On the contrary , GANs do not rely on an explicit likelihood and utilize more complex loss function provided by a discriminator . As a result , they produce higher quality images . However , the original formulation of GANs ( Goodfellow et al. , 2014 ) lacks an important encoding property that allows many practical applications . For example , it is used in semi-supervised learning ( Kingma et al. , 2014 ) , in a manipulation of object properties using low dimensional manifold ( Creswell et al. , 2017 ) and in an optimization utilizing the known structure of embeddings ( Gómez-Bombarelli et al. , 2018 ) . VAE-GAN hybrids are of great interest due to their potential ability to learn latent representations like VAEs , while generating high-quality objects like GANs . In such generative models with a bidirectional mapping between the data space and the latent space one of the desired properties is to have good reconstructions ( x ≈ G ( E ( x ) ) ) . In many hybrid approaches ( Rosca et al. , 2017 ; Ulyanov et al. , 2018 ; Zhu et al. , 2017 ; Brock et al. , 2017 ; Tolstikhin et al. , 2017 ) as well as in VAE-like methods it is achieved by minimizing L1 or L2 pixel-wise norm between x and G ( E ( x ) ) . However , the main drawback of using these standard reconstruction losses is that they enforce the generative model to recover too many unnecessary details of the source object x . For example , to reconstruct a bird picture we do not need an exact position of the bird on an image , but the pixel-wise loss penalizes a lot for shifted reconstructions . Recently , Li et al . ( 2017 ) improved ALI model ( Dumoulin et al. , 2017 ; Donahue et al. , 2017 ) by introducing a reconstruction loss in the form of a discriminator which classifies pairs ( x , x ) and ( x , G ( E ( x ) ) ) . However , in such approach , the discriminator tends to detect the fake pair ( x , G ( E ( x ) ) ) just by checking the identity of x and G ( E ( x ) ) which leads to vanishing gradients . In this paper , we propose a novel autoencoding model which matches the distributions in the data space and in the latent space independently as in Zhu et al . ( 2017 ) . To ensure good reconstructions , we introduce an augmented adversarial reconstruction loss as a discriminator which classifies pairs ( x , a ( x ) ) and ( x , G ( E ( x ) ) ) where a ( · ) is a stochastic augmentation function . This enforces the discriminator to take into account content invariant to the augmentation , thus making training more robust . We call this approach Pairwise Augmented Generative Adversarial Networks ( PAGANs ) . Measuring a reconstruction quality of autoencoding models is challenging . A standard reconstruction metric RMSE does not perform the content-based comparison . To deal with this problem we propose a novel metric Reconstruction Inception Dissimilarity ( RID ) which is robust to content-preserving transformations ( e.g. , small shifts of an image ) . We show qualitative results on common datasets such as MNIST ( LeCun & Cortes , 2010 ) , CIFAR10 ( Krizhevsky et al. , 2009 ) and CelebA ( Liu et al. , 2015 ) . PAGANs outperform existing VAE-GAN hybrids in Inception Score ( Salimans et al. , 2016 ) and Fréchet Inception Distance ( Heusel et al. , 2017 ) except for the recently announced method PD-WGAN ( Gemici et al. , 2018 ) on CIFAR10 dataset . 2 PRELIMINARIES . Let us consider an adversarial learning framework where our goal is to match the true distribution p∗ ( x ) to the model distribution pθ ( x ) . As it was proposed in the original paper Goodfellow et al . ( 2014 ) , the model distribution pθ ( x ) is induced by the generator Gθ : z −→ x where z is sampled from a prior p ( z ) . To match the distributions p∗ ( x ) and pθ ( x ) in an adversarial manner , we introduce a discriminator Dψ : x −→ [ 0 , 1 ] . It takes an object x and predicts the probability that this object is sampled from the true distribution p∗ ( x ) . The training procedure of GANs ( Goodfellow et al. , 2014 ) is based on the minimax game of two players : the generator Gθ and the discriminator Dψ . This game is defined as follows min θ max ψ V ( θ , ψ ) = Ep∗ ( x ) logDψ ( x ) + Epz ( z ) log ( 1−Dψ ( Gθ ( z ) ) ) ( 1 ) where V ( θ , ψ ) is a value function for this game . The optimal discriminator Dψ∗ given fixed generator Gθ is Dψ∗ ( x ) = p∗ ( x ) p∗ ( x ) + pθ ( x ) ( 2 ) and then the value function for the generator V ( θ , ψ∗ ) given the optimal discriminator Dψ∗ is equivalent to the Jensen-Shanon divergence between the model distribution pθ ( x ) and the true distribution p∗ ( x ) , i.e . θ∗ = argmin θ V ( θ , ψ∗ ) = argmin θ JSD ( p∗ ( x ) ‖pθ ( x ) ) . ( 3 ) However , in practice , the gradient of the value function V ( θ , ψ ) with respect to the generator ’ s parameters θ vanishes to zero . Therefore , Goodfellow et al . ( 2014 ) proposed to train the generator Gθ by minimizing − logDψ ( Gθ ( z ) ) instead of log ( 1 − Dψ ( Gθ ( z ) ) ) . This loss for the generator provides much more stable gradients and has the same fixed point as the minimax game of Dψ and Gθ . 3 PAIRWISE AUGMENTED GENERATIVE ADVERSARIAL NETWORKS . In PAGANs model our aim is not only to learn how to generate real objects with the generator Gθ ( z ) where z is sampled from prior p ( z ) but at the same time learn an inverse mapping ( encoder ) Eϕ : x −→ z. Additionally , we use the third stochastic transformation a : x −→ y without parameters which is called augmenter . It produces the augmentation y of the source object x . Let us consider the distributions which are induced by these three mappings • pθ ( x|z ) - the conditional distribution of outputs of the generator Gθ ( z ) given z ; • qϕ ( z|x ) - the conditional distribution of outputs of the encoder Eϕ ( x ) given x ; • r ( y|x ) - the conditional distribution over the augmentations a ( x ) given a source object x . Within the PAGANs model our goal is to find such optimal parameters θ∗ and ϕ∗ that ensure 1. generator matching : pθ∗ ( x ) = p∗ ( x ) where pθ∗ ( x ) = ∫ pθ∗ ( x|z ) p ( z ) dz , i.e . the generator Gθ∗ samples objects from the true distribution p∗ ( x ) ; 2. encoder matching : qϕ∗ ( z ) = p ( z ) where qϕ∗ ( z ) = ∫ qϕ∗ ( z|x ) p∗ ( x ) dx , i.e . the encoder Eϕ generates embeddings z as the prior p ( z ) ; 3. reconstruction matching : pθ∗ , ϕ∗ ( y|x ) = r ( y|x ) where pθ∗ , ϕ∗ ( y|x ) = ∫ pθ∗ ( y|z ) qϕ∗ ( z|x ) dz , ( 4 ) i.e . reconstructions Gθ∗ ( Eϕ∗ ( x ) ) are distributed as augmentations r ( y|x ) of the source object x . 3.1 GENERATOR & ENCODER MATCHING . In order to deal with generator and encoder matching problems we can use the framework of the vanilla GANs ( Goodfellow et al. , 2014 ) . We introduce two discriminators Dψx and Dψz for two minimax games : • generator matching : min θ max ψx Vx ( θ , ψx ) = Ep∗ ( x ) logDψx ( x ) + Epθ ( x ) log ( 1−Dψx ( x ) ) ( 5 ) • encoder matching : min ϕ max ψz Vz ( ϕ , ψz ) = Ep ( z ) logDψz ( z ) + Eqϕ ( z ) log ( 1−Dψz ( z ) ) ( 6 ) Then the value functions Vx and Vz given the optimal discriminators Dψ∗x and Dψ∗z are equivalent to Jensen-Shanon divergence : θ∗ = argmin θ Vx ( θ , ψ ∗ x ) = argmin θ JSD ( p∗ ( x ) ‖pθ ( x ) ) ( 7 ) ϕ∗ = argmin ϕ Vz ( ϕ , ψ ∗ z ) = argmin ϕ JSD ( p ( z ) ‖qϕ ( z ) ) ( 8 ) 3.2 RECONSTRUCTION MATCHING : AUGMENTED ADVERSARIAL RECONSTRUCTION LOSS . The solution of the reconstruction matching problem ensures that reconstructions Gθ ( Eϕ ( x ) ) correspond to the source object x up to defined random augmentations a ( x ) . In PAGANs model we introduce the minimax game for training the adversarial distance between the reconstructions and augmentations of the source object x . We consider the discriminator Dψ which takes a pair ( x , y ) and classifies it into one of the following classes : • the real class : pairs ( x , y ) from the distribution p∗ ( x ) r ( y|x ) , i.e . the object x is taken from the true distribution p∗ ( x ) and the second y is obtained from the x by the random augmentation a ( x ) ; • the fake class : pairs ( x , y ) from the distribution p∗ ( x ) pθ , ϕ ( y|x ) = p∗ ( x ) ∫ pθ ( y|z ) qϕ ( z|x ) dz , ( 9 ) i.e . x is sampled from p∗ ( x ) then z is generated from the conditional distribution qϕ ( z|x ) by the encoder Eϕ ( x ) and y is produced by the generator Gϕ ( z ) from the conditional model distribution pθ ( y|z ) . Then the minimax problem is min θ , ϕ max ψ V ( θ , ϕ , ψ ) ( 10 ) where V ( θ , ϕ , ψ ) = Ep∗ ( x ) r ( y|x ) logDψ ( x , y ) + Ep∗ ( x ) pθ , ϕ ( y|x ) log ( 1−Dψ ( x , y ) ) ( 11 ) Let us prove that such minimax game will match the distributions r ( y|x ) and pθ , ϕ ( y|x ) . At first , we find the optimal discriminator : Proposition 1 . Given a fixed generator Gθ and a fixed encoder Eϕ , the optimal discriminator Dψ∗ is Dψ∗ ( x , y ) = r ( y|x ) r ( y|x ) + pθ , ϕ ( y|x ) ( 12 ) Proof . Given in Appendix A.1 . Then we can prove that given an optimal discriminator the value function V ( θ , ϕ , ψ ) is equivalent to the expected Jensen-Shanon divergence between the distributions r ( y|x ) and pθ , ϕ ( y|x ) . Proposition 2 . The minimization of the value function V under an optimal discriminator Dψ∗ is equivalent to the minimization of the expected Jensen-Shanon divergence between r ( y|x ) and pθ , ϕ ( y|x ) , i.e . θ∗ , ϕ∗ = argmin θ , ϕ V ( θ , ϕ , ψ∗ ) = argmin θ , ϕ Ep∗ ( x ) JSD ( r ( y|x ) ‖pθ , ϕ ( y|x ) ) ( 13 ) Proof . Given in Appendix A.2 . If r ( y|x ) = δx ( y ) then the optimal discriminator Dψ∗ ( x , y ) will learn an indicator I { x = y } as was proved in Li et al . ( 2017 ) . As a consequence , the objectives of the generator and the encoder are very unstable and have vanishing gradients in practice . On the contrary , if the distribution r ( y|x ) is non-degenerate as in our model then the value function V ( θ , ϕ , ψ ) will be well-behaved and much more stable which we observed in practice .

Summary: The paper propose a adversary method to train a bidirectional GAN with both an encoder and decoder. Comparing to the existing works, the main contribution is the introducing of an augmented reconstruction loss by training a discriminator to distinguish the augmentation data from the reconstructed data. Experimental results are demonstrated to show the generating and reconstruction performance.