Intervention Adversarial Auto-Encoder

In this paper we propose a new method to stabilize the training process of the latent variables of adversarial auto-encoders , which we name Intervention Adversarial auto-encoder ( IVAAE ) . The main idea is to introduce a sequence of distributions that bridge the distribution of the learned latent variable and its prior distribution . We theoretically and heuristically demonstrate that such bridge-like distributions , realized by a multi-output discriminator , have an effect on guiding the initial latent distribution towards the target one and hence stabilizing the training process . Several different types of the bridge distributions are proposed . We also apply a novel use of Stein variational gradient descent ( SVGD ) ( Liu & Wang , 2016 ) , by which point assemble develops in a smooth and gradual fashion . We conduct experiments on multiple real-world datasets . It shows that IVAAE enjoys a more stable training process and achieves a better generating performance compared to the vanilla Adversarial auto-encoder ( AAE ) ( Makhzani & Shlens , 2015 ) . 1 INTRODUCTION . Generative models are widely used for image and texture production . Among them there are two base models which are most appealing to scholars for their elegant theoretical foundation and close combination with neural network , namely Variational Auto-encoders ( VAE ) ( Kingma & Welling , 2013 ) and Generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) . VAE maximizes a lower bound of the log-likelihood called ELBO which decomposes to a reconstruction loss term and a regularization term within an auto-encoder structure . GAN , however , goes beyond the likelihood concern , trying to generate data directly using a learned map from the latent space that is trained in an adversarial manner . Both models have some weakness . For example , GAN ’ s training is quite unstable and sometimes may have mode collapse problem . VAE usually produces less sharp pictures . The regularity of the latent space is also an important concern for VAE that affects the quality of the generated images . There ’ re many attempts to alleviate the above drawbacks . Adversarial Auto-encoders ( AAE ) combines the techniques of both VAE and GAN network . It imposes a discriminator on the latent space of an auto-encoder to classify the latent distribution p ( z ) and q ( z ) , where the adversarial term serves as the regularizer . Such choice improves the regularization effect of the latent variable and alleviates the mode collapse problem due to an auto-encoder structure . Wasserstein Auto-Encoders ( WAE ) ( Tolstikhin et al. , 2017 ) generalize different regularization approaches on latent space , proposing adversarial method ( WAE-GAN ) and kernel based method ( WAE-MMD ) . It provides AAE with theoretical support as well . Intervention Generative Adversarial Network ( IVGAN ) ( Liang et al. , 2020 ) , another attempt to combine GAN with the encoder structure , stabilizes GAN ’ s training process by intervening on the latent distribution so that the reconstructed data distribution could approach the target more robustly . In this work , we will propose this novel regularization method on AAE models . As basically an autoencoder structure , the key question is how to let the original distribution in latent space approach the prior distribution . We intervened on the encoded data , which inherently brings intervened distribution . We will show that a loss term in our model can be transformed to a certain measure of the overlap among multiple distributions . When minimizing the loss in terms of these constructed variables , we expect not only the original distribution to approach the target one , but the distribution of the intervened data ( we call bridge distribution ) to serve as a guidance to have such approaching process more fast and robust as well . 2 METHODOLOGY . Training a GAN model has always been a challenge ( e.g. , the process is known to be unstable and prone to mode collapse ) . AAE ’ s auto-encoder structure alleviates the problem with a reconstruction term . It first updates the auto-encoder upon the reconstruction loss , then use the discriminator to distinguish the encoded latent variables and generated ones following Gaussian distribution , updating the discriminator and encoder in an adversarial manner . However , such architecture doesn ’ t completely solve GAN ’ s problem . Decreasing the JS divergence has a relatively low approaching efficiency between two distributions . Adversarial training on latent space doesn ’ t avoid this problem at all . Therefore there ’ s a desire to apply a more powerful regularization term to have the approaching process more efficiently and stable . We consider to construct a series of “ bridge distributions ” by certain transformations so that they in a sense lie between the encoded distribution and target one . We let the discriminator to classify these multiple distributions , and update the encoder to let them approach each other . Intuitively such bridge distributions would serve as a guidance to help the latent variables follow the target distribution more quickly and robustly , but we need more theoretical arguments first . Before we expound on further details , we want to turn our attention to how to construct the transformations mentioned above . We first explain the meaning of intervention . It is similar to what is exhibited in IVGAN ( Liang et al. , 2020 ) . For completeness we show it here in our context . Definition 1 Let X be the set of all random variables in Rd on the probability space ( Ω , X , P ) . T is the set of all mappings from Rd to Rd . For a distribution function with support in Rd , we call T ∈ T “ P-intervention ” if X ∼ P ⇒ T ( X ) ∼ P , for all X ∈ X . If S ∈ T satisfies that T ( X ) ∼ P , ∀T ∈ S ⇒ X ∼ P , we denote S “ Complete Intervention Group ” . A rough understanding of the above definition is that complete intervention group let the meet of subsequent multiple distributions embed their final distributing pattern , i.e . P ( x ) . Combining the idea in our model , the intervention can be performed on our encoded data . We hope when these originally different distributions approach each other , they all approach the prior distribution , which is exactly the feature of the transformation proposed above . Therefore the complete intervention group can serve to ensure the identifiability of our method . To be more concrete , we set the P to be standard Gaussian and propose several intervention patterns that possess the above feature . A simplest example of P-intervention is to replace some of the dimension with normal distributed one . Therefore we have our Blockwise substitution : Rd space is divided equally by t parts , where t|d . , For 0 ≤ k ≤ t , let Tk be the transformation that substitute the first k blocks with random variables following standard normal distribution independent with the original variable . Then we get a complete intervention group { Tk } . There ’ s other ways to create intervention groups . For example , we let x be replaced by xcos ( kπ2t ) + zsin ( kπ2t ) , for k = 0 , 1 , ... , t , where z follows standard normal distribution independent with x . We call such construction pattern Radial substitution . Except for the two substitution method , we also apply Stein variational gradient descent ( SVGD ) ( Liu & Wang , 2016 ) to construct the bridge distribution by iteratively updating points . By setting the target distribution P the updated particles will approximately approach P. More details will be specified in the next section . Now we propose our model called Intervention Adversarial Auto-Encoder ( IVAAE ) . [ Figure 1 ] shows its specific structure , which contains an encoder E , decoder G and a discriminator D. Images x is fed into the E to get the encoded latent code z , which is decoded by G to obtain the reconstructed images x . The encoded z is transformed by a complete intervention group Tk ( T0 represent the identical mapping ) to get zk which is discriminated by D with output of size t + 1 . Through maximizing and minimizing alternatively a cross entropy loss , D and E is both updated . Intervention Loss The key distinction of our model from AAE is the intervention loss . More specifically , when encoded latent z = E ( x ) is obtained , for every k , 0 ≤ k ≤ t , we intervene z by Tk to get zk . Then we feed zk into the discriminator to obtain a t + 1 size output d , and compute the cross entropy between d and labels ek , averaging on k , finally to update discriminator by gradient flow . The encoder update in the same manner . Therefore the intervention loss is an unbiased estimator of its theoretical form as follow : LIV ( E , D ) = −Ek∼U [ t ] Ex∼pdata [ log ( Dk ( Tk ( E ( x ) ) ) ) ] = −Ek∼U [ t ] Ez∼pk [ log ( Dk ( z ) ) ] ( 1 ) where Dk , pk represent the ( k + 1 ) th digit of D output , the distribution of zk = Tk ( E ( x ) ) , respectively . Then we give theoretical results that evidence the rationality of our proposed models . Theorem 2 Given t + 1 latent variables { zk } tk=0 with density function support in Rd , and their probabilistic density function { pk } tk=0 , the classifier D is trained to minimize the cross entropy loss as equation 1 under the constraint that ∑t i=0Dj ( z ) = 1 . Then the optimal classifier satisfies : D∗i ( z ) = pi ( z ) ∑t j=0 pj ( z ) And the corresponding optimal intervention loss can be expressed as follows : LIV ( E , D ) = −JS ( p0 , p2 , ... , pt ) + log ( t+ 1 ) where JS ( q1 , q2 , ... , qk ) = 1k ∑k j=1 ( KL ( qj ||q ) ) , q = 1 k ∑k j=1 qj . Proof : See Appendix . Theorem 2 tells us that when the discriminator reaches optimal , the intervention loss can be regarded as a certain distance among multiple distributions . We know when t = 1 , the loss becomes the original JS divergence which conforms with GAN . However , it is known that JS divergence isn ’ t a perfect measure because the gradient would be zero if two variables have distributions with disjoint support . According to the theorem , situation for multiple distributions will suffer less from such problem , for bridge distributions provide greater chances for intersection which consequently produces gradient . Therefore it would be more stable to update the encoder . We now further illustrate the advantage of the multi-object adversarial training . Although GAN ’ s convergence has been theoretically proved ( Goodfellow et al. , 2014 ) , the training process , is wellknown delicate and unstable . One important reason is that the optimal discriminator can always achieve too perfect no matter how close it is between the generated data manifold and the real data manifold ( Arjovsky & Bottou , 2017 ) . One cause for this is that two manifolds is disjoint with probability 1 due to their low dimension ( Arjovsky & Bottou , 2017 ) on the data space . Algorithm 1 Intervention Adversarial Auto-Encoder ( IVAAE ) Input : learning rate : α , dimension of the space : d , number of bridge distributions : t Interventions : Tk , 0 ≤ k ≤ t , hyperparameters : λ , µ1 , µ2 , minibatch size : m. Output : θD , θE , θG 1 : for number of training iterations do 2 : Sample { xj } mj=1 from the training set 3 : Sample { zj } mj=1 from the prior P ( z ) 4 : Compute ẑj = E ( xj ) , j = 1 , ... , m 5 : Compute x̂j = G ( ẑj ) , j = 1 , ... , m 6 : for k = 0 , ... , t do 7 : Compute the intervened latent variables 8 : ẑjk = Tk ( ẑj ) , j = 1 , ... , m 9 : end for 10 : Update the parameters of D by : 11 : θD ← θD + αmtµ1∇θD ∑m j=1 ∑t k=0 logDk ( ẑjk ) 12 : Calculate L̂recon , L̂IV 13 : Update the parameters of G by : 14 : θG ← θG + αm∇θGλL̂recon 15 : Update the parameters of E by : 16 : θE ← θE + αm∇θG { λL̂recon + µ2L̂IV } 17 : end for 18 : return θD , θE , θG Different from GAN , AAE applies discriminator on the latent distribution , therefore what it learns isn ’ t a low-rank manifold , but a continuous distribution with full dimension . But the problem doesn ’ t get solved in AAE setting . We address it in a heuristic way . We run AAE on real-world dataset for 50 epochs , then fix the encoder and decoder and update the discriminator alone . We observe how perfect the discriminator will develop . Figure 2 records the changing curve of adversarial loss every iteration . According to Figrue 2 , the adversarial loss still decreases remarkably toward 0 , especially when latent dimension goes high . That ’ s because of the limit batch size . IVAAE could alleviate this problem by boosting the difficulty for discriminator to classify more than two distributions and equivalently introducing more particles . We apply our models IVAAE . We choose t = 4 , blockwise substitution , with the rest of structure all the same as the AAE model . The loss is recorded with fixed encoder and decoder after 50 epochs , too . We note that although the adversarial loss is decreasing as well , the drop is much more moderate than AAE . We also examine the consequence of such loss decline . We choose the adversarial loss for encoder : −Epdata [ log ( D ( E ( x ) ) ) ] , and record the gradient norm of the encoder after the encoder and decoder fixed . According to Figure 3 , AAE is less stable compared to IVAAE .

Summary: This paper presents Intervention Adversarial auto-encoder to stabilize the training process of the latent variables of adversarial auto-encoders. Intervention Adversarial auto-encoder adopts a sequence of distributions to bridge the distribution of the learned latent variable and its prior distribution. The bridge distributions are implemented by a multi-output discriminator, which guides the initial latent distribution to the target distribution with a stable training process. The paper discusses several different types of the bridge distributions and applies Stein variational gradient descent. Experiments on multiple real-world datasets are conducted.