Implicit Autoencoders

1 INTRODUCTION . Deep generative models have achieved remarkable success in recent years . One of the most successful models is the generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) , which employs a two player min-max game . The generative model , G , samples the noise vector z ∼ p ( z ) and generates the sample G ( z ) . The discriminator , D ( x ) , is trained to identify whether a point x comes from the data distribution or the model distribution ; and the generator is trained to maximally confuse the discriminator . The cost function of GAN is min G max D Ex∼pdata [ logD ( x ) ] + Ez∼p ( z ) [ log ( 1−D ( G ( z ) ) ] . ( 1 ) GANs can be viewed as a general framework for learning implicit distributions ( Mohamed & Lakshminarayanan , 2016 ; Huszár , 2017 ) . Implicit distributions are probability distributions that are obtained by passing a noise vector through a deterministic function that is parametrized by a neural network . In the probabilistic machine learning problems , implicit distributions trained with the GAN framework can learn distributions that are more expressive than the tractable distributions trained with the maximum-likelihood framework . Variational autoencoders ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) are another successful generative models that use neural networks to parametrize the posterior and the conditional likelihood distributions . Both networks are jointly trained to maximize a variational lower bound on the data log-likelihood . One of the limitations of VAEs is that they learn factorized distributions for both the posterior and the conditional likelihood distributions . In this paper , we propose the “ implicit autoencoder ” ( IAE ) that uses implicit distributions for learning more expressive posterior and conditional likelihood distributions . Learning a more expressive posterior will result in a tighter variational bound ; and learning a more expressive conditional likelihood distribution will result in a global vs. local decomposition of information between the prior and the conditional likelihood . This enables the latent code to only capture the information that we care about such as the high-level and abstract information , while the remaining low-level information of data is separately captured by the noise vector of the implicit decoder . Implicit distributions have been previously used in learning generative models in works such as adversarial autoencoders ( AAE ) ( Makhzani et al. , 2015 ) , adversarial variational Bayes ( AVB ) ( Mescheder et al. , 2017 ) , ALI ( Dumoulin et al. , 2016 ) , BiGAN ( Donahue et al. , 2016 ) and other works such as ( Huszár , 2017 ; Tran et al. , 2017 ) . The global vs. local decomposition of information has also been studied in previous works such as PixelCNN autoencoders ( van den Oord et al. , 2016 ) , PixelVAE ( Gulrajani et al. , 2016 ) , variational lossy autoencoders ( Chen et al. , 2016b ) , PixelGAN autoencoders ( Makhzani & Frey , 2017 ) , or other works such as ( Bowman et al. , 2015 ; Graves et al. , 2018 ; Alemi et al. ) . In the next section , we first propose the IAE and then establish its connections with the related works . 2 IMPLICIT AUTOENCODERS . Let x be a datapoint that comes from the data distribution pdata ( x ) . The encoder of the implicit autoencoder ( Figure 1 ) defines an implicit variational posterior distribution q ( z|x ) with the function ẑ = fφ ( x , ) that takes the input x along with the input noise vector and outputs ẑ . The decoder of the implicit autoencoder defines an implicit conditional likelihood distribution p ( x|z ) with the function x̂ = gθ ( ẑ , n ) that takes the code ẑ along with the latent noise vector n and outputs a reconstruction of the image x̂ . In this paper , we refer to ẑ as the latent code or the global code , and refer to the latent noise vector n as the local code . Let p ( z ) be a fixed prior distribution , p ( x , z ) = p ( z ) p ( x|z ) be the joint model distribution , and p ( x ) be the model distribution . The variational distribution q ( z|x ) induces the joint data distribution q ( x , z ) , the aggregated posterior distribution q ( z ) , and the inverse posterior/encoder distribution q ( x|z ) as follows : q ( x , z ) = q ( z|x ) pdata ( x ) ẑ ∼ q ( z ) = ∫ x q ( x , z ) dx q ( x|z ) = q ( x , z ) q ( z ) ( 2 ) Maximum likelihood learning is equivalent to matching the model distribution p ( x ) to the data distribution pdata ( x ) ; and learning with variational inference is equivalent to matching the joint model distribution p ( x , z ) to the joint data distribution q ( x , z ) . The entropy of the data distributionHdata ( x ) , the entropy of the latent code H ( z ) , the mutual information I ( x ; z ) , and the conditional entropies H ( x|z ) andH ( z|x ) are all defined under the joint data distribution q ( x , z ) and its marginals pdata ( x ) and q ( z ) . Using the aggregated posterior distribution q ( z ) , we can define the joint reconstruction distribution r ( x , z ) and the aggregated reconstruction distribution r ( x ) as follows : r ( x , z ) = q ( z ) p ( x|z ) x̂ ∼ r ( x ) = ∫ z r ( x , z ) dz ( 3 ) Note that in general we have r ( x , z ) 6= q ( x , z ) 6= p ( x , z ) , q ( z ) 6= p ( z ) , and r ( x ) 6= pdata ( x ) 6= p ( x ) . We now use different forms of the aggregated evidence lower bound ( ELBO ) to describe the IAE and establish its connections with VAEs and AAEs . Ex∼pdata ( x ) [ log p ( x ) ] ≥ −Ex∼pdata ( x ) [ Eq ( z|x ) [ − log p ( x|z ) ] ] ︸ ︷︷ ︸ VAE Reconstruction −Ex∼pdata ( x ) [ KL ( q ( z|x ) ‖p ( z ) ) ] ︸ ︷︷ ︸ VAE Regularization ( 4 ) = −Ex∼pdata ( x ) [ Eq ( z|x ) [ − log p ( x|z ) ] ] ︸ ︷︷ ︸ AAE Reconstruction −KL ( q ( z ) ‖p ( z ) ) ︸ ︷︷ ︸ AAE Regularization − I ( z ; x ) ︸ ︷︷ ︸ Mutual Info . ( 5 ) = −Ez∼q ( z ) [ KL ( q ( x|z ) ‖p ( x|z ) ) ] ︸ ︷︷ ︸ IAE Reconstruction −KL ( q ( z ) ‖p ( z ) ) ︸ ︷︷ ︸ IAE Regularization − Hdata ( x ) ︸ ︷︷ ︸ Entropy of Data ( 6 ) = −KL ( q ( x , z ) ‖r ( x , z ) ) ︸ ︷︷ ︸ IAE Reconstruction −KL ( q ( z ) ‖p ( z ) ) ︸ ︷︷ ︸ IAE Regularization − Hdata ( x ) ︸ ︷︷ ︸ Entropy of Data ( 7 ) See Appendix A for the proof . The standard formulation of the VAE ( Equation 4 ) only enables us to learn factorized Gaussian posterior and conditional likelihood distributions . The AAE ( Makhzani et al. , 2015 ) ( Equation 5 ) and the AVB ( Mescheder et al. , 2017 ) enable us to learn implicit posterior distributions , but their conditional likelihood distribution is still a factorized Gaussian distribution . However , the IAE enables us to learn implicit distributions for both the posterior and the conditional likelihood distributions . Similar to VAEs and AAEs , the IAE ( Equation 6 ) has a reconstruction cost function and a regularization cost function , but trains each of them with a GAN . The IAE reconstruction cost is Ez∼q ( z ) [ KL ( q ( x|z ) ‖p ( x|z ) ) ] . The standard VAE uses a factorized decoder , which has a very limited stochasticity . Thus , the standard VAE performs almost deterministic reconstructions by learning to invert the deterministic mapping of the encoder . The IAE , however , uses a powerful implicit decoder to perform stochastic reconstructions , by learning to match the expressive decoder distribution p ( x|z ) to the inverse encoder distribution q ( x|z ) . We note that there are other variants of VAEs that can also learn expressive decoder distributions by using autoregressive neural networks . We will discuss these models later in this section . Equation 8 contrasts the reconstruction cost of standard autoencoders that is used in VAEs/AAEs , with the reconstruction cost of IAEs . Ex∼pdata ( x ) [ Eq ( z|x ) [ − log p ( x|z ) ] ] ︸ ︷︷ ︸ AE Reconstruction = Ez∼q ( z ) [ KL ( q ( x|z ) ‖p ( x|z ) ) ] ︸ ︷︷ ︸ IAE Reconstruction + H ( x|z ) ︸ ︷︷ ︸ Cond . Entropy ( 8 ) We can see from Equation 8 that similar to IAEs , the reconstruction cost of the autoencoder encourages matching the decoder distribution to the inverse encoder distribution . But in autoencoders , the cost function also encourages minimizing the conditional entropy H ( x|z ) , or maximizing the mutual information I ( x , z ) . Maximizing the mutual information in autoencoders enforces the latent code to capture both the global and local information . In contrast , in IAEs , the reconstruction cost does not penalize the encoder for losing the local information , as long as the decoder can invert the encoder distribution . In order to minimize the reconstruction cost function of the IAE , we re-write it in the form of a distribution matching cost function between the joint data distribution and the joint reconstruction distribution KL ( q ( x , z ) ‖r ( x , z ) ) ( Equation 7 ) . This KL divergence is approximately minimized with the reconstruction GAN . The IAE has also a regularization cost function KL ( q ( z ) ‖p ( z ) ) that matches the aggregated posterior distribution with a fixed prior distribution . This is the same regularization cost function used in AAEs ( Equation 5 ) , and is approximately minimized with the regularization GAN . Note that the last term in Equation 7 is the entropy of the data distribution that is fixed . Training Process . We now describe the training process . We pass a given point x ∼ pdata ( x ) through the encoder and the decoder to obtain ẑ ∼ q ( z ) and x̂ ∼ r ( x ) . We now train the discriminator of the reconstruction GAN to identify the positive example ( x , ẑ ) from the negative example ( x̂ , ẑ ) . Suppose this discriminator function at its optimality is D∗ ( x , z ) . We try to confuse this discriminator by backpropagating through the negative example ( x̂ , ẑ ) 1 and updating the encoder and decoder weights . 1We could also back-propagate through both the positive example ( x , ẑ ) and the negative example ( x̂ , ẑ ) to optimize the reconstruction cost . We observed that in the deterministic decoder case , it does not make any difference in the performance ; but in the stochastic decoder case , it results in learning less useful representations , since it enables the encoder to change its weights freely in a way that is not compatible with the decoder . More specifically , the generative loss of the reconstruction GAN is T ∗ ( x , z ) = − logD∗ ( x , z ) , which defines the reconstruction cost of the IAE . We use the re-parametrization trick to update the encoder and decoder weights by computing the unbiased Monte Carlo estimate of the gradient of the reconstruction cost T ∗ ( x , z ) with respect to ( φ , θ ) as follows : ∇ ( φ , θ ) Er ( x , z ) [ T ∗ ( x , z ) ] = ∇ ( φ , θ ) Eqφ ( z ) pθ ( x|z ) [ T ∗ ( x , z ) ] ( 9 ) = Ex∼pdata ( x ) E En [ ∇ ( φ , θ ) T ∗ ( gθ ( fφ ( x , ) , n ) , fφ ( x , ) ) ] ( 10 ) We call this process the adversarial reconstruction . Similarly , we train the discriminator of the regularization GAN to identify the positive example z ∼ p ( z ) from the negative example ẑ ∼ q ( z ) . This discriminator now defines the regularization cost function , which can provide us with a gradient to update only the encoder weights . We call this process the adversarial regularization . Optimizing the adversarial regularization and reconstruction cost functions encourages p ( x|z ) = q ( x|z ) and p ( z ) = q ( z ) , which results in the model distribution capturing the data distribution p ( x ) = pdata ( x ) . We note that in this work , we use the original formulation of GANs ( Goodfellow et al. , 2014 ) to match the distributions . As a result , the gradient that we obtain from the adversarial training , only approximately follows the gradient of the variational bound on the data log-likelihood . However , as shown in ( Nowozin et al. , 2016 ) , the objective of the GAN can be modified to optimize any f -divergence including the KL divergence . Bits-Back Interpretation of the IAE Objective . In Appendix B , we describe an information theoretic interpretation of the ELBO of IAEs ( Equation 7 ) using the Bits-Back coding argument ( Hinton & Van Camp , 1993 ; Chen et al. , 2016b ; Graves et al. , 2018 ) . Global vs. Local Decomposition of Information in IAEs . In IAEs , the dimension of the latent vector along with its prior distribution defines the capacity of the latent code , and the dimension of the latent noise vector along with its distribution defines the capacity of the implicit decoder . By adjusting these dimensions and distributions , we can have a full control over the decomposition of information between the latent code and the implicit decoder . In one extreme case , by removing the noise vector , we can have a fully deterministic autoencoder that captures all the information by its latent code . In the other extreme case , we can remove the global latent code and have an unconditional implicit distribution that can capture the whole data distribution by itself . The global vs. local decomposition of information in IAEs is further discussed in Appendix C from an information theoretic perspective . In IAEs , we can choose to only optimize the reconstruction cost or both the reconstruction and the regularization costs . In the following , we discuss four special cases of the IAE and establish connections with the related methods .

Summary: The paper proposed an implicit auto-encoder, featuring both the encoder and decoder constituted by implicit distributions. Adversary training is used train the models, similar to the technique used in the AVB model. The main difference with AVB is the use of an implicit decoder, which endows the model with the ability to disentangle the data into high-level abstract representation and local representation. Although sharing some similarities, the extension of using implicit decoder is interesting, and leading to some interesting results.