From Variational to Deterministic Autoencoders

1 INTRODUCTION . Generative models lie at the core of machine learning . By capturing the mechanisms behind the data generation process , one can reason about data probabilistically , access and traverse the lowdimensional manifold the data is assumed to live on , and ultimately generate new data . It is therefore not surprising that generative models have gained momentum in applications such as computer vision ( Sohn et al. , 2015 ; Brock et al. , 2019 ) , NLP ( Bowman et al. , 2016 ; Severyn et al. , 2017 ) , and chemistry ( Kusner et al. , 2017 ; Jin et al. , 2018 ; Gómez-Bombarelli et al. , 2018 ) . Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) cast learning representations for high-dimensional distributions as a variational inference problem . Learning a VAE amounts to the optimization of an objective balancing the quality of samples that are autoencoded through a stochastic encoder–decoder pair while encouraging the latent space to follow a fixed prior distribution . Since their introduction , VAEs have become one of the frameworks of choice among the different generative models . VAEs promise theoretically well-founded and more stable training than Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and more efficient sampling mechanisms than autoregressive models ( Larochelle & Murray , 2011 ; Germain et al. , 2015 ) . However , the VAE framework is still far from delivering the promised generative mechanism , as there are several practical and theoretical challenges yet to be solved . A major weakness of VAEs is ∗Equal contribution . 1An implementation is available at : https : //github.com/ParthaEth/Regularized_ autoencoders-RAE- the tendency to strike an unsatisfying compromise between sample quality and reconstruction quality . In practice , this has been attributed to overly simplistic prior distributions ( Tomczak & Welling , 2018 ; Dai & Wipf , 2019 ) or alternatively , to the inherent over-regularization induced by the KL divergence term in the VAE objective ( Tolstikhin et al. , 2017 ) . Most importantly , the VAE objective itself poses several challenges as it admits trivial solutions that decouple the latent space from the input ( Chen et al. , 2017 ; Zhao et al. , 2017 ) , leading to the posterior collapse phenomenon in conjunction with powerful decoders ( van den Oord et al. , 2017 ) . Furthermore , due to its variational formulation , training a VAE requires approximating expectations through sampling at the cost of increased variance in gradients ( Burda et al. , 2015 ; Tucker et al. , 2017 ) , making initialization , validation , and annealing of hyperparameters essential in practice ( Bowman et al. , 2016 ; Higgins et al. , 2017 ; Bauer & Mnih , 2019 ) . Lastly , even after a satisfactory convergence of the objective , the learned aggregated posterior distribution rarely matches the assumed latent prior in practice ( Kingma et al. , 2016 ; Bauer & Mnih , 2019 ; Dai & Wipf , 2019 ) , ultimately hurting the quality of generated samples . All in all , much of the attention around VAEs is still directed towards “ fixing ” the aforementioned drawbacks associated with them . In this work , we take a different route : we question whether the variational framework adopted by VAEs is necessary for generative modeling and , in particular , to obtain a smooth latent space . We propose to adopt a simpler , deterministic version of VAEs that scales better , is simpler to optimize , and , most importantly , still produces a meaningful latent space and equivalently good or better samples than VAEs or stronger alternatives , e.g. , Wasserstein Autoencoders ( WAEs ) ( Tolstikhin et al. , 2017 ) . We do so by observing that , under commonly used distributional assumptions , training a stochastic encoder–decoder pair in VAEs does not differ from training a deterministic architecture where noise is added to the decoder ’ s input . We investigate how to substitute this noise injection mechanism with other regularization schemes in the proposed deterministic Regularized Autoencoders ( RAEs ) , and we thoroughly analyze how this affects performance . Finally , we equip RAEs with a generative mechanism via a simple ex-post density estimation step on the learned latent space . In summary , our contributions are as follows : i ) we introduce the RAE framework for generative modeling as a drop-in replacement for many common VAE architectures ; ii ) we propose an ex-post density estimation scheme which greatly improves sample quality for VAEs , WAEs and RAEs without the need to retrain the models ; iii ) we conduct a rigorous empirical evaluation to compare RAEs with VAEs and several baselines on standard image datasets and on more challenging structured domains such as molecule generation ( Kusner et al. , 2017 ; Gómez-Bombarelli et al. , 2018 ) . 2 VARIATIONAL AUTOENCODERS . For a general discussion , we consider a collection of high-dimensional i.i.d . samples X = { xi } Ni=1 drawn from the true data distribution pdata ( x ) over a random variable X taking values in the input space . The aim of generative modeling is to learn from X a mechanism to draw new samples xnew ∼ pdata . Variational Autoencoders provide a powerful latent variable framework to infer such a mechanism . The generative process of the VAE is defined as znew ∼ p ( Z ) , xnew ∼ pθ ( X |Z = znew ) ( 1 ) where p ( Z ) is a fixed prior distribution over a low-dimensional latent space Z . A stochastic decoder Dθ ( z ) = x ∼ pθ ( x | z ) = p ( X | gθ ( z ) ) ( 2 ) links the latent space to the input space through the likelihood distribution pθ , where gθ is an expressive non-linear function parameterized by θ.2 As a result , a VAE estimates pdata ( x ) as the infinite mixture model pθ ( x ) = ∫ pθ ( x | z ) p ( z ) dz . At the same time , the input space is mapped to the latent space via a stochastic encoder Eφ ( x ) = z ∼ qφ ( z |x ) = q ( Z | fφ ( x ) ) ( 3 ) where qφ ( z |x ) is the posterior distribution given by a second function fφ parameterized by φ. Computing the marginal log-likelihood log pθ ( x ) is generally intractable . One therefore follows a variational approach , maximizing the evidence lower bound ( ELBO ) for a sample x : log pθ ( x ) ≥ ELBO ( φ , θ , x ) = Ez∼qφ ( z |x ) log pθ ( x | z ) −KL ( qφ ( z |x ) ||p ( z ) ) ( 4 ) 2With slight abuse of notation , we use lowercase letters for both random variables and their realizations , e.g. , pθ ( x | z ) instead of p ( X |Z = z ) , when it is clear to discriminate between the two . Maximizing Eq . 4 over data X w.r.t . model parameters φ , θ corresponds to minimizing the loss argmin φ , θ Ex∼pdata LELBO = Ex∼pdata LREC + LKL ( 5 ) where LREC and LKL are defined for a sample x as follows : LREC = −Ez∼qφ ( z |x ) log pθ ( x | z ) LKL = KL ( qφ ( z |x ) ||p ( z ) ) ( 6 ) Intuitively , the reconstruction loss LREC takes into account the quality of autoencoded samples x through Dθ ( Eφ ( x ) ) , while the KL-divergence term LKL encourages qφ ( z |x ) to match the prior p ( z ) for each z which acts as a regularizer during training ( Hoffman & Johnson , 2016 ) . 2.1 PRACTICE AND SHORTCOMINGS OF VAES . To fit a VAE to data through Eq . 5 one has to specify the parametric forms for p ( z ) , qφ ( z |x ) , pθ ( x | z ) , and hence the deterministic mappings fφ and gθ . In practice , the choice for the above distributions is guided by trading off computational complexity with model expressiveness . In the most commonly adopted formulation of the VAE , qφ ( z |x ) and pθ ( x | z ) are assumed to be Gaussian : Eφ ( x ) ∼ N ( Z|µφ ( x ) , diag ( σφ ( x ) ) ) Dθ ( Eφ ( x ) ) ∼ N ( X|µθ ( z ) , diag ( σθ ( z ) ) ) ( 7 ) with means µφ , µθ and covariance parameters σφ , σθ given by fφ and gθ . In practice , the covariance of the decoder is set to the identity matrix for all z , i.e. , σθ ( z ) = 1 ( Dai & Wipf , 2019 ) . The expectation of LREC in Eq . 6 must be approximated via k Monte Carlo point estimates . It is expected that the quality of the Monte Carlo estimate , and hence convergence during learning and sample quality increases for larger k ( Burda et al. , 2015 ) . However , only a 1-sample approximation is generally carried out ( Kingma & Welling , 2014 ) since memory and time requirements are prohibitive for large k. With the 1-sample approximation , LREC can be computed as the mean squared error between input samples and their mean reconstructions µθ by a decoder that is deterministic in practice : LREC = ||x− µθ ( Eφ ( x ) ) ||22 ( 8 ) Gradients w.r.t . the encoder parameters φ are computed through the expectation of LREC in Eq . 6 via the reparametrization trick ( Kingma & Welling , 2014 ) where the stochasticity of Eφ is relegated to an auxiliary random variable which does not depend on φ : Eφ ( x ) = µφ ( x ) + σφ ( x ) , ∼ N ( 0 , I ) ( 9 ) where denotes the Hadamard product . An additional simplifying assumption involves fixing the prior p ( z ) to be a d-dimensional isotropic Gaussian N ( Z |0 , I ) . For this choice , the KL-divergence for a sample x is given in closed form : 2LKL = ||µφ ( x ) ||22 + d+ ∑d i σφ ( x ) i − logσφ ( x ) i . While the above assumptions make VAEs easy to implement , the stochasticity in the encoder and decoder are still problematic in practice ( Makhzani et al. , 2016 ; Tolstikhin et al. , 2017 ; Dai & Wipf , 2019 ) . In particular , one has to carefully balance the trade-off between the LKL term and LREC during optimization ( Dai & Wipf , 2019 ; Bauer & Mnih , 2019 ) . A too-large weight on the LKL term can dominateLELBO , having the effect of over-regularization . As this would smooth the latent space , it can directly affect sample quality in a negative way . Heuristics to avoid this include manually finetuning or gradually annealing the importance of LKL during training ( Bowman et al. , 2016 ; Bauer & Mnih , 2019 ) . We also observe this trade-off in a practical experiment in Appendix A . Even after employing the full array of approximations and “ tricks ” to reach convergence of Eq . 5 for a satisfactory set of parameters , there is no guarantee that the learned latent space is distributed according to the assumed prior distribution . In other words , the aggregated posterior distribution qφ ( z ) = Ex∼pdataq ( z|x ) has been shown not to conform well to p ( z ) after training ( Tolstikhin et al. , 2017 ; Bauer & Mnih , 2019 ; Dai & Wipf , 2019 ) . This critical issue severely hinders the generative mechanism of VAEs ( cf . Eq . 1 ) since latent codes sampled from p ( z ) ( instead of q ( z ) ) might lead to regions of the latent space that are previously unseen toDθ during training . This results in generating out-of-distribution samples . We refer the reader to Appendix H for a visual demonstration of this phenomenon on the latent space of VAEs . We analyze solutions to this problem in Section 4 .

Summary: This paper propose an extension to deterministic autoencoders. Motivated from VAEs, the authors propose RAEs, which replace the noise injection in the encoders of VAEs with an explicit regularization term on the latent representations. As a result, the model becomes a deterministic autoencoder with a L_2 regularization on the latent representation z. To make the model generalize well, the authors also add a decoder regularization term L_REG. In addition, due to the encoder in RAE is deterministic, the authors propose several ex-post density estimation techniques for generating samples.