The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders 

1 INTRODUCTION . Many modern generative models of choice ( e.g . Generative Adversarial Networks ( Goodfellow et al. , 2014 ) , Variational Autoencoders ( Kingma & Welling , 2013 ) ) are modeled as non-linear , possibly stochastic transformations of a simple latent distribution ( e.g . a standard Gaussian ) . A particularly common task is modeling the inferential ( encoder ) direction : that is , modeling the posterior distribution on the latents z given an observable sample x . Such a task is useful both at train time and at test time . At train time , fitting generative models like variational autoencoders via maximum likelihood often relies on variational methods , which require the joint training of a generative model ( i.e . generator/decoder ) , as well as an inference model ( i.e . encoder ) which models the posterior distribution of the latent given the observables . At test time , the posterior distribution very often has some practical use , e.g . useful , potentially interpretable feature embeddings for data ( Salimans et al. , 2016 ; Berthelot et al. , 2018 ) , “ intervening ” on the latent space to change the sample in some targeted manner ( Shen et al. , 2020 ) , etc . As such , the question of the “ complexity '' of the inference model ( i.e . number of parameters to represent it using a neural network-based encoder ) as a function of the “ complexity '' of the forward model is of paramount importance , so that training is computationally tractable : Question : How complex does the inference ( encoder ) model need to be relative to the complexity of the generative ( decoder ) model ? In this paper we identify an important property of the generative direction governing the complexity of the inference direction for variational autoencoders : bijectivity/invertibility of the mean of the generative direction . We prove that when the mean of the generative direction is invertible , the complexity of the inference direction is not much greater than the complexity of the generative direction . Conversely , when the mean of the generative direction is not invertible , modulo standard computational complexity conjectures from cryptography , we can exhibit instances where the inference direction has to be much more complex . On the mathematical level , our techniques involve a neural simulation of a Langevin random walk to sample from the posterior of the latent variables and uncover novel connections between Langevin diffusions and ( hierarchical ) deep latent Gaussians . On the lower bound side , we provide a reduction from the existence of one-way Boolean permutations in computational complexity : that is , permutations that are easy to calculate , but hard to invert . We show that the existence of a small encoder for non-invertible generators would allow us to design an invertor for any Boolean permutation , thus violating the existence a one-way permutation . This is the first time such ideas have been applied to generative models . Note that a non-invertible generator results in a distribution supported on a lower dimensional manifold . Thus , our results show that learning deep generative models is harder when data lies on a sufficiently complex low-dimensional manifold , corroborating similarly flavored empirical observations ( Dai & Wipf , 2019 ; Arjovsky et al. , 2017 ) . 2 OUR RESULTS . The Variational Autoencoder ( VAE ) ( Kingma & Welling , 2013 ) is one of the most commonly used paradigms in generative models . It ’ s trained by fitting a generator which maps latent variables z to observables x , denoted by pθ ( x|z ) , as well as an encoder which maps the observables to the latent space , denoted by qφ ( z|x ) . Here φ and θ are the encoder parameters and generator parameters respectively . Given n training samples { x ( i ) } ni=1 , the VAE objective is given by max φ , θ 1 n n∑ i=1 Ez∼qφ ( .|x ( i ) ) [ log pθ ( x ( i ) |z ) ] −KL ( qφ ( z|x ( i ) ) ||p ( z ) ) where p ( z ) is typically chosen to be a standard Gaussian . This loss can be viewed as a variational relaxation of the maximum likelihood objective , where the encoder qφ , in the limit of infinite representational power , is intended to model the posterior distribution pθ ( z|x ( i ) ) . Setup : We will consider a setting in which the data distribution itself is given by some ground-truth generator G : Rdl → Rdo , and ask how complex ( in terms of number of parameters ) the encoder needs to be ( as a function of the number of parameters of G ) , s.t . it approximates the posterior distribution p ( z|x ) of the generator . We will consider two standard probabilistic models for the generator/encoder respectively . Definition 1 ( Latent Gaussian ) . A latent Gaussian is the conditional distribution given by a stochastic pushforward of a Gaussian distribution . That is , for latent variable z ∈ Rdl and observable x ∈ Rdo , for a neural network G : Rdl → Rdo and noise parameter β2 , we have p ( x|z ) = N ( G ( z ) , β2Ido ) and p ( z ) = N ( 0 , Idl ) . In other words , a sample from this distribution can be generated by sampling independently z ∼ N ( 0 , Idl ) and ξ ∼ N ( 0 , β2Ido ) and outputting x = G ( z ) + ξ . This is a standard neural parametrization of a generator with ( scaled ) identity covariance matrix , a fairly common choice in practical implementations of VAEs ( Kingma & Welling , 2013 ; Dai & Wipf , 2019 ) . We will also define a probabilistic model which is a composition of latent Gaussians ( i.e . consists of multiple stochastic layers ) , which is also common , particularly when modeling encoders in VAEs , as they can model potentially non-Gaussian posteriors ( Burda et al. , 2015 ; Rezende et al. , 2014 ) : Definition 2 ( Deep Latent Gaussian ) . A deep latent Gaussian is the conditional distribution given by a sequence of stochastic pushforwards of a Gaussian distribution . That is , for observable z0 ∈ Rd0 and latent variables { zi ∈ Rdi } Li=1 , for neural networks { Gi : Rdi−1 → Rdi } Li=1 and noise parameters { β2i } Li=1 , the conditional distribution p ( zL|z0 ) is a deep latent Gaussian when p ( zi|zi−1 ) = N ( Gi ( zi−1 ) , β2i Idi ) , ∀i ∈ [ L ] and p ( z0 ) = N ( 0 , Id0 ) . In other words , a deep latent Gaussian is a distribution , which can be sampled by ancestral sampling , one layer at a time . Note that this class of distributions is convenient as a choice for an encoder in a VAE , since compositions are amenable to the reparametrization trick of Kingma & Welling ( 2013 ) —the randomness for each of the layers can be “ presampled ” and appropriately transformed ( Burda et al. , 2015 ; Rezende et al. , 2014 ) . Then , we ask the following : Question : If a VAE generator is modeled as a latent Gaussian ( that is , p ( x|z ) ≡ N ( G ( z ) , β2I ) ) , s.t . the corresponding G has at most N parameters , and we wish to approximate the posterior p ( z|x ) by a deep latent Gaussian s.t . the total size of the networks in it have at most N ′ parameters , how large must N ′ be as function of N ? We will work in the setting dl = do = d , and prove a dichotomy based on the invertibility of G : namely , if G : Rd → Rd is bijective , and β ≤ O ( 1 d1.5 √ log d/ ) , the poste- rior p ( z|x ) can be -approximated in total variation distance by a deep latent Gaussian of size N ′ = O ( N · poly ( d , 1/β , 1/ ) ) . Thus , if the neural network G is invertible , and for a fixed and a small-enough variance term β2 , we can approximate the posterior with a deep latent Gaussian polynomially larger than G. On the other hand , if G is not bijective , if one-way-functions exist ( a widely believed computational complexity conjecture ) , we will show there exists a VAE generator G : Rd → Rd of size polynomial in d , for which the posterior p ( z|x ) can not be approximated in total variation distance for even an inverse polynomial fraction of inputs x , unless the inferential network is of size exponential in d. Remark 1 : Note that it is in fact sufficient to investigate only the case where dl = do = d. Let us elaborate why : • For Theorem 1 , a function G : Rdl → Rdo can be bijective , Lipschitz and strongly invertible ( i.e . satisfies Assumptions 1 and 2 ) only if dl = do , so Theorem 1 ( upper bound ) would not make any sense unless the dimensions are equal . ( Note : there are bijective maps between , say , R and R2 , e.g . space-filling curves , but these can not be Lipschitz , as Lipschitz maps preserve the Hausdorff dimension , and the Hausdorff dimensions of R and R2 are 1 and 2 respectively ) . • Theorem 2 is a lower bound – meaning we aim to exhibit an example of a hard instance when G is not bijective . Having exhibited a hard instance for the dl = do = d case , it is trivially easy to construct a hard instance in the theorem where the output dimension is larger—which is the more common setting in practice . To do so , consider a circuit C̃ : { ±1 } dl → { ±1 } do , which equals to C , the one-way-circuit from Conjecture 1 on the first dl coordinates , and is equal to 1 ( i.e . is constant ) on the last do − dl coordinates . Then C̃ is one-way too—since the last do − dl values are just fixed to 1 , if we can invert it , we can invert C too . The reduction in the proof of Theorem 2 then can just be performed with C̃ instead , giving a lower bound instance for the dl < do case . 2.1 UPPER BOUNDS FOR BIJECTIVE GENERATORS . We first lay out the assumptions on the mapG . The first is a quantitative characterization of bijectivity ; and the second requires upper bounds on the derivatives of G up to order 3 . We also have a centering assumption . We state these below . Assumption 1 ( Strong invertibility ) . We will assume that the latent and observable spaces have the same dimension ( denoted d ) , and G : Rd → Rd is bijective . Moreover , we will assume there exists a positive constant m > 0 such that ∀z1 , z2 ∈ Rd , ‖G ( z1 ) −G ( z2 ) ‖ ≥ m · ‖z1 − z2‖ . Remark 2 : This is a stronger quantitative version of invertibility . Furthermore , the infinitesimal version of this condition ( i.e . ‖z1−z2‖ → 0 ) implies that the smallest magnitude of the singular values of the Jacobian at any point is lower bounded by m , that is ∀z ∈ Rd , min i∈ [ d ] |σi ( JG ( z ) ) | ≥ m > 0 . Since m is strictly positive , this in particular means that the Jacobian is full rank everywhere . Remark 3 : Note , we do not require that G is layerwise invertible ( i.e . that the each map from one layer to the next is invertible ) – if that is the case , at least in the limit β → 0 , the existence of an inference decoder of comparable size toG is rather obvious : we simply invert each layer one at a time . This is important , as many architectures based on convolutions perform operations which increase the dimension ( i.e . map from a lower to a higher dimensional space ) , followed by pooling ( which decrease the dimension ) . Nevertheless , it has been observed that these architectures are invertible in practice—Lipton & Tripathi ( 2017 ) manage to get almost 100 % success at inverting an off-the-shelf trained model—thus justifying this assumption . Assumption 2 ( Smoothness ) . There exists a finite positive constant M > 0 such that : ∀z1 , z2 ∈ Rd , ‖G ( z1 ) −G ( z2 ) ‖ ≤M · ‖z1 − z2‖ Moreover , we will assume that G has continuous partial derivatives up to order 3 at every z ∈ Rd and the derivatives are bounded by finite positive constants M2 and M3 as ∀z ∈ Rd , ∥∥∇2G ( z ) ∥∥ op ≤M2 < ∞ , ∥∥∇3G ( z ) ∥∥ op ≤M3 < ∞ Remark 4 : This is a mild assumption , stating that the map G is smooth to third order . The infinitesimal version of this means that the largest magnitude of the singular values of the Jacobian at any point is upper bounded by M , that is ∀z ∈ Rd , max i∈ [ d ] |σi ( JG ( z ) ) | = ‖JG ( z ) ‖op ≤M < ∞ . Remark 5 : A neural network with activation function σ will satisfy this assumption when σ : R→ R is Lipschitz , and maxa |σ′ ( a ) | & maxa |σ′′ ( a ) | are finite . Assumption 3 ( Centering ) . The map G : Rd → Rd satisfies G ( 0 ) = 0 . Remark 6 : This assumption is for convenience of stating the bounds—we effectively need the “ range ” of majority of the samples x under the distribution of the generator . All the results can be easily restated by including a dependence on ‖G ( 0 ) ‖ . Our main result is then stated below . Throughout , the O ( . ) notation hides dependence on the map constants , namely m , M , M2 , M3 . We will denote by dTV ( p , q ) the total variation distance between the distributions p , q. Theorem 1 ( Main , invertible generator ) . Consider a VAE generator given by a latent Gaussian with µ = 0 , Σ = I , noise parameter β2 and generator G : Rd → Rd satisfying Assumptions 1 and 2 , which has N parameters and a differentiable activation function σ . Then , for β ≤ O 1 d1.5 √ log d ( 1 ) there exists a deep latent Gaussian withN ′ = O ( N · poly ( d , 1β , 1 ) ) parameters and activation functions { σ , σ′ , ρ } and a neural network φ with O ( N ′ ) parameters and activation functions { σ , σ′ , ρ } , where ρ ( x ) = x2 , such that with probability 1−exp ( −O ( d ) ) over a sample x from the VAE generator , φ ( x ) produces values for the parameters of the deep latent Gaussian such that the distribution q ( z|x ) it specifies satisfies dTV ( q ( z|x ) , p ( z|x ) ) ≤ . Remark 7 : Having an encoder q ( z|x ) with parameters produced by a neural network taking x as input is fairly standard in practice—it is known as amortized variational inference . ( Mnih & Gregor , 2014 ; Kingma & Welling , 2013 ; Rezende et al. , 2014 ) Remark 8 : The addition of ρ in the activation functions is for convenience of stating the bound . Using usual techniques in universal approximation it can be simulated using any other smooth activation , see Appendix G .

Summary: The paper answers the question how complex inference models need to be to accurately estimate posterior distributions. The conclusion is when a latent Gaussian model with N parameters satisfies (1) strong invertibility, (2) 3-th smoothness, the posterior can be approximated by a deep latent Gaussian model with O(N) parameters. A special case is also given to show that strong invertibility is necessary to the conclusion.