The Effects of Invertibility on the Representational Complexity of Encoders in Variational Autoencoders 

1 INTRODUCTION . Many modern generative models of choice ( e.g . Generative Adversarial Networks ( Goodfellow et al. , 2014 ) , Variational Autoencoders ( Kingma & Welling , 2013 ) ) are modeled as non-linear , possibly stochastic transformations of a simple latent distribution ( e.g . a standard Gaussian ) . A particularly common task is modeling the inferential ( encoder ) direction : that is , modeling the posterior distribution on the latents z given an observable sample x . Such a task is useful both at train time and at test time . At train time , fitting generative models like variational autoencoders via maximum likelihood often relies on variational methods , which require the joint training of a generative model ( i.e . generator/decoder ) , as well as an inference model ( i.e . encoder ) which models the posterior distribution of the latent given the observables . At test time , the posterior distribution very often has some practical use , e.g . useful , potentially interpretable feature embeddings for data ( Salimans et al. , 2016 ; Berthelot et al. , 2018 ) , “ intervening ” on the latent space to change the sample in some targeted manner ( Shen et al. , 2020 ) , etc . As such , the question of the “ complexity '' of the inference model ( i.e . number of parameters to represent it using a neural network-based encoder ) as a function of the “ complexity '' of the forward model is of paramount importance , so that training is computationally tractable : Question : How complex does the inference ( encoder ) model need to be relative to the complexity of the generative ( decoder ) model ? In this paper we identify an important property of the generative direction governing the complexity of the inference direction for variational autoencoders : bijectivity/invertibility of the mean of the generative direction . We prove that when the mean of the generative direction is invertible , the complexity of the inference direction is not much greater than the complexity of the generative direction . Conversely , when the mean of the generative direction is not invertible , modulo standard computational complexity conjectures from cryptography , we can exhibit instances where the inference direction has to be much more complex . On the mathematical level , our techniques involve a neural simulation of a Langevin random walk to sample from the posterior of the latent variables and uncover novel connections between Langevin diffusions and ( hierarchical ) deep latent Gaussians . On the lower bound side , we provide a reduction from the existence of one-way Boolean permutations in computational complexity : that is , permutations that are easy to calculate , but hard to invert . We show that the existence of a small encoder for non-invertible generators would allow us to design an invertor for any Boolean permutation , thus violating the existence a one-way permutation . This is the first time such ideas have been applied to generative models . Note that a non-invertible generator results in a distribution supported on a lower dimensional manifold . Thus , our results show that learning deep generative models is harder when data lies on a sufficiently complex low-dimensional manifold , corroborating similarly flavored empirical observations ( Dai & Wipf , 2019 ; Arjovsky et al. , 2017 ) . 2 OUR RESULTS . The Variational Autoencoder ( VAE ) ( Kingma & Welling , 2013 ) is one of the most commonly used paradigms in generative models . It ’ s trained by fitting a generator which maps latent variables z to observables x , denoted by pθ ( x|z ) , as well as an encoder which maps the observables to the latent space , denoted by qφ ( z|x ) . Here φ and θ are the encoder parameters and generator parameters respectively . Given n training samples { x ( i ) } ni=1 , the VAE objective is given by max φ , θ 1 n n∑ i=1 Ez∼qφ ( .|x ( i ) ) [ log pθ ( x ( i ) |z ) ] −KL ( qφ ( z|x ( i ) ) ||p ( z ) ) where p ( z ) is typically chosen to be a standard Gaussian . This loss can be viewed as a variational relaxation of the maximum likelihood objective , where the encoder qφ , in the limit of infinite representational power , is intended to model the posterior distribution pθ ( z|x ( i ) ) . Setup : We will consider a setting in which the data distribution itself is given by some ground-truth generator G : Rdl → Rdo , and ask how complex ( in terms of number of parameters ) the encoder needs to be ( as a function of the number of parameters of G ) , s.t . it approximates the posterior distribution p ( z|x ) of the generator . We will consider two standard probabilistic models for the generator/encoder respectively . Definition 1 ( Latent Gaussian ) . A latent Gaussian is the conditional distribution given by a stochastic pushforward of a Gaussian distribution . That is , for latent variable z ∈ Rdl and observable x ∈ Rdo , for a neural network G : Rdl → Rdo and noise parameter β2 , we have p ( x|z ) = N ( G ( z ) , β2Ido ) and p ( z ) = N ( 0 , Idl ) . In other words , a sample from this distribution can be generated by sampling independently z ∼ N ( 0 , Idl ) and ξ ∼ N ( 0 , β2Ido ) and outputting x = G ( z ) + ξ . This is a standard neural parametrization of a generator with ( scaled ) identity covariance matrix , a fairly common choice in practical implementations of VAEs ( Kingma & Welling , 2013 ; Dai & Wipf , 2019 ) . We will also define a probabilistic model which is a composition of latent Gaussians ( i.e . consists of multiple stochastic layers ) , which is also common , particularly when modeling encoders in VAEs , as they can model potentially non-Gaussian posteriors ( Burda et al. , 2015 ; Rezende et al. , 2014 ) : Definition 2 ( Deep Latent Gaussian ) . A deep latent Gaussian is the conditional distribution given by a sequence of stochastic pushforwards of a Gaussian distribution . That is , for observable z0 ∈ Rd0 and latent variables { zi ∈ Rdi } Li=1 , for neural networks { Gi : Rdi−1 → Rdi } Li=1 and noise parameters { β2i } Li=1 , the conditional distribution p ( zL|z0 ) is a deep latent Gaussian when p ( zi|zi−1 ) = N ( Gi ( zi−1 ) , β2i Idi ) , ∀i ∈ [ L ] and p ( z0 ) = N ( 0 , Id0 ) . In other words , a deep latent Gaussian is a distribution , which can be sampled by ancestral sampling , one layer at a time . Note that this class of distributions is convenient as a choice for an encoder in a VAE , since compositions are amenable to the reparametrization trick of Kingma & Welling ( 2013 ) —the randomness for each of the layers can be “ presampled ” and appropriately transformed ( Burda et al. , 2015 ; Rezende et al. , 2014 ) . Then , we ask the following : Question : If a VAE generator is modeled as a latent Gaussian ( that is , p ( x|z ) ≡ N ( G ( z ) , β2I ) ) , s.t . the corresponding G has at most N parameters , and we wish to approximate the posterior p ( z|x ) by a deep latent Gaussian s.t . the total size of the networks in it have at most N ′ parameters , how large must N ′ be as function of N ? We will work in the setting dl = do = d , and prove a dichotomy based on the invertibility of G : namely , if G : Rd → Rd is bijective , and β ≤ O ( 1 d1.5 √ log d/ ) , the poste- rior p ( z|x ) can be -approximated in total variation distance by a deep latent Gaussian of size N ′ = O ( N · poly ( d , 1/β , 1/ ) ) . Thus , if the neural network G is invertible , and for a fixed and a small-enough variance term β2 , we can approximate the posterior with a deep latent Gaussian polynomially larger than G. On the other hand , if G is not bijective , if one-way-functions exist ( a widely believed computational complexity conjecture ) , we will show there exists a VAE generator G : Rd → Rd of size polynomial in d , for which the posterior p ( z|x ) can not be approximated in total variation distance for even an inverse polynomial fraction of inputs x , unless the inferential network is of size exponential in d. Remark 1 : Note that it is in fact sufficient to investigate only the case where dl = do = d. Let us elaborate why : • For Theorem 1 , a function G : Rdl → Rdo can be bijective , Lipschitz and strongly invertible ( i.e . satisfies Assumptions 1 and 2 ) only if dl = do , so Theorem 1 ( upper bound ) would not make any sense unless the dimensions are equal . ( Note : there are bijective maps between , say , R and R2 , e.g . space-filling curves , but these can not be Lipschitz , as Lipschitz maps preserve the Hausdorff dimension , and the Hausdorff dimensions of R and R2 are 1 and 2 respectively ) . • Theorem 2 is a lower bound – meaning we aim to exhibit an example of a hard instance when G is not bijective . Having exhibited a hard instance for the dl = do = d case , it is trivially easy to construct a hard instance in the theorem where the output dimension is larger—which is the more common setting in practice . To do so , consider a circuit C̃ : { ±1 } dl → { ±1 } do , which equals to C , the one-way-circuit from Conjecture 1 on the first dl coordinates , and is equal to 1 ( i.e . is constant ) on the last do − dl coordinates . Then C̃ is one-way too—since the last do − dl values are just fixed to 1 , if we can invert it , we can invert C too . The reduction in the proof of Theorem 2 then can just be performed with C̃ instead , giving a lower bound instance for the dl < do case . 2.1 UPPER BOUNDS FOR BIJECTIVE GENERATORS . We first lay out the assumptions on the mapG . The first is a quantitative characterization of bijectivity ; and the second requires upper bounds on the derivatives of G up to order 3 . We also have a centering assumption . We state these below . Assumption 1 ( Strong invertibility ) . We will assume that the latent and observable spaces have the same dimension ( denoted d ) , and G : Rd → Rd is bijective . Moreover , we will assume there exists a positive constant m > 0 such that ∀z1 , z2 ∈ Rd , ‖G ( z1 ) −G ( z2 ) ‖ ≥ m · ‖z1 − z2‖ . Remark 2 : This is a stronger quantitative version of invertibility . Furthermore , the infinitesimal version of this condition ( i.e . ‖z1−z2‖ → 0 ) implies that the smallest magnitude of the singular values of the Jacobian at any point is lower bounded by m , that is ∀z ∈ Rd , min i∈ [ d ] |σi ( JG ( z ) ) | ≥ m > 0 . Since m is strictly positive , this in particular means that the Jacobian is full rank everywhere . Remark 3 : Note , we do not require that G is layerwise invertible ( i.e . that the each map from one layer to the next is invertible ) – if that is the case , at least in the limit β → 0 , the existence of an inference decoder of comparable size toG is rather obvious : we simply invert each layer one at a time . This is important , as many architectures based on convolutions perform operations which increase the dimension ( i.e . map from a lower to a higher dimensional space ) , followed by pooling ( which decrease the dimension ) . Nevertheless , it has been observed that these architectures are invertible in practice—Lipton & Tripathi ( 2017 ) manage to get almost 100 % success at inverting an off-the-shelf trained model—thus justifying this assumption . Assumption 2 ( Smoothness ) . There exists a finite positive constant M > 0 such that : ∀z1 , z2 ∈ Rd , ‖G ( z1 ) −G ( z2 ) ‖ ≤M · ‖z1 − z2‖ Moreover , we will assume that G has continuous partial derivatives up to order 3 at every z ∈ Rd and the derivatives are bounded by finite positive constants M2 and M3 as ∀z ∈ Rd , ∥∥∇2G ( z ) ∥∥ op ≤M2 < ∞ , ∥∥∇3G ( z ) ∥∥ op ≤M3 < ∞ Remark 4 : This is a mild assumption , stating that the map G is smooth to third order . The infinitesimal version of this means that the largest magnitude of the singular values of the Jacobian at any point is upper bounded by M , that is ∀z ∈ Rd , max i∈ [ d ] |σi ( JG ( z ) ) | = ‖JG ( z ) ‖op ≤M < ∞ . Remark 5 : A neural network with activation function σ will satisfy this assumption when σ : R→ R is Lipschitz , and maxa |σ′ ( a ) | & maxa |σ′′ ( a ) | are finite . Assumption 3 ( Centering ) . The map G : Rd → Rd satisfies G ( 0 ) = 0 . Remark 6 : This assumption is for convenience of stating the bounds—we effectively need the “ range ” of majority of the samples x under the distribution of the generator . All the results can be easily restated by including a dependence on ‖G ( 0 ) ‖ . Our main result is then stated below . Throughout , the O ( . ) notation hides dependence on the map constants , namely m , M , M2 , M3 . We will denote by dTV ( p , q ) the total variation distance between the distributions p , q. Theorem 1 ( Main , invertible generator ) . Consider a VAE generator given by a latent Gaussian with µ = 0 , Σ = I , noise parameter β2 and generator G : Rd → Rd satisfying Assumptions 1 and 2 , which has N parameters and a differentiable activation function σ . Then , for β ≤ O 1 d1.5 √ log d ( 1 ) there exists a deep latent Gaussian withN ′ = O ( N · poly ( d , 1β , 1 ) ) parameters and activation functions { σ , σ′ , ρ } and a neural network φ with O ( N ′ ) parameters and activation functions { σ , σ′ , ρ } , where ρ ( x ) = x2 , such that with probability 1−exp ( −O ( d ) ) over a sample x from the VAE generator , φ ( x ) produces values for the parameters of the deep latent Gaussian such that the distribution q ( z|x ) it specifies satisfies dTV ( q ( z|x ) , p ( z|x ) ) ≤ . Remark 7 : Having an encoder q ( z|x ) with parameters produced by a neural network taking x as input is fairly standard in practice—it is known as amortized variational inference . ( Mnih & Gregor , 2014 ; Kingma & Welling , 2013 ; Rezende et al. , 2014 ) Remark 8 : The addition of ρ in the activation functions is for convenience of stating the bound . Using usual techniques in universal approximation it can be simulated using any other smooth activation , see Appendix G .

Summary: This purely mathematical paper investigates the important question of how does the necessary level of complexity of an inference subnetwork depend on the the complexity of the corresponding generative subnetwork, in a VAE model. The paper introduces a specific measure of invertibility, and uses it to show that a generative subnetwork that is sufficiently strongly invertible only requires an inference model at similar complexity, while some "non-invertible" generative subnetworks will require an exponentially more complex inference subnetwork. The paper provides a lengthy and comprehensive proof of both.