Latent Variables on Spheres for Sampling and Inference

1 INTRODUCTION . Deep generative models , such as Variational Auto-Encoder ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) and Generative Adversarial Network ( GAN ) ( Goodfellow et al. , 2014 ) , play more and more important role in machine learning and computer vision . However , the problem of variational inference in VAE is still challenging , especially for high-dimensional data like images . To be formal , let X = { x1 , . . . , xn } denote the set of observable data points and Z = { z1 , . . . , zn } the set of desired latent vectors , where xi ∈ Rdx and zi ∈ Rdz . Let pg ( x|z ) denote the likelihood of generated sample conditioned on latent variable z and p ( z ) the prior , where g denotes the decoder . The encoder f in VAE parameterizes the variational posterior qf ( z|x ) in light of the lower bound of the marginal log-likelihood log pg ( x ) = log ∫ pg ( x|z ) p ( z ) dz = log ∫ qf ( z|x ) qf ( z|x ) pg ( x|z ) p ( z ) dz ( 1 ) ≥ −DKL [ qf ( z|x ) ||p ( z ) ] + Eq [ log pg ( x|z ) ] . ( 2 ) The first term DKL [ qf ( z|x ) ||p ( z ) ] constrains the encoded latent codes to the prior via the KLdivergence , and the second term Eq [ log pg ( x|z ) ] serves to guarantee the reconstruction accuracy of inputs . For a Gaussian pg ( x|z ) of diagonal covariance matrix , log pg ( x|z ) reduces to the varianceweighted squared error ( Doersch , 2016 ) . The lower-bound approximation of the log-likelihood provides a feasible solution for VAE . But it also causes new problems . For example , the generated sample g ( z ) deviates from the real distribution of X when sampling from the given prior due to that the learnt qf ( z|x ) is incapable of matching the prior distribution well . Besides , the reconstruction g ( f ( x ) ) is not satisfactory either . For imagery data , bluriness usually occurs . In order to manipulate real images for GAN models , we usually need to formulate an encoder via the framework of VAE . The variational inference also applies in this scenario . The problems of VAE are the obstacles of putting the GAN encoder in the right way either . There are other methods of learning an encoder for GAN in the adversarial way such as ( Dumoulin et al. , 2017 ; Li et al. , 2017 ; Ulyanov et al. , 2017 ; Heljakka et al. , 2018 ) . However , the structural precision of reconstruction is generally inferior to the VAE framework , because the high-level semantics of objects outweigh low-level structures for such methods ( Donahue & Simonyan , 2019 ) . Besides , the concise architecture of VAE is more preferred in this scenario . Therefore , learning precise latent variables z = f ( x ) is critical to applications of VAE and GAN . Using a different theory in this paper , we propose a simple method to circumvent the problem . Our contributions are summarized as follows . 1 ) We introduce the volume concentration of highdimensional spheres . Based on the concentration property , we point out that projecting on a sphere for data that are distributed according to the spherical mass produces little difference from the viewpoint of the volume in high-dimensional spaces . Thus , it is plausible to perform inference pertaining to VAE on the sphere . 2 ) We further analyze the probability distribution of distances between two arbitrary sets of random points on the sphere in high dimensions and illustrate the phenomenon of distance convergence . Furthermore , we prove that the Wasserstein distance between two arbitrary datasets randomly drawn from a high-dimensional sphere are nearly identical , meaning that the data on the sphere are distribution-robust for generative models with respect to Wasserstein distance . 3 ) Based on our theoretical analysis , we propose a very simple algorithm for sampling generative models . The same principle is also harnessed to reformulate VAE . The spherical normalization is simply put on latent variables instead of variational inference while preserving randomness of latent variables by centerization . In contrast to VAE and variational inference , we name such an autoencoder as Spherical Auto-Encoder ( SAE ) and the associated inference as spherical inference . 4 ) We perform extensive experiments to validate our theoretical analysis and claims with sampling and inference . 2 LATENT VARIABLES ON SPHERE . For latent variables or data points sampled from some priors , the projection on the unit sphere can can be easily performed by z ← z/‖z‖ . ( 3 ) This spherical normalization for priors fed into the generator is employed in StyleGAN that is the phenomenal algorithm in GANs ( Karras et al. , 2018b ) . To test the robustness of StyleGAN against the diverse distributions , we conduct two groups of experiments with the input z sphere-normalized and not sphere-normalized when training StyleGAN . As shown in Figure 1 , the diversity of generated faces is good for two different distributions with normalized z , whereas the face modes become similar for the case of the uniform distribution that z is not normalized . This experiment indicates that StyleGAN with sphere-normalized z is much more robust to the variation of variable modes from different distributions . Inspired by this comparison , we interpret the benefit of using random variables on spheres by virtue of high-dimensional geometry in this section . Based on these theories , a novel algorithm is proposed for random sampling and spherical inference for GAN and VAE . 2.1 VOLUME CONCENTRATION . For high-dimensional spaces , there are many counter-intuitive phenomena that will not happen in low-dimensional spaces . For a convenient analysis , we assume that the center of the sphere Sd embedded in Rd+1 is at the origin . We first present the concentration property of sphere volume in Rd+1 . One can find the proof in ( Blum et al. , 2020 ) . Theorem 1 . Let V ( r ) and V ( ( 1− ) r ) denote the volumes of the two concentric spheres of radius r and ( 1− ) r , respectively , where 0 < < 1 . Then V ( ( 1− ) r ) /V ( r ) = ( 1− ) d. ( 4 ) And if = t/d , V ( ( 1− ) r ) /V ( r ) → e−t when d→∞ , where t is a constant . Theorem 1 says that the volume of the d-dimensional sphere of radius ( 1− ) r rapidly goes to zero when d goes large , meaning that the interior of the high-dimensional sphere is empty . In other words , nearly all the volume of the sphere is contained in the thin annulus of width r = rt/d . The width becomes very thin when d grows . For example , the annulus of width that is 0.9 % of the radius contains 99 % of all the volume for the sphere in R512 . To help understand this counter-intuitive geometric property , we make a schematic illustration in Figure 2 . The probabilistic operations can be very beneficial from the volume concentration of spheres . Suppose that we perform the probabilistic optimization pertaining to latent variables sampled according to the distribution of the sphere volume . The probability mass is also empty due to the volume concentration . Therefore , the error is controllable if we perform it on the sphere , provided that these latent variables lie in high-dimensional spaces . For VAE , therefore , we can write a feasible approximation log pg ( x ) = log ∫ Int ( Sd ) pg ( x|z ) p ( z ) dz ≈ log ∫ Sd pg ( x|z ) p ( z ) dz , ( 5 ) where Int ( Sd ) denotes the interior of Sd , ‖z‖ ≤ r , and d is sufficiently large . The spherical approximation for log pg ( x ) is an alternative scheme except the lower bound approximation presented in equation ( 1 ) . In fact , the distributions defined on the sphere have been already exploited to reformulate VAE , such as the von Mises-Fisher distribution ( Davidson et al. , 2018 ; Xu & Durrett , 2018 ) . But the algorithms proposed in ( Davidson et al. , 2018 ; Xu & Durrett , 2018 ) still fall into the category using the variational inference like the vanilla VAE . To eliminate this constraint , we need more geometric analysis presented in the following section . 2.2 DISTANCE CONVERGENCE . To dig deeper , we examine the pairwise distance between two arbitrary points randomly sampled on Sd . The following important lemma was proved by ( Lord , 1954 ; Lehnen & Wesenberg , 2002 ) . Lemma 1 . Let ξ denote the Euclidean distance between two points randomly sampled on the sphere Sd of radius r. Then the probability distribution of ξ is ρ ( ξ ) = ξd−2 c ( d ) rd−1 [ 1− ( ξ 2r ) 2 ] d−32 , ( 6 ) where the coefficient c ( d ) is given by c ( d ) = √ πΓ ( d−1 2 ) /Γ ( d 2 ) . And the mean distance ξµ and the standard deviation ξσ are ξµ = 2d−1r [ Γ ( d 2 ) ] 2 √ πΓ ( d− 12 ) and ξσ = √2r√1− ξ2µ 2r2 , ( 7 ) respectively , where Γ is the Gamma function . Furthermore , ξµ → √ 2r ( 1− 18d ) and ξσ → r√2d when d goes large . Lemma 1 tells that the pairwise distances between two arbitrary points randomly sampled on Sd approach to be mutually identical and converge to the mean ξµ = √ 2r when d grows . The associated standard deviation ξσ → 0 . This result is in some extent surprising compared to the intuition in low-dimensional spaces . We display the average distance and its standard deviation in Figure 3 , showing that the convergence process is fast . Taking R512 for example , we calculate that ξµ = 1.4139 and ξσ = 0.0313 . The standard deviation is only 2.21 % of the average distance , meaning that the distance discrepancy between arbitrary zi and zj is rather small . This surprising phenomenon is also observed for neighborly polytopes when solving the sparse solution of underdetermined linear equation ( Donoho , 2005 ) and for nearest neighbor search in high dimensions ( Beyer et al. , 1999 ) . With Lemma 1 , we can study the property of two different random datasets on Sd , which serves to distribution-free sampling and spherical inference in generative models . To this end , we first introduce the computational definition of Wasserstein distance . Let Z = { z1 , . . . , zn } and Z′ = { z′1 , . . . , z′n } be the datasets of random variables drawn from Sd at random , respectively . Then the 2-Wasserstein distance is defined as W 22 ( Z , Z′ ) = min ω ∑n i=1 ∑n j=1 ωij‖zi − z′j‖2 ( 8 ) s.t . ∑n i=1ωij = ∑n j=1ωij = 1 , ( 9 ) where ω is the doubly stochastic matrix . By Lemma 1 , it is not hard to derive the following Theorem . Theorem 2 . 1 W2 ( Z , Z′ ) → √ 2nr with zero standard deviation when d→∞ . Theorem 2 says that despite the diverse distributions , the 2-Wasserstein distance between two arbitrary sets of random variables on the sphere converges to a constant when the dimension is sufficiently large . For generative models , this unique characteristic for datasets randomly sampled from highdimensional spheres bring great convenience for distribution-robust sampling and spherical inference . For example , if Z and Z′ obeys the different distributions , the functional role of Z′ nearly coincides 1It suffices to note that the case described in Theorem 2 is essentially different from the approximate solution of Wasserstein distance via Monte Carlo sampling , where the points sampled from the unit sphere are applied as projection subspaces . with that of Z with respect to Wasserstein distance , provided that both Z and Z′ are randomly drawn from the high-dimensional sphere . The specific distributions of Z and Z′ affect the result negligibly under such a condition . We will present the specific application of Theorem 2 in the following section . In fact , we can obtain the bounds of W2 ( Z , Z′ ) using the proven proposition about the nearlyorthogonal property of two random points on high-dimensional spheres ( Cai et al. , 2013 ) . However , Theorem 2 is sufficient to solve the problem raised in this paper . So , we bypass this analysis to simplify the theory for easy readability .

Summary: This paper considers the L2 normalization of samples “z” from a given prior p(z) in Generative Adversarial Netowks (GAN) and autoencoders. The L2 normalization corresponds to projecting samples onto the surface of a unit-hypersphere. Hence, to attempt to justify this normalization, the authors rely on some already established results regarding high dimensional hyperspheres. In particular, the focus is on the fact that, the Euclidean distance between any given point on a hypersphere and another randomly sampled point on the hypersphere tends to a constant, when the number of dimensions goes to infinity. This result is then used to show that the Wasserstein distance between two arbitrary distributions on a hypersphere converges to a constant when the number of dimensions grows. Based on this result, the authors claim that projecting the latent samples onto the surface of a hypersphere would make GAN less sensitive to the choice of the prior distribution. Moreover, they claim that such normalization would also benefits inference, and that it addresses the issue of variational inference in VAE.