Latent Variables on Spheres for Sampling and Inference
Authors: Deli Zhao
Jiapeng Zhu
Bo Zhang
Venue:  (2020)
Keywords: variational autoencoder
generative adversarial network

Abstract: Variational inference is a fundamental problem in Variational AutoEncoder (VAE). The optimization with lower bound of marginal log-likelihood results in the distribution of latent variables approximate to a given prior probability, which is the dilemma of employing VAE to solve real-world problems. By virtue of high-dimensional geometry, we propose a very simple algorithm completely different from existing ones to alleviate the variational inference in VAE. We analyze the unique characteristics of random variables on spheres in high dimensions and prove that Wasserstein distance between two arbitrary data sets randomly drawn from a sphere are nearly identical when the dimension is sufficiently large. Based on our theory, a novel algorithm for distribution-robust sampling is devised. Moreover, we reform the latent space of VAE by constraining latent variables on the sphere, thus freeing VAE from the approximate optimization of posterior probability via variational inference. The new algorithm is named Spherical AutoEncoder (SAE). Extensive experiments by sampling and inference tasks validate our theoretical analysis and the superiority of SAE.

Under review as a conference paper at ICLR 2020
LATENT VARIABLES ON SPHERES FOR SAMPLING AND SPHERICAL INFERENCE
Anonymous authors Paper under double-blind review
ABSTRACT
Variational inference is a fundamental problem in Variational Auto-Encoder (VAE). By virtue of high-dimensional geometry, we propose very simple algorithm completely different from existing ones to solve the inference problem in VAE. We analyze the unique characteristics of random variables on spheres in high di- mensions and prove that the Wasserstein distance between two arbitrary datasets randomly drawn from a sphere are nearly identical when the dimension is suf- ficiently large. Based on our theory, novel algorithm for distribution-robust sampling devised. Moreover, we reform the latent space of VAE by constraining latent random variables on the sphere, thus freeing VAE from the approximate optimization pertaining to the variational posterior probability. The new algorithm is named Spherical uuto-Encoder (SAE), which is in essence the vanilla autoen- coder with the spherical constraint on the latent space. The associated inference is called the spherical inference, which is geometrically deterministic but is much more robust to various probabilistic priors than the variational inference in VAE for sampling. The experiments on sampling and inference validate our theoretical analysis and the superiority of SAE.
INTRODUCTION
Deep generative models, such as Variational Auto-Encoder (VAE) (Kingma Welling 2013 Rezende et al. 2014) and Generative Adversarial Network (GAN) (Goodfellow et al. 20144), play more and more important role in machine learning and computer vision. However, the problem variational inference in VAE is still challenging, especially for dinensiinnii data like images.
To be formal, let X = ( .x denote the set of observable data points and L = (z1...., zn} the set of desired latent vectors, where æ; E Idd and 2; E IRd, Let py (x[2) denote the likelihood of generated sample conditioned on latent variable z and p(z) the prior, where g denotes the decoder. The encoder f in VAE parameterizes the variational posterior ę, (2 (x) in light of the lower bound of the marginal log-likelihood
gf(2| (x)
log Ps (xx) = log Pa (p(a))))(())))) = log
(=| Py (o)2)))
(1)
2 - (x)1)pl) [))(((>]] + E. (x
(2)
The first term DKL [gr (z|xx)|p((2) constrains the encoded latent codes to the prior via the KL- divergence, and the second term loo (2 x)] serves to guarantee the reconstruction accuracy of inputs. For a Gaussian (=]=)) of diagonal covariance matrix, log log,, (xl2z) reduces to the variance- weighted squared error (Doersch] 2016).
The lower-bound approximation of the log-likelihond provides feasible solution for VAE. But also causes new problems. For example, the generated sample g(z) deviates from the real distribution of when sampling from the given prior due to that the learnt gr (z (x) is incapable of matching the prior distribution well. Besides, the reconstruction (x(()) is not satisfactory either. For imagery data, bluriness usually occurs.
In order to manipulate real images for GAN models, usually need to formulate encoder via the framework of VAE. The variational inference also applies in this scenario. The problems VAE are the obstacles of putting the GAN encoder in the right way either. There are other methods of learning

---

Under review as a conference paper at ICLR 2020
(a) The prior is projected on the sphere.
(b) The prior z is not projected on the sphere.
Figure 1: Generated faces by StyleGAN with different priors. For (a) and (b), the first shows the generated faces with the normal distribution and the second row displays the faces with the uniform distribution. The priors are both normal when training.
an encoder for GAN in the adversarial way such as (Dumoulin et al. 2017; Li et al. 2017 Ulyanov et al. 2017: Heljakka et al. 2018). However, the structural precision of reconstruction generally inferior to the VAE framework, because the high-level semantics of objects outweigh low-level structures for such methods (Donahue & Simonyan 2019). Besides, the concise architecture of VAE is more preferred in this scenario. Therefore, learning precise latent variables z = f(x) is critical to applications of VAE and GAN.
Using a different theory in this paper, we propose a simple method to circumvent the problem. Our contributions are summarized as follows. 1) We introduce the volume concentration of high- dimensional spheres. Based on the concentration property, we point out that projecting data that are distributed according to the spherical mass produces little difference from the viewpoint of the volume in high-dimensional spaces. Thus, it is plausible to perform inference pertaining to VAE on the sphere. 2) We further analyze the probability distribution of distances between two arbitrary sets of random points on the sphere in high dimensions and illustrate the phenomenon of distance convergence. Furthermore, we prove that the Wasserstein distance between two arbitrary datasets randomly drawn from high-dimensional sphere nearly identical, meaning that the data on the sphere are distribution-role 3) Based
sphere for
for generative models with respect to Wasserstein distance. theoretical analysis, we propose very simple algorithm for sampling generative models. The same principle is also harnessed to reformulate VAE. The spherical normalization is simply put on latent variables instead of variational inference while preserving randomness of latent variables by centerization. In contrast to VAE and variational inference, we name such an autoencoder as Spherical Auto-Encodee (SAE) and the associated inference as spherical inference. 4) We perform extensive experiments to validate our theoretical analysis and claims with sampling and inference.
LATENT VARIABLES ON SPHERE
For latent variables or data points sampled from some priors, the projection on the unit sphere can can be casily performed by
サー 2/|
(3)
This spherical normalization for priors fed into the generator is employed in StyleGAN that the phenomenal algorithm in GANs (Karras et al.) (0018)). To test the robustness of StyleGAN against the diverse distributions, we conduct two groups of experiments with the input z sphere- nabber-nomm and not opphere-nomallize when training StyleGAN. As shown in Figure] l the diversity of generated
2

---

Under review as a conference paper at ICLR 2020
(a) Now-dimensions space. (b) High- feth-ddmmesionall space.
Figure Volume of spheres in different dimensional spaces. The volume of the sphere in the high-dimensional space is highly concentrated near the surface. The interior is nearly empty.
faces is good for two different distributions with normalized 2, whereas the face modes become similar for the case of the uniform distribution that is not normalized. This experiment indicates that StyleGAN with sphere-norme normalzee z is much more robust to the variation of variable modes from different distributions.
Inspired by this comparison, we interpret the benefit of using random variables on spheres by virtue of high-ding dibeeeimniionll geometry in this section. Based on these theories, novel algorithm is proposed for random sampling and spherical inference for GAN and VAE.
2.1 VOLUME CONCENTRATION
For high-dimensional spaces, there are many counter-intuitive phenomena that will not happen in low-dimensional spaces. For a convenient analysis, we assume that the center of sphere gd embedded in R++1 is at the origin. We first present the concentration property of sphere volume in Rd++1 One can find the proof in (Blum et al. 2020).
Theorem 1. Let (r) and V ((1- - c)r) denote the volumes of the two concentric spheres of radius r and (1 - e)r, respectively, where 0 < e 1. Then
V ( - c)r) /V (1-
And i t/d, V ((1 - (V) et when d DG, where is constant.
Theorem says that the volume of the d-dimensional sphere radius (1 (1-e)r e)r rapidly goes to zero when d goes large, meaning that the interior of the high-dimensional sphere is empty. In other words, nearly all the volume of the sphere is contained the thin annulus of width er = rt/d. The width becomes very thin when d grows. For example, the annulus of width that 0.9% of the radius contains 99% of all the volume for the sphere in 5112 To help understand this counter-intuitive geometric property, we make schematic illustration in Figure
The probabilistic operations can be very beneficial from the volume concentration of spheres. Suppose that we perform the probabilistic optimization pertaining latent variables sampled according the distribution of the sphere volume. The probability mass is also empty due to the volume concentration. Therefore, the error is controllable if we perform it on the sphere, provided that these latent variables lie in nigh-dimensional spaces. For VAE, therefore, we can write feasible approximation
Pg pp(x)=) = log / P9 (g(2)2)p(zddz
log Py = log
(5)
(mn(Bd))
where Int(Sa) denotes the interior of Sd. ||| S and d is sufficiently large. The spherical approximation for log (z) is an alternative scheme except the lower bound approximation presented in equation (I). In fact, the distributions defined on the sphere have been already exploited to re- formulate VAE, such as the von Mises-Fisher distribution (Davidson et al.) 2018; Xu & Durrett 2018). But the algorithms proposed in (Davidson et al. [018 20 Xu & Durrett 2018) 1) still fall into the category using the variational inference like the vanilla VAE. To climinate this constraint, we need more geometric analysis presented in the following section.
2.2 DISTANCE CONVERGENCE
To dig deeper, we examine the pairwise distance between two arbitrary points randomly sampled on gd. The following important lemma was proved by (Lord] 1954 Lehnen & Wesenberg 2002).

---

Under review as a conference paper at ICLR 2020
2
Average distance std
1.414
0.5
2 32
128
256
512
Dimension (a)
(b)
Figure 3: The illustration of average distance between two points randomly sampled on unit spheres of various dimensions. (a) The average distance (red curve) and standard deviation (gray background). (b) Schematic illustration of pairwise distances between two arbitrary points in high dimension. They nearly identical.
Lemma 1. Let denote the Euclidean distance between points randomly sampled on the sphere gd of radius r. Then the probability distribution of & is
4-3
5d-2
5
P(E)
1
(6)
c(d)rd-I
2r
where the coefficient c(d) is given by c(d) VTI (d-1) /T (g)
And the mean distance Eμ and the
standard deviation Eo are
2d-1, T (5))
F2 and c. = 1-
E VTI
2r
respectively, where I' is the Gamma function. Furthermore, Ep /2r (1 (1-2 8a) and &o - V2d when d goes large.
Lemma 1 tells that the pairwise distances between arbitrary points randomly sampled $ approach to be mutually identical and converge to the mean Eμ- /2r when d grows. The associated standard deviation &o + O. This result is in some extent surprising compared to the intuition in low-dimensional spaces. We display the average distance and its standard deviation in Figure 3 showing that the convergence process is fast. Taking 52 for example, we calculate that Eμ. 1.4139 and & 0.0313. The standard deviation only 2.21% of the average distance, meaning that the distance discrepancy between arbitrary z; and zj is rather small. This surprising phenomenon is also observed for neighborly polytopes when solving the sparse solution of underdetermined linear equation (Donoho 2005) and for nearest neighbor search in high dimensions (Beyer et al. 1999).
With Lemma 1 we can study the property two different random datasets on Sd, which serves to distribution free sampling and spherical inference in generative models. To this end, we first introduce the computational definition of Wasserstein distance, Let Z {ZI...., zn) and Z' = (z'...., 2 be the datasets of random variables drawn from gd at random, respectively. Then the 2- Wassersteii distance is defined as
W 2 Z') min Σ 1= "" j= Wig || 2/112
s.t. E Wky D = 1,
where w is the doubly stochastic matrix. By Lemma I is not hard derive the following Theorem. Theorem 2. W2,Z, Z') - 22n with zero standard deviation when d + co.
Theorem 2 says that despite the diverse distributions, the 2- Wasserstein distance between two arbitrary sets of random variables on the sphere converges to a constant when the dimension is sufficiently large. For generative models, this unique characteristic for datasets randomly sampled from high- dimensional spheres bring great convenience for distribution-robust sampling and spherical inference. For example, if Z and z' obeys the different distributions, the functional role Z' nearly coincides
It suffices to note that the case described in Theorem 2 is essentially different from the approximate solution of Wasserstein distance via Monte Carlo sampling, where the points sampled from the unit sphere are applicd as projection subspaces.
4

---

Under review as a conference paper at ICLR 2020
with that of Z with respect Wasserstein distance, provided that both Z and Z' randomly drawn from the high-dingg nigh-dimensional sphere. The specific distributions and z' affect the result negligibly under such condition. We will present the specific application of Theorem in the following section.
In fact, we can obtain the bounds of (2 2') using the proven proposition about the nearly- orthogonal property of two random points on high-dimensional spheres (Cai et al., 2013). However, Theorem is is sufficient to solve the problem raised in this paper. So, we bypass this analysis to simplify the theory for easy readability.
3 ALGORITHM FOR SAMPLING AND INFERENCE
We will detail the distribution- algorithms for sampling and spherical inference. A new simplified form of VAE will be presented in this section well.
3.1 SAMPLING
To acquire generative results from VAEs and GANs, we need to sample random vectors from a pre-defined prior and then feed them into the decoder or the generator. According to Theorem T and Theorem 2 however, this prior limitation can be eliminated if we project these random vectors on the corresponding sphere. To achieve this, we perform two-step manipulations on the sampled The centerization
dataset from arbitrary prior distributions. The procedure is detailed in Algorithm [ operation is motivated from the central limit theorem in probability, which transforms the distribution- specific E to be nearly distribution cirrrbution-anosticc agnostic (on the sphere). The spherization is to project these centerized vectors on the unit sphere. We find that in practice, this simple algorithm works well to reduce the bias caused by various distributions or data modes for VAEs and GANs.
Algorithm 1 Distribution-robuss sampling for generative models
Sample {=1 ................................ 2n P(z)
P(z) is an arbitrary distribution x is the j-th entry z;
2: Centerization by 2 + z
ΣS for each z;
3: Spherization by ž; + zi/ ||> for each z;
4: Return = (ž1,.... )n}
3.2 SPHERICALL INFERENCE
According Theorem randomly sampled from the high-dimensional sphere. Theorem guarantees that the error can be negligible even if they deviate from the sphere, long as they are distributed near the spherical surface. This tolerance to various modes of random variables allow us to devise a simple solution to replace the variational inference for VAE, i.c. the spherical inference we call. To be specific, we only need to constrain the centerized latent variables on the sphere, as opposed to the conventional way of employing the Kl.-divergence DKL, (g) [p=)]]]]]]]]])]))]]))]]])) and its variants with diverse priors. The sequential mappings of the autoencoder under our framework can be shown by
may know that sampling is robust to random variables if they
11 z, /
(10)
encoder
spherical constraint on the latent space
decoder
where = 1, of and 1 is the all-one vector. We can write the objective function for this type of autoencoder as
min || a11 s.t. spherical constraint on 2
(11)
f.g
where lp denotes the p-norm. The objective function of our algorithm is much simpler than that of VAE and variants based on the variational inference or various sophisticated regularizers on the latent space.
clear that we utilize the geometric characteristics of latent variables on the sphere rather than some additional losses to optimize the latent space. Our algorithm is geometric and free from the probability optimization whose performance is usually limited with the approximation dilemma. In fact, the framework of VAE in (10) reduces to standard autoencoder with the spherical constraint. There is no variational inference needed here. To highlight this critical difference, we call our algorithm Spherical Auto- Anto-Fncoder (SAE).
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Little attention has been paid on examining geometry of latent spaces in the field of generative models. So find few works directly related ours. Most relevant ones the application of von Mises-Fisher (vMF) distribution as the probability prior (Davidson et al., 2018 Xu & Durrett (0118). The vMF distribution is defined on the sphere. The sampling and variational inference
both performed with latent variables drawn on the sphere. However, the algorithms proposed in (Davidson al. 2018 Xu Durrett 2018) both rely on the variational inference as VAE does with inequality (1) For our algorithm, the whole framework is deterministic and there is no approximation involved for inferring latent codes.
For sampling, our geometric analysis is directly inspired by ProGAN Karras et al. (00 8a) and
StyleGAN (Karras et al. (2018b) that have already applied the spherical normalization for sampled inputs. We study the related theory and extend the case arbitrary distributions for both GANs and VAEs. Another related method to sample priors along the great circle when performing the interpolation in the latent space for GANs (White, 2016). The empirical results show that such sampling yields more smooth interpolated generation. This algorithm is perfectly compatible with our theory and algorithm. Therefore, can also be harnessed in our algorithm when performing interpolation as well.
Wasserstein Auto-Encoder (WAE) (Tolstikhin al. 2018) is alternative way of optimizing the model distribution and the prior distribution using Wasserstein distance. SAE is different from WAE because we do not really use Wasserstein distance for computation in the latent space. We just leverage Wasserstein distance to establish Theorem for the theoretical analysis. Adversarial Auto- Encoder (AAE) (Makhzani al., 2015) is another interesting method of replacing the variational inference with ad veresaaia learning in the latent space. But both WAE and AAE need some priors to match, which are essentially different from SAE. VAE (Higgins et al., (017) improves the flexibility of VAE by using regularization coefficient to modulate the capacity of latent information. However, 3- VAE restricted by priors like VAE.
EXPERIMENT
We conduct the experiments to test our theory and algorithms in this section. Three aspects pertaining to generative algorithms are taken into account, including sampling GANs, learning the variants of autoencoder, and sampling the decoders.
The FFHQ dataset (Karras et al. (018b) is a more complex face dataset with large variations of faces captured in the wild. We test VAH and our SAE algorithm with this benchmark dataset. We use the image size of 1 128 x 128, which is larger than the commonly chosen size in the related work and also more challenging than 64 x 64 or 32 x 32 for (variational) autoencoders to reconstruct.
5.1 SAMPLING GAN
Our first experiment is test the sampling effect using four distributions. We employ StyleGAN trained with random variables sampled from the normal distribution. The other three distributions are opted to test the generation with different priors for training, i.c. the uniform, Poisson, and Chi-squared distributions. The shapes of these three distributions are significantly distinctive from that of the normal distribution. Thus, the generalization capability of the generative model can be effectively unveiled when fed with priors that are not involved during training. We follow the experimental protocol in (Karras et al.) 2018a b) that StyleGAN is trained on the FFHQ face dataset and Fréchet inception distance (FID) (Borji (018) is used as the quality metrics of generative results. We take d_ = 512, which is set in StyleGAN. This dimension also used for both VAE and SAE for face data.
From Table we can see that the generative results by the normal distribution significantly better than the others when tested with the original samples. The uniform distribution is as good as the normal distribution when projected on the sphere. This is because the values for each random vector overall symmetrically distributed according to the origin. They satisfy the condition in Theorem 2 after the spherical projection. The accuracy of Poisson and Chi-squared distributions is considerably improved after centerization, even better than the vanilla uniform distribution. But the

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of sampling. The quantitative results are FIDs.
vanilla sampling
centerization after sampling no spherization
Distribution
spherization
spherization
spherization
Normal
6.20
6.16
6.27
6.12
Uniform 33.93
6.16
33.86
6.16
Poisson 23.70
26.85
18.15
6.19
Chi-squared
25.26
27.07
11.34
6.16
FFHQ faces
VAE
SAE (ours)
Figure 4: Reconstructed faces by VAE and SAE. SAE only uses the spherical constraint in equation (T0 instead of the variational inference in VAE.
accuracy difference between all the compared distributions is rather negligible after centerization and spherization, empirically verifying the theory presented in Theorem 2
5.2
AUTOENCODERS
We compare the vanilla VAE with the normal distribution (Kingma & Welling 2013) with our SAE algorithm for reconstruction and sampling task
From Figure 4 we can see that the face quality of SAE outperforms that of VAE. The imagery details like semantic structures are preserved much better for SAE. For example, the sunglasses the sixth image is successfully recovered by SAE, whereas VAE distorts the face due to this occlusion. It worth emphasizing that the blurriness for images reconstructed by SAE is much less than that by VAE, implying that the spherical inference is superior to the variational inference in VAE. The different accuracy measurements in Table 2 also indicate the consistently better performance of SAE.
To test the generative capability of the models, we also perform the experiment of sampling the decoders as done in section 5.1 Prior samples are drawn from the normal, uniform, Poisson, and Chi- squared distributions, respectively, and then fed into the decoders generate faces. Figure(Jillustratt the generated faces of significantly different quality with respect to four types of samplings. The style of the generated faces by SAE keeps consistent, meaning that SAE is rather robust different probability priors. This also empirically verifies the correctness of Theorem by solving the real problem. As a comparison, the quality of the generated faces by VAE varies with probability priors. In other words, VAE is sensitive to the outputs of the encoder with the variational inference, which is probably the underlying reason of the difficulty of training VAE with sophisticated architectures. We also present the experimental results on MNIST and CelebA in Appendix.
We fail to train convergent model for the spherical VAE (S-VAE) with von Mises-Fisher distribution on the FFHQ dataset. So this algorithm not compared here. The experiment MNIST is provided Appendix.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Quantitative comparison of face reconstruction.
Metric
FID
SWD
MSE
VAH
134.22
77.68
0.091
SAE (ours)
91.02
56.58
0.063
VAE
SAE (ours)
Normal distribution
Uniform distribution
Poisson distribution
Chi-squared distribution
Figure 5: Generated faces with inputs of different priors. With the pre-trained decoders of VAE and SAE, the faces generated with the random vectors sampled from the four probability priors.
CONCLUSION
In this paper, attempt address the issue of the variational inference in VAE and the limitation of prior-sensitive sampling in GAN. By analyzing the geometry of volume concentration and distance convergence on the high-dimensional sphere, prove that the Wasserstein distance converges to be a constant for two datasets randomly sampled from the sphere when the dimension goes large. Based on this unique characteristie, we propose very simple algorithm for sampling and spherical inference. The sampled data from priors are first centerized and then projected onto the unit sphere before being fed into decoders (or generators). Such random variables on the sphere are robust the diverse prior distributions. With our theory, the vanilla VAE can be reduce to standard autoencoder with the spherical constraint on the latent space. other words, the conventional variational inference in VAE is replaced by the simple operations of centerization and spherization. The new autoencoder unto-EEcoder (SAE). The experiments on the FFHQ face data validate the
named as Spherical effectiveness of our algorithm for sampling and spherical inference. It is worth noting that the applications our theory and the novel algorithm are not limited for VAEs and GANs. Interested readers may explore the possibility in their scenarios.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? International Conference on Database Theory, pp. 217-235, 1999.
Avrim Blum, John Hopcroft, and Ravi Kannan. Foundations of Data Science. Cambridge University Press, 2020.
Ali Borji. Pros and cons of GAN evaluation measures. arXiv:1802.034448 2018.
Tony Cai, Jianqing Fan, and Tiefeng Jiangd. Distributions of angles in random packing on spheres. Journal of Machine Learning Research, 4: 8377-188664 2013.
Tim R. Davidson, Luca Falorsi, Nicola De Cao. Thomas Kipf, and Jakub M. Tomezak. Hyperspherical variational auto-encoders. In Proceedings of the Thirly-Fourth Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
Carl Doersch. Tutorial on variational autoencoders. arXiv:1606.05903 2016.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv:190700244.. 2019.
David L. Donoho. Neighborly polytopes and sparse solutions of underdetermined linear equations. Technical report, Stanford University, 2005.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky. and Aaron Courville. Adversarially learned inference. International Conference on Learning Representations 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Ari Heljakka, Amo Solin, and Juho Kannala. Pioneer networks: Progressively growing generative autoencoder. | 0000 2018
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. DAta:VAE: Learning basic visual concepts with constrained variational framework. In International Conference Learning Representations (ICLR), 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018a.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv: 1 0204444, 2018b.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2th International Conference on Learning Representations (ICLR), 2013.
A1 Lehnen and Gary Wesenberg. The sphere game in dimensions. 2002. htttp://faculyy: madisonco. sonco Legee wwwpwwwwww.....eeeeee.p.e wwwpaannnn/pppeee/yyeeeeee .hmm
Chunyuan Li, Hao Liu, Changyou Chen, Yunchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. LICE: Towards understanding adversarial learning for joint distribution matching. 179.01109 2017.
R. D. Lord. The distribution of distance in a hypersphere. The Annals of Mathematical Statistics, 25 (4):794-798, 1954.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv: /5/ 1.0644, 2015.

---

Under review as a conference paper at ICLR 2020
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), pp. 1278-1286, 2014.
lyya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR), 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. It takes (only) two: Adversarial generator- encoder networks. In arXiv: 1704.v 023044, 2017.
Tom White. Sampling generative networks.
04468, 2016.
Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 RECONSTRUCTION ON FFHQ
VAL
FFHQ faces
VAE (normal)
SAE (ours)
FFHQ faces
VAE (normal)
SAE (ours)
FFHQ faces
VAE (normal)
SAE (ours)
Figure Reconstructed faces by VAE and SAE, Our SAE algorithm only uses the spherical constraint in equation (10) instead of priors.
11

---

Under review as a conference paper at ICLR 2020
A.2 SAMPLING VAE AND SAE ON FFHQ
VAE (normal)
SAE (ours)
Normal distribution
Uniform distribution
Poisson
Chi-squared distribution
Figure 7: Generated faces with inputs of different priors. With the pre-trained decoders of VAE and SAE, the faces generated with random vectors sampled from the four probability priors.
12

---

Under review as a conference paper at ICLR 2020
VAE (normal)
SAE (ours)
Normal distribution
Uniform distribution
Poisson
Chi-squared distribution
Figure 8: Generated faces with inputs of different priors. With the pre-trained decoders, the faces are generated with random vectors sampled from the four probability priors.
13

---

Under review as a conference paper at ICLR 2020
A.3 RECONSTRUCTION ON CELEBA
CelebA faces
VAE (normal)
SAE (ours)
CelebA faces
VAE (normal)
SAE (ours)
THE
Ge
CelebA faces
VAE (normal)
SAE (ours)
Figure Reconstructed faces by VAE and SAE. Our SAE algorithm only uses the spherical constraint in (TO) instead of priors.
14

---

Under review as a conference paper at ICLR 2020
A.4
SAMPLING ON CELEBA
VAE (normal)
SAE (ours)
Normal distribution
Uniform distribution
Poisson
Chi-squared distribution
Figure 10: Generated faces with inputs of different priors. With the pre- pre-trained decoders, the faces are generated with random vectors sampled from the four probability priors.
15

---

Under review as a conference paper at ICLR 2020
A.5
RECONSTRUCTION ON MNIST
7
7
0
4
1
4
9
7
4
6
4
3
0
0
8
5
6
6
S
7
8
5
9
0
6
9
0
1522
2
9
7
3
2
9
7
0
/
6
4
6
7
3
97
8
4
9
6
6
5
7
6
2
7
8
4
7
3
7
8
2
0
2
9
4 40740100
6
3
6
9
3
1
4
5
5
5
6
3
3 3
7
6
9
6
0
1054
5
1
6
4
5
4 2
5
4
1 7 4
9
9
2
(
9
4
8
7
4
4
7
2
3
4 4 146355600
6
9
9
4
4
9
2
8
8
8
5
0
119572
8
3
5
4
6
7
9
O
5
8
9
2
5
0
/
MNIST letters
7 7
3
/
O
4
/
4
9
7
5
0
4
3
0
7
2
8
5
8
6
5
7
8
9
0
6
9
0
1
5
0
7
1
7
3
7
9
7
0
/
6
4
9
7
3
9 7 9 4
9
6
5
9
6
2
7
8
4
7
3
7
1
9
2
0
2
9
9 A0944
0
3
13166
6
/
3
6
9
3
1
4
5
3
|
5
6
0
3
3 4 1
2
7
6
9
6
O
5
4
6
5
4
4
5
5
1 7 4
5
2
9
9
2
1
9
4
3
7
4
9
7
2
3
4 4
3
5
5
6
0
9
9
9
4
4
2
8
8
8
5
O
4
2
9
7
7
2
9
3
5
4
7
6
7
9
O
5
6
9
2
5
0
/
S-VAE (von Mises-
7
3
0
4
1
9
9
7
4
3
0
7
0
8
5
6
6
S
7
8
8
9
0
6
9
0
1
S
3
7
4
3
9
7
0
/
6
4
7
3
9
7
3
4
7
6
6
5
9
6
2
9
8
4
7
3
7
1
8
2
0
4
9
8
4
4
0
1
3
1
6
3
6
9
3
1
4
2
5
]
5
6
0
3
4
3
30727
0
2
1
1
9
7
6
9
6
0
5
4
6
5
4
6
5
0
1
7
4
4
3
5
1
2
9
9
0
/
9
4
8
7
f
4
9
7
2
3
2
4
4
3
5
5
6
0
3
9
0
4
4
9
8
1
8
5
0
4
9
T
7
2
9
3
5
4
7
6
7
9
O
5
8
9
2
5
O
7
VAE (normal)
7
2
/
0
4
/
4
9
7
(
4
3
07
0
8
5
6
6
S
7
8 3
6
0
6
9
0
2
9
7
3
2
9
7
0
/
6
4
7 A
3
97
3
4
9
6
6
5
1
6
2
7
8
4
7
3
7
8
2
0
9
4 0 1074013
6
3
6
9
3
1
4
5
5
1
5
6
0
3
4
3 4 447271211
7
C
9
6
0
5
4
1
6
5
6
5
4
5 7 0
1 7 4 1
/
9
4
8
7
4
4
7
2
3
2
4 4 6
446355603
9
7
9
4
4
9
8
1
8
8
5
4
9
5
2
9
3
5
4
6
7
9
O
5
8
9
2
5
0
SAE (ours)
Figure 11: Reconstructed letters by VAEs with different priors latent spaces. Our SAE algorithm only uses the spherical constraint in equation 10 instead of priors. For all experiments on MNIST, take d. 10.
16

---

Under review as a conference paper at ICLR 2020
A.6
SAMPLING S- S-VAE, VAE, AND SAE ON MNIST
S-VAD (von Mises-Fisher,
VAE (normal)
SAE (ours)
7
2
*
:
6
3
5
5
6 0
2
6 3
2
3.
9
3
7
9
8
9
7 9
9
4
1
6
7
0
6
3
7
9
9
7
7
4
7
0
/
J
C
G 5
2
8
3
0
9
_ 1
6
1
7
4
7
4
4
>
6
0
1
9
7
/ 4
0 2
4
1
1
9
8
7
à
7
0 /
1
1
9 9
7
2
0
2
4
6
3
1
7
9
1
9
///
5
6
1
2
3
6
5
0
r
9
7
4
0
9 9
5
4 4
0
1
0
L 1
,
5
0
4
2
7
2
0 7
9
J
7
8
7
P
5 S
8
0
9
3
1
7
L
4
1
7
8
Normal distribution
L
3
9
1
3 9
7
3
9
6
6
9
4
6
1.
!
6 5 U
6
0
6
2
3
7
2
3
7
5
0
9
2
6
1
6
1
4
5
5
I
7
6
b
3
7
3
& 7
4
5
1
3
8
9
6
0
L
61
U
8
f
L 0 20 0 3
7
3
8
/
6
6
9
5
6
5
Q
7
1
4
5
3 /
& 1
+ 9
|
3
3
9
à
6
3
9
3
4
3
7
0
8
3
T
2
T
8 9 3
$
8
7
8
5
1
6
+
4 Q
A
G
4
5
9
9
0
9
9
6
4
8
9
9
2 3
6
9
9
?
8
/ 4
9
3
1
J
7
7
4
/
9
87
5
5
3
7
5
9
5
G 2
Q
6
2
C 7
Uniform distribution
4
4
1
9
q
3
6
5
5
4
9 3 3 3
0
Y
6
6
1
5
1
7
Y
1
4
9
8
5
2
3
0
[
S
4
7
1
9
6
4
7
4
4 4
;
6
5
0
6
5
9 8
8
0 2
2
4
/
7 1
0 J 18 44/ 3 9 2 0
7
b
9
9 0
0
6
5
9
8
9
28
J
3
)
6
2
5
4
4
I
4
5
5
0
1 100
6
5
6
7
4
F
4
4 9
9
9 9
0
3
6
9
8
8 8
J
Y
4
4
7
4
9
7
8
3
3
9 9 O
2
7
1
4
D
9
4
3
8
7
Poisson
Y
8
0
8
2
3
R
7
2
6
3
0
9 0
7
2
b
1
2
5
\
9 1
0
9
2
6
0
6
5
9
6
1
9
b
7
1
6
0 0 7
5
0
3
0
00005
6 6
2
5
5
6 0
6
9
5 5
3 2 8
85844
c.
6
9
7
8
y D
5
9
0 24
3
Q
6
7
#
9 A
9
2 5
1
3
3
7
9
4 4
6
3
6
5 9
2
7
3
8
6
C
7
9
Chi Chi-squared distribution
Figure 12: Generated letters with inputs different priors. With the pre-trained decoders, the letters are generated with random vectors sampled from the four probability priors.
17

---

Under review as a conference paper at ICLR 2020
A.7 VISUALIZATION INFERENCE
(a) VAE
(b) SAE (ours)
Figure 13: Visualization of inferred codes z on CelebA with t-SNE. We randomly sample 5,000 faces from CelebA for illustration. The distribution of the latent codes from variational inference (VAE) shows standard normal one. However, the distribution of the latent codes from the spherical inference (SAE) is prone to be globally uniform while maintaining the variation density.
18

---

Under review as a conference paper at ICLR 2020
(a) VAE
(b) S-VAE (von Mises Mises-Fisher)
(b) SAE (ours)
Figure 14; Visualization of inferred codes MNIST with t-SNE. We randomly sample 500 letters from each class in MNIST to form the whole set for illustration. The latent codes derived from SAE much better than that from VAE and S-VAE. The margins between different classes are wider. meaning that the latent codes from the spherical inference conveys more discriminative information in the way of unsupervised learning. This experiment also indicates that SAE captures the intrinsic structure of multi-class data better than VAE and S V-E.
19


Reviews:
[{"review_id": "HJga0cRjYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an interesting idea that by regularizing the structure of the latent space into a sphere, we can free VAE from variantional inference framework.\n\nHowever, here are several concerns about this paper:\n1. is this a variant of the Wasserstein auto-encoder?\n2. the image quality of VAE (CelebA) is not that bad in other VAE papers, maybe tuning the \\beta-VAE can also achieve the same quantitative and qualitative results.\n3. Can you visualize the latent space (z) for the CelebA dataset, also comparing with the results from VAE?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1e_Hdw6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper considers the L2 normalization of samples “z” from a given prior p(z) in Generative Adversarial Netowks (GAN) and autoencoders. The L2 normalization corresponds to projecting samples onto the surface of a unit-hypersphere. Hence, to attempt to justify this normalization, the authors rely on some already established results regarding high dimensional hyperspheres. In particular, the focus is on the fact that, the Euclidean distance between any given point on a hypersphere and another randomly sampled point on the hypersphere tends to a constant, when the number of dimensions goes to infinity. This result is then used to show that the Wasserstein distance between two arbitrary distributions on a hypersphere converges to a constant when the number of dimensions grows. Based on this result, the authors claim that projecting the latent samples onto the surface of a hypersphere would make GAN less sensitive to the choice of the prior distribution. Moreover, they claim that such normalization would also benefits inference, and that it addresses the issue of variational inference in VAE.\n\nMain comments. \n\nThis paper is hard to follow and requires substantial improvements in terms of writing, owing to several grammatical and semantic issues. Moreover, there is a lack rigor; some important claims are supported neither by experiments nor by theoretical analysis. Experiments in the main paper are also weak. I can therefore not recommend acceptance. My detailed comments are below.\n\n- An important claim in this paper is that the proposed approach “alleviates variational inference in VAE”. However, this requires clarification as well as theoretical/empirical justifications.\n- In the introduction, it is stated that generated samples from VAE may deviate from real data samples, because “the posterior q(z|x) cannot match  the prior p(z) perfectly”. However, in VAE we do not expect the posterior to match the prior perfectly, as this would result in useless data representations or inference. Generation issues in VAE may rather be explained by the fact that, in this context we optimize a lower bound on the KL-divergence between the empirical data distribution and the model distribution. The latter objective does not penalize the model distribution if it puts some of its mass in regions where the empirical data distribution is very low or even zero.\n- Theorem 2 (on the convergence of the Wasserstein distance (W2) on high dimensional hyperspheres) does not seem to hold if, for instance, P and P’ are empirical distributions with overlapping supports. Further, even when the above Theorem holds, the W2 distance may be relatively high since it is proportional to the square root of the number of samples.\n- Moreover, why and how would Theorem 2 justify improved inference when projecting latent samples onto a hypersphere?\n- Please consider revising the following statement in the introduction: “The encoder f in VAE approximates the posterior q(z|x)”. The encoder “f” in VAE parametrizes the variational posterior.\n- Some typos,\n\t- Abstract, “… by sampling and inference tasks”  -- “on sampling …”\n\t- Introduction second paragraph after eq 2. “… it also causes the new problems” – “ … causes new problems”\n\t- Section 2.1, “For convenient analysis …” – “For a convenient …” \n\t- Second paragraph after Theorem 1. “… perform probabilistic optimizations … ” – “… optimization …” \n\t- Section 5.2, second paragraph. Is it Figure 9?\n\nThe main recommendations I would make are as follows.\n- Consider revising the paper to improve its writing.\n- Provide rigorous theoretical analysis and discussions to support the main claims. \n- Improve experiments by including more datasets and baselines (e.g., hyperspherical VAE [1]), as well conduct more targeted experiments to give more insights regarding the effect of the L2 normalization on inference and generation. \n\n[1] Davidson, Tim R., et al. \"Hyperspherical variational auto-encoders.\" UAI, 2018.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeMRX6atB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel autoencoder algorithm, named Spherical AutoEncoder (SAE). In this paper, the authors argue that the sphere structure has good properties in high-dimensional. To leverage the properties, proposed algorithm centerizes latent variables and projects them onto unit sphere. To show the empirical performance of the proposed approach, the authors perform image reconstruction and generation using FFHQ dataset and MNIST dataset.\n\nComments:\nI think the proposed approach, using spherical latent space, is interesting and make sense.\n\n- As mentioned in section 3.2, the proposed algorithm is reduced to standard autoencoder since it is free from posterior inference. Then, to clarify the algorithm, it seems necessary to provide the formulation of objective functions.\n- Is the objective still valid or reasonable even it is derived from the equation (10) without posterior inference?\n- How does the objective change when centerization and spherization are applied to the GAN?\n- Compared with using von Mises-Fisher distribution in the vanilla VAE, the advantage of the proposed method is not clear. To my understanding, the main difference seems to be whether using lower bound with posterior inference or deterministic framework without such approximation. However, there are no theoretical or empirical results to show the benefit of the proposed method. If theoretical or empirical results with reasonable intuition is provided, it will make the proposed algorithm more valuable.\n\nQuestions:\n- Compare to ProGAN and StyleGAN, is the contribution of the paper to applying centerization to GAN and centerization and spherization to autoencoder?\n- What dimension do you use as latent dimension in the experiments?\n- Does the choice of prior distribution affect the experimental results? If so, is there any compatible reason with the intuition of SAE?\n\nTypo:\nUnder equation (10) in page 5: \\tilde{z} should be \\hat{z}.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve VAE/GAN by performing variational inference with a constraint that the latent variables lie on a sphere. The reviewers find some technical issues with the paper (R3's comment regarding theorem 3). They also found that the method is not motivated well, and the paper is not convincing. Based on this feedback, I recommend to reject the paper.
Decision: Reject