Cramer-Wold AutoEncoder

1 INTRODUCTION . One of the crucial aspects in construction of generative models is devising effective method for computing and minimizing distance between the true and the model distribution . Originally in Variational Autencoder ( VAE ) ( Kingma & Welling , 2014 ) this computation was carried out using variational methods . An important improvement was brought by the introduction of Wasserstein metric ( Tolstikhin et al. , 2017 ) and the construction of WAE-GAN and WAE-MMD models , which relax the need for variational methods . WAE-GAN requires a separate optimization problem to be solved to approximate the used divergence measure , while in WAE-MMD the discriminator has the closed-form obtained from a characteristic kernel , i.e . one that is injective on distributions Muandet et al . ( 2017 ) . A recent contribution to this trend of simplifying the construction of generative models is Sliced-Wasserstein Autoencoder ( SWAE , Kolouri et al . ( 2018 ) ) , where a significantly simpler AutoEncoder based model based on Wasserstein distance is proposed . The main innovation of SWAE was the introduction of the sliced-Wasserstein distance – a fast to estimate metric for comparing two distributions , based on the mean Wasserstein distance of one-dimensional projections . However , even in SWAE there is no close analytic formula that would enable computing the distance of the sample from the standard normal distribution . Consequently in SWAE two types of sampling are needed : ( i ) sampling from the prior distribution and ( ii ) sampling over one-dimensional projections . Our main contribution is introduction of the CramerWold distance between distributions , which has a closed-form for the distance of a sample from standard multivariate normal distribution . Its important feature is that it is given by a characteristic kernel which has a closed-form given by equation 7 for the product of radial Gaussians1 . We use it to construct an AutoEncoder based generative model , called Cramer-Wold AutoEncoder ( CWAE ) , in which the cost function , for a normal prior distribution , has a closed analytic formula . Thus 1For more information we refer the reader to Appendix , Sections A and B. CWAE can be seen as a borderline model between SWAE and WAE-MMD , as we use a characteristic kernel to discriminate distributions , but its formula comes from the application of the sliced approach . An introduction of sliced approach is given in the following section , the comparison of CWAE and WAE-MMD is given in Appendix , Section B . We benchmark CWAE against generative models using a non-parametrized distance functions : WAE-MMD ( WAE variant from Tolstikhin et al . ( 2017 ) based on the classical MMD distance ) and SWAE . Our results show that , while simplifying the optimization procedure by forgoing the need to sample data projections or from the prior distribution we obtain an AutoEncoder based generative model which obtains quantitatively and qualitatively similar results to WAE-MMD and SWAE models , see Section 5 . Let us now briefly describe the outline of the paper . In the following two sections we introduce and theoretically investigate the Cramer-Wold distance . However , the reader interested mainly in the construction of our generative AutoEncoder model can proceed directly to Section 4 . Section 5 contains experiments . Conclusions are given in Section 6 . 2 CRAMER-WOLD DISTANCE : CONSTRUCTION . Motivated by the prevalent use of normal distribution as prior in modern generative models , we investigate whether it is possible to simplify optimization of such models . As the first step towards this , in this section we introduce Cramer-Wold distance , which has a simple analytical formula for computing normality of a high-dimensional sample . On a high level our approach uses the traditional L2 distance of kernel-based density estimation , computed across multiple single-dimensional projections of the true data and the output distribution of the model . We base our construction on the following two popular tricks of the trade : Sliced-based decomposition of a distribution : Following the footsteps of Kolouri et al . ( 2018 ) ; Deshpande et al . ( 2018 ) , the first idea is to leverage the Cramer-Wold Theorem ( Cramér & Wold , 1936 ) and Radon Transform ( Deans , 1983 ) to reduce computing distance between two distributions to one dimensional calculations . For v in the unit sphere SD ⊂ RD , the projection of the set X ⊂ RD onto the space spanned by v is given by vTX and the projection of N ( m , αI ) is N ( vTm , α ) . Cramer-Wold theorem states that two multivariate distributions can be uniquely identified by their all one-dimensional projections . For example , to obtain the key component of SWAE model , i.e . the sliced-Wasserstein distance between two samples X , Y ∈ RD , we compute the mean Wasserstein distance between all one-dimensional projections : dW ( X , Y ) = ∫ SD dW ( v TX , vTY ) dσD ( v ) , ( 1 ) where SD denotes the unit sphere in RD and σD is the normalized surface measure on SD . This approach is effective since the one-dimensional Wasserstein distance between samples has the closed form , and therefore to estimate ( 1 ) one has to sample only over the projections . Smoothing distributions : Using the sliced-based decomposition requires us to define distance between two sets of samples , in a single dimensional space . To this end we will use a trick-of-trade applied commonly in statistics in order to compare samples or distributions which is to first smoothen ( sample ) distribution with a Gaussian kernel . For the sample R = ( ri ) i=1 .. n ⊂ R by its smoothing with Gaussian kernel N ( 0 , γ ) we understand smγ ( R ) = 1 n ∑ i N ( ri , γ ) , where by N ( m , S ) we denote the one-dimensional normal density with mean m and variance S. This produces a distribution with regular density , and is commonly used in kernel density estimation . If R comes from the normal distribution with standard deviation close to one , the asymptotically optimal choice of γ is given by the Silverman ’ s rule of thumb γ = ( 43n ) 2/5 , see Silverman ( 1986 ) . For continuous density f , its smoothing smγ ( f ) is given by the convolution with N ( 0 , γ ) , and in the special case of Gaussians we have smγ ( N ( m , S ) ) = N ( m , S + γ ) . While in general kernel density estimations works well only in low-dimensional spaces , this fits the bill for us , as we will only compute distances on single dimensional projections of the data . Cramer-Wold distance . We are now ready to introduce the Cramer-Wold distance . In a nutshell , we propose to compute the squared distance between two samples by considering the mean squared L2 distance between their smoothed projections over all single dimensional subspaces . By the squared L2 distance between functions f , g : R → R we refer to ‖f − g‖22 = ∫ |f ( x ) − g ( x ) |2dx . A key feature of this distance is that it permits a closed-form in the case of normal distribution . More precisely , the following algorithm fully defines the Cramer-Wold distance between two samples X = ( xi ) i=1 .. n , Y = ( yj ) j=1 .. k ⊂ RD ( for illustration of Steps 1 and 2 see Figure 1 ) : 1. given v in the unit sphere S ( 0 , 1 ) ⊂ RD consider the projections vTX = ( vTxi ) i=1 .. n and vTY = ( vT yj ) j=1 .. k , 2. compute the squared L2 distance of the densities smγ ( vTX ) and smγ ( vTX ) : ‖smγ ( vTX ) − smγ ( vTY ) ‖22 , 3. to obtain squared Cramer-Wold distance average ( integrate ) the above formula over all possible v ∈ SD . 3 CRAMER-WOLD DISTANCE : THEORY . The key theoretical outcome of this paper is that the result of the computation of the Cramer-Wold distance from the previous section can be simplified to a closed form solution . Consequently , to compute the distance of two samples there is no need of finding the optimal transport like in WAE or the necessity to sample over the projections like in SWAE . For the case of simplicity we provide in this section the formulas for the distance between two samples and the distance of a sample from the standard normal density . The general definition of Cramer-Wold metric is presented in Appendix , Section A. Theorem 3.1 . Let X = ( xi ) i=1 .. n , Y = ( yj ) j=1 .. n ⊂ RD be given2 . We formally define the squared Cramer-Wold distance by the formula d2cw ( X , Y ) : = ∫ SD ‖smγ ( vTX ) − smγ ( vTY ) ‖22 dσD ( v ) . Then d2cw ( X , Y ) = 1 2n2 √ πγ ( ∑ ii′ φD ( ‖xi−xi′‖ 2 4γ ) + ∑ jj′ φD ( ‖yj−yj′‖ 2 4γ ) − 2 ∑ ij φD ( ‖xi−yj‖2 4γ ) ) , ( 2 ) where φD ( s ) = 1F1 ( 12 ; D 2 ; −s ) and 1F1 is the Kummer ’ s confluent hypergeometric function ( see , e.g. , Barnard et al . ( 1998 ) ) . Moreover , φD ( s ) has the following asymptotic formula valid for D ≥ 20 : φD ( s ) ≈ ( 1 + 4s2D−3 ) −1/2 . ( 3 ) To prove the Theorem 3.1 we will need the following crucial technical proposition . Proposition 3.1 . Let z ∈ RD and γ > 0 be given . Then∫ SD N ( vT z , γ ) ( 0 ) dσD ( v ) = 1√ 2πγ φD ( ‖z‖2 2γ ) . ( 4 ) Proof . By applying orthonormal change of coordinates without loss of generality we may assume that z = ( z1 , 0 , . . . , 0 ) , and then vT z = z1v1 for v = ( v1 , . . . , vD ) . Consequently we get∫ SD N ( vT z , γ ) ( 0 ) dσD ( v ) = ∫ SD N ( z1v1 , γ ) ( 0 ) dσD ( v ) . Making use of the formula for slice integration of functions on spheres ( Axler et al. , 1992 , Corollary A.6 ) we get : ∫ SD f dσD = VD−1 VD ∫ 1 −1 ( 1− x 2 ) ( D−3 ) /2 ∫ SD−1 f ( x , √ 1− x2 ζ ) dσD−1 ( ζ ) dx , 2For clarity of presentation we provide here the formula for the case of samples of equal size . where VK denotes the surface volume of a sphere SK ⊂ RK . Applying the above equality for the function f ( v1 , . . . , vD ) = N ( z1v1 , γ ) ( 0 ) and s = z21/ ( 2γ ) = ‖z‖2/ ( 2γ ) we consequently get that the LHS of ( 4 ) simplifies to VD−1 VD 1√ 2πγ ∫ 1 −1 ( 1− x 2 ) ( D−3 ) /2 exp ( −sx2 ) dx , which completes the proof since VK = 2·π K 2 Γ ( K2 ) and ∫ 1 −1 exp ( −sx 2 ) ( 1 − x2 ) ( D−3 ) /2 dx = √ π Γ ( D−12 ) Γ ( D2 ) 1F1 ( 1 2 ; D 2 ; −s ) . Proof of Theorem 3.1 . Directly from the definition of smoothing we obtain that d2cw ( X , Y ) = ∫ SD ∥∥ 1 n ∑ i N ( vTxi , γ ) − 1n ∑ j N ( vT yj , γ ) ∥∥2 2 dσD ( v ) . ( 5 ) Now applying the one-dimensional formula the the L2-scalar product of two Gaussians : 〈N ( r1 , γ1 ) , N ( r2 , γ2 ) 〉2 = N ( r1 − r2 , γ1 + γ2 ) ( 0 ) and the equality ‖f − g‖22 = 〈f , f〉2 + 〈g , g〉2 − 2〈f , g〉2 ( where 〈f , g〉2 = ∫ f ( x ) g ( x ) dx ) , we simplify the squared-L2 norm in the integral of RHS of ( 5 ) to ‖ 1n ∑ i N ( vTxi , γ ) − 1n ∑ i N ( vT yj , γ ) ∥∥2 2 = 1n2 〈 ∑ i N ( vTxi , γ ) , ∑ i N ( vTxi , γ ) 〉2 + 1n2 〈 ∑ j N ( vT yj , γ ) , ∑ j N ( vT yj , γ ) 〉2 − 2n2 〈 ∑ i N ( vTxi , γ ) , ∑ j N ( vT yj , γ ) 〉2 = 1n2 ∑ ii′ N ( vT ( xi − xi′ ) , 2γ ) ( 0 ) + 1n2 ∑ jj′ N ( vT ( yj − yj′ ) , 2γ ) ( 0 ) − 2n2 ∑ ij N ( vT ( xi − yj ) , 2γ ) ( 0 ) . Applying directly Proposition 3.1 we obtain formula ( 2 ) . Proof of the formula for the asymptotics of the function φD is provided in the Appendix . Thus to estimate the distance of a given sample X to some prior distribution f , one can follow the common approach and take the distance between X and a sample from f . As the main theoretical result of the paper we view the following theorem , which says that in the case of standard Gaussian multivariate prior , we can completely reduce the need for sampling ( we omit the proof since it is similar to that of Theorem 3.1 ) . Theorem 3.2 . Let X = ( xi ) i=1 .. n ⊂ RD be a given sample . We formally define d2cw ( X , N ( 0 , I ) ) : = ∫ SD ‖smγ ( vTX ) − smγ ( N ( 0 , 1 ) ) ‖22dσD ( v ) . Then d2cw ( X , N ( 0 , I ) ) = 1 2n2 √ π ( 1√ γ ∑ i , j φD ( ‖xi−xj‖2 4γ ) + n2√ 1+γ − 2n√ γ+ 12 ∑ i φD ( ‖xi‖2 2+4γ ) ) . ( 6 ) One can easily obtain the general formula for the distance between mixtures of radial distributions . This follows from the fact that the Cramer-Wold distance is given by a scalar product 〈· , ·〉cw which has a closed-form for the product of two radial Gaussians : 〈N ( x , αI ) , N ( y , βI ) 〉cw = 1√ 2π ( α+β+2γ ) φD ( ‖x−y‖2 2 ( α+β+2γ ) ) . ( 7 ) The above formula means that Cramer-Wold distance is defined by Cramer-Wold kernel , for more details see Appendix , Section A .

Summary: The paper introduces a novel regularized auto-encoder architecture called the Cramer-Wold AutoEncoders (CWAE). It's objective (Eq. 7) consists of two terms: (i) a standard reconstruction term making sure the the encoder-decoder pair aligns nicely to accurately reconstruct all the training images and (ii) the regularizer, which roughly speaking requires the encoded training distribution to look similar to the standard normal (which is a prior used in the generative model being trained). The main novelty of the paper is in the form of this regularizer. The authors introduce what they call "the Cramer-Wold distance" (for definitions see Theorems 3.1 and 3.2) which is defined between two finite sets of D-dimensional points. The authors provide empirical studies showing that the proposed CWAE method achieves the same quality of samples (measured with FID scores) as the WAE-MMD model [1] previously reported in the literature, while running faster (by up to factor of 2 reduction in the training time, as the authors report).