Title: CRAMER-WOLD AUTOENCODER

Abstract: Assessing distance between the true and the sample distribution is a key component of many state of the art generative models, such as Wasserstein Autoencoder (WAE). Inspired by prior work on Sliced-Wasserstein Autoencoders (SWAE) and WAE-MMD (WAE using maximum mean discrepancy based distance function) we construct a new generative model -Cramer-Wold AutoEncoder (CWAE). CWAE cost function is based upon a characteristic kernel we introduce, called Cramer-Wold kernel, and has a a simple closed-form in the case of normal prior. As a consequence, while simplifying the optimization procedure (no need of sampling necessary to evaluate the distance function in the training loop), CWAE performance matches quantitatively and qualitatively that of WAE-MMD and often improves upon SWAE.

1. INTRODUCTION:

One of the crucial aspects in construction of generative models is devising effective method for computing and minimizing distance between the true and the model distribution. Originally in Variational Autencoder (VAE) (Kingma & Welling, 2014) this computation was carried out using variational methods. An important improvement was brought by the introduction of Wasserstein metric (Tolstikhin et al., 2017) and the construction of WAE-GAN and WAE-MMD models, which relax the need for variational methods. WAE-GAN requires a separate optimization problem to be solved to approximate the used divergence measure, while in WAE-MMD the discriminator has the closed-form obtained from a characteristic kernel, i.e. one that is injective on distributions Muandet et al. (2017) . A recent contribution to this trend of simplifying the construction of generative models is Sliced-Wasserstein Autoencoder (SWAE, Kolouri et al. (2018) ), where a significantly simpler AutoEncoder based model based on Wasserstein distance is proposed. The main innovation of SWAE was the introduction of the sliced-Wasserstein distance -a fast to estimate metric for comparing two distributions, based on the mean Wasserstein distance of one-dimensional projections. However, even in SWAE there is no close analytic formula that would enable computing the distance of the sample from the standard normal distribution. Consequently in SWAE two types of sampling are needed: (i) sampling from the prior distribution and (ii) sampling over one-dimensional projections.

Our main contribution is introduction of the Cramer-Wold distance between distributions, which has a closed-form for the distance of a sample from standard multivariate normal distribution. Its important feature is that it is given by a characteristic kernel which has a closed-form given by equation 7 for the product of radial Gaussians 1 . We use it to construct an AutoEncoder based generative model, called Cramer-Wold AutoEncoder (CWAE), in which the cost function, for a normal prior distribution, has a closed analytic formula. Thus CWAE can be seen as a borderline model between SWAE and WAE-MMD, as we use a characteristic kernel to discriminate distributions, but its formula comes from the application of the sliced approach. An introduction of sliced approach is given in the following section, the comparison of CWAE and WAE-MMD is given in Appendix, Section B. We benchmark CWAE against generative models using a non-parametrized distance functions: WAE-MMD (WAE variant from Tolstikhin et al. (2017) based on the classical MMD distance) and SWAE. Our results show that, while simplifying the optimization procedure by forgoing the need to sample data projections or from the prior distribution we obtain an AutoEncoder based generative model which obtains quantitatively and qualitatively similar results to WAE-MMD and SWAE models, see Section 5.

Let us now briefly describe the outline of the paper. In the following two sections we introduce and theoretically investigate the Cramer-Wold distance. However, the reader interested mainly in the construction of our generative AutoEncoder model can proceed directly to Section 4. Section 5 contains experiments. Conclusions are given in Section 6.

2. CRAMER-WOLD DISTANCE: CONSTRUCTION:

Motivated by the prevalent use of normal distribution as prior in modern generative models, we investigate whether it is possible to simplify optimization of such models. As the first step towards this, in this section we introduce Cramer-Wold distance, which has a simple analytical formula for computing normality of a high-dimensional sample. On a high level our approach uses the traditional L 2 distance of kernel-based density estimation, computed across multiple single-dimensional projections of the true data and the output distribution of the model. We base our construction on the following two popular tricks of the trade:

Sliced-based decomposition of a distribution: Following the footsteps of Kolouri et al. (2018) ; Deshpande et al. (2018) , the first idea is to leverage the Cramer-Wold Theorem (Cramér & Wold, 1936) and Radon Transform (Deans, 1983) to reduce computing distance between two distributions to one dimensional calculations. For v in the unit sphere S D ⊂ R D , the projection of the set X ⊂ R D onto the space spanned by v is given by v T X and the projection of N (m, αI) is N (v T m, α). Cramer-Wold theorem states that two multivariate distributions can be uniquely identified by their all one-dimensional projections. For example, to obtain the key component of SWAE model, i.e. the sliced-Wasserstein distance between two samples X, Y ∈ R D , we compute the mean Wasserstein distance between all one-dimensional projections:

EQUATION

where S D denotes the unit sphere in R D and σ D is the normalized surface measure on S D . This approach is effective since the one-dimensional Wasserstein distance between samples has the closed form, and therefore to estimate (1) one has to sample only over the projections.

Smoothing distributions: Using the sliced-based decomposition requires us to define distance between two sets of samples, in a single dimensional space. To this end we will use a trick-of-trade applied commonly in statistics in order to compare samples or distributions which is to first smoothen (sample) distribution with a Gaussian kernel. For the sample R = (r i ) i=1..n ⊂ R by its smoothing with Gaussian kernel N (0, γ) we understand

sm γ (R) = 1 n i N (r i , γ),

where by N (m, S) we denote the one-dimensional normal density with mean m and variance S. This produces a distribution with regular density, and is commonly used in kernel density estimation.

If R comes from the normal distribution with standard deviation close to one, the asymptotically optimal choice of γ is given by the Silverman's rule of thumb γ = ( 4 3n ) 2/5 , see Silverman (1986) . For continuous density f , its smoothing sm γ (f ) is given by the convolution with N (0, γ), and in the special case of Gaussians we have sm γ (N (m, S)) = N (m, S + γ). While in general kernel density estimations works well only in low-dimensional spaces, this fits the bill for us, as we will only compute distances on single dimensional projections of the data.

Cramer-Wold distance. We are now ready to introduce the Cramer-Wold distance. In a nutshell, we propose to compute the squared distance between two samples by considering the mean squared L 2 distance between their smoothed projections over all single dimensional subspaces. By the squared L 2 distance between functions f, g : R → R we refer to f -g2 2 = |f (x) -g(x)| 2 dx. A key feature of this distance is that it permits a closed-form in the case of normal distribution.

More precisely, the following algorithm fully defines the Cramer-Wold distance between two samples X = (x i ) i=1..n , Y = (y j ) j=1..k ⊂ R D (for illustration of Steps 1 and 2 see Figure 1 ):

1. given v in the unit sphere S(0, 1) ⊂ R D consider the projections v T X = (v T x i ) i=1..n and v T Y = (v T y j ) j=1..k , 2. compute the squared L 2 distance of the densities sm γ (v T X) and sm γ (v T X):

sm γ (v T X) -sm γ (v T Y ) 2 2 ,

3. to obtain squared Cramer-Wold distance average (integrate) the above formula over all possible v ∈ S D .

3. CRAMER-WOLD DISTANCE: THEORY:

The key theoretical outcome of this paper is that the result of the computation of the Cramer-Wold distance from the previous section can be simplified to a closed form solution. Consequently, to compute the distance of two samples there is no need of finding the optimal transport like in WAE or the necessity to sample over the projections like in SWAE. For the case of simplicity we provide in this section the formulas for the distance between two samples and the distance of a sample from the standard normal density. The general definition of Cramer-Wold metric is presented in Appendix, Section A. Theorem 3.1. Let X = (x i ) i=1..n , Y = (y j ) j=1..n ⊂ R D be given 2 . We formally define the squared Cramer-Wold distance by the formula

d 2 cw (X, Y ) := S D sm γ (v T X) -sm γ (v T Y ) 2 2 dσ D (v).

Then

EQUATION

where φ D (s) = 1 F 1 ( 1 2 ; D 2 ; -s) and 1 F 1 is the Kummer's confluent hypergeometric function (see, e.g., Barnard et al. (1998) ). Moreover, φ D (s) has the following asymptotic formula valid for D ≥ 20:

EQUATION

)

To prove the Theorem 3.1 we will need the following crucial technical proposition. Proposition 3.1. Let z ∈ R D and γ > 0 be given. Then

EQUATION

)

Proof. By applying orthonormal change of coordinates without loss of generality we may assume that z = (z 1 , 0, . . . , 0), and then v T z = z 1 v 1 for v = (v 1 , . . . , v D ). Consequently we get

S D N (v T z, γ)(0) dσ D (v) = S D N (z 1 v 1 , γ)(0) dσ D (v)

. Making use of the formula for slice integration of functions on spheres (Axler et al., 1992, Corollary A.6 ) we get:

S D f dσ D = V D-1 V D 1 -1 (1 -x 2 ) (D-3)/2 S D-1 f (x, √ 1 -x 2 ζ) dσ D-1 (ζ) dx,

where V K denotes the surface volume of a sphere S K ⊂ R K . Applying the above equality for the function

f (v 1 , . . . , v D ) = N (z 1 v 1 , γ)(0) and s = z 2 1 /(2γ) = z 2 /(2γ)

we consequently get that the LHS of (4) simplifies to

V D-1 V D 1 √ 2πγ 1 -1 (1 -x 2 ) (D-3)/2 exp(-sx 2 ) dx, which completes the proof since V K = 2•π K 2 Γ( K 2 )

and

1 -1 exp(-sx 2 )(1 -x 2 ) (D-3)/2 dx = √ π Γ( D-1 2 ) Γ( D 2 ) 1 F 1 1 2 ; D 2 ; -s .

Proof of Theorem 3.1. Directly from the definition of smoothing we obtain that

EQUATION

Now applying the one-dimensional formula the the L 2 -scalar product of two Gaussians:

N (r 1 , γ 1 ), N (r 2 , γ 2 ) 2 = N (r 1 -r 2 , γ 1 + γ 2 )(0)

and the equality f -

g 2 2 = f, f 2 + g, g 2 -2 f, g 2 (where f, g 2 = f (x)g(x)

dx), we simplify the squared-L 2 norm in the integral of RHS of ( 5) to

1 n i N (v T x i , γ) -1 n i N (v T y j , γ) 2 2 = 1 n 2 i N (v T x i , γ), i N (v T x i , γ) 2 + 1 n 2 j N (v T y j , γ), j N (v T y j , γ) 2 -2 n 2 i N (v T x i , γ), j N (v T y j , γ) 2 = 1 n 2 ii N (v T (x i -x i ), 2γ)(0) + 1 n 2 jj N (v T (y j -y j ), 2γ)(0) -2 n 2 ij N (v T (x i -y j ), 2γ)(0).

Applying directly Proposition 3.1 we obtain formula (2). Proof of the formula for the asymptotics of the function φ D is provided in the Appendix.

Thus to estimate the distance of a given sample X to some prior distribution f , one can follow the common approach and take the distance between X and a sample from f . As the main theoretical result of the paper we view the following theorem, which says that in the case of standard Gaussian multivariate prior, we can completely reduce the need for sampling (we omit the proof since it is similar to that of Theorem 3.1). Theorem 3.2. Let X = (x i ) i=1..n ⊂ R D be a given sample. We formally define

d 2 cw (X, N (0, I)) := S D sm γ (v T X) -sm γ (N (0, 1)) 2 2 dσ D (v).

Then

d 2 cw (X, N (0, I)) = 1 2n 2 √ π 1 √ γ i,j φ D ( xi-xj 2 4γ ) + n 2 √ 1+γ -2n √ γ+ 1 2 i φ D ( xi 2 2+4γ ) . (6)

One can easily obtain the general formula for the distance between mixtures of radial distributions. This follows from the fact that the Cramer-Wold distance is given by a scalar product •, • cw which has a closed-form for the product of two radial Gaussians:

EQUATION

The above formula means that Cramer-Wold distance is defined by Cramer-Wold kernel, for more details see Appendix, Section A.

4. CRAMER-WOLD AUTOENCODER MODEL (CWAE):

This section is devoted to the construction of CWAE. Since we base our construction on the AutoEncoder, to establish notation let us formalize it here.

AutoEncoder. Let X = (x i ) i=1..n ⊂ R N be a given data set. The basic aim of AE is to transport the data to a typically, but not necessarily, less dimensional latent space Z = R D with reconstruction error as small as possible. Thus, we search for an encoder E : R n → Z and decoder D : Z → R n functions, which minimize the reconstruction error on the data set X:

M SE(X; E, D) = 1 n n i=1 x i -D(Ex i ) 2 .

AutoEncoder based generative model. CWAE, similarly to WAE, is a classical AutoEncoder model with modified cost function which forces the model to be generative, i.e. ensures that the data transported to the latent space comes from the (typically Gaussian) prior f . This statement is formalized by the following important remark, see also (Tolstikhin et al., 2017) . Remark 4.1. Let X be an N -dimensional random vector from which our data set was drawn, and let Y be a random vector with a density f on latent Z.

Suppose that we have constructed functions E : R N → Z and D : Z → R N (representing the encoder and the decoder) such that3 

1. D(Ex) = x for x ∈ image(X),

2. random vector EX has the distribution f .

Then by the point 1 we obtain that D(EX) = X, and therefore DY has the same distribution as D(EX) = X.

This means that to produce samples from X we can instead produce samples from Y and map them by the decoder D.

Since an estimator of the image of the random vector X is given by its sample X, we conclude that a generative model is correct if it has small reconstruction error and resembles the prior distribution in the latent. Thus, to construct a generative AutoEncoder model (with Gaussian prior), we add to its cost function a measure of distance of a given sample from normal distribution.

CWAE cost function.:

Once the crucial ingredient of CWAE is ready, we can describe its cost function. To ensure that the data transported to the latent space Z are distributed according to the standard normal density, we add to the cost function logarithm4 of the Cramer-Wold distance from standard multivariate normal density

EQUATION

Since the use of special functions involved in the formula for Cramer-Wold distance might be cumbersome, we apply in all experiments (except for the illustrative 2D case) the asymptotic form (12) of function φ D :

2 √ πd 2 cw (X) ≈ 1 n 2 ij (γ n + xi-xj 2 2D-3 ) -1/2 + (1 + γ n ) -1/2 -2 n i (γ n + 1 2 + xi 2 2D-3 ) -1/2 ,

where γ n = ( 4 3n ) 2/5 is chosen by the Silverman's rule of thumb (Silverman, 1986) .

Comparison with WAE and SWAE models. Finally, let us briefly recapitulate differences between the introduced CWAE, WAE variants of (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018) . In contrast to WAE-MMD and SWAE, CWAE model does not require sampling from normal distribution (as in WAE-MMD) or over slices (as in SWAE) to evaluate its cost function, and in this sense uses a closed formula cost function. In contrast to WAE-GAN, our objective does not require a separately trained neural network to approximate the optimal transport function, thus avoiding pitfalls of adversarial training. In this paper we are interested in WAE-MMD and SWAE models, which do not use parameterized distance functions, e.g. trained adversarially like in WAE-GAN. However, in future work we plan to introduce an adversarial version of CWAE and compare it with WAE-GAN.

5. EXPERIMENTS:

In this section we empirically validate the proposed CWAE model on standard benchmarks for generative models: CelebA, Cifar-10 and MNIST. We will compare CWAE model with WAE-MMD (Tolstikhin et al., 2017) and SWAE (Kolouri et al., 2018) . As we will see, our results match those of WAE-MMD, and in some cases improve upon SWAE, while using a simpler to optimize cost function (see the previous section for a more detailed discussion). The rest of this section is structured as follows. In Section 5.2 we report results on standard qualitative tests, as well as a visual investigations of the latent space. In Section 5.3 we will turn our attention to quantitative tests using Fréchet Inception Distance and other metrics.

5.1. EXPERIMENTATION SETUP:

In the experiment we have used two basic architecture types. Experiments on MNIST were performed using a feedforward network for both encoder and decoder, and a 20 neuron latent layer, all with ReLU activations. In case of CIFAR-10 and CelebA data sets we used convolution-deconvolution architectures. Please refer to Appendix E for full details. The quality of a generative model is typically evaluated by examining samples or interpolations. We present such a comparison between CWAE with WAE-MMD in Figure 2 . We follow the same procedure as in (Tolstikhin et al., 2017) . In particular, we use the same base neural architecture for both CWAE and WAE-MMD. We consider for each model (i) interpolation between two random examples from the test set (leftmost in Figure 2 ), (ii) reconstruction of a random example from the test set (middle column in Figure 2 ), and finally a sample reconstructed from a random point sampled from the prior distribution (right column in Figure 2 ). The experiment shows that there are no perceptual differences between CWAE and WAE-MMD generative distribution.

5.2. QUALITATIVE TESTS:

In the next experiment we qualitatively assess normality of the latent space. This will allow us to ensure that CWAE does not compromise on the normality of its latent distribution, which recall is part of the cost function for all the models except AE. We compare CWAE5 with AE, VAE, WAE and SWAE on the MNIST data with using 2-dimensional latent space and a two dimensional Gaussian prior distribution. Results are reported in Figure 3 . As is readily visible, the latent distribution of CWAE is as close, or perhaps even closer, to the normal distribution than that of the other models. Furthermore, the AutoEncoder presented in the second figure is noticeably different from a Gaussian distribution, which is to be expected because it does not optimize for normality in contrast to the other models.

To summarize, both in terms of perceptual quality and satisfying normality objective, CWAE matches WAE-MMD. The next section will provide more quantitative studies.

5.3. QUANTITATIVE TESTS:

In order to quantitatively compare CWAE with other models, in the first experiment we follow the common methodology and use the Fréchet Inception Distance (FID) introduced by Heusel et al. (2017) . Further, we evaluate the sharpness of generated samples using the Laplace filter following Tolstikhin et al. (2017) . Results for CWAE and WAE are summarized in Tab. 1. In agreement with the qualitative studies, we observe FID and sharpness scores of CWAE to be similar to WAE-MMD. 

(•) of a sample X = (x i ) i=1..n ⊂ R D : b 1,D (X) = 1 n 2 j,k (x T j x k ) 3 , and b 2,D (X) = 1 n j x j 4 ,

are close to that of standard normal density. The expected Mardia's skewness and kurtosis for standard multivariate normal distribution is 0 and D(D + 2), respectively. To enable easier comparison in experiments we consider also the value of the normalized Mardia's kurtosis given by b 2,D (X) -D(D + 2), which equals zero for the standard normal density. Optimal value of kurtosis, (i.e. for normal distribution) is given by a dash line.

Results are presented in Figure 4 and Table 2 . In Figure 4 Moreover, motivated by the above approach based on normality tests6 we have verified how the Cramer-Wold metric works as a Gaussian goodness of fit, however, the results were not satisfactory.

The tests based on Cramer-Wold metric were, in general, in the middle of compared tests (Mardia, Henze-Zirkler and Royston tests).

On the whole, WAE-MMD and CWAE achieve, practically speaking, the same level of performance in terms of FID score, sharpness, and our newly introduced normality test. Additionally, CWAE fares better in many of these metrics than SWAE.

6. CONCLUSIONS:

In the paper we have presented a new autoencoder based generative model CWAE, which matches results of WAE-MMD, while using a cost function given by a simple closed analytic formula. We hope this result will encourage future work in developing simpler to optimize analogs of strong neural models.

Crucial in the construction of CWAE is the use of the developed Cramer-Wold metric between samples and distributions, which can be effectively computed for Gaussian mixtures. As a consequence we obtain a reliable measure of the divergence from normality. Future work could explore use of the Cramer-Wold distance in other settings, in particular in adversarial models.

APPENDICES A CRAMER-WOLD KERNEL:

In this section we first formally define the Cramer-Wold metric, and later show that it is given by a characteristic kernel which has closed-form for spherical Gaussians. For more information on the kernels, and in general kernel embedding of distributions we refer the reader to (Muandet et al., 2017) .

Let us first introduce the general definition of the cw-metric. To do so we generalize the notion of smoothing for arbitrary measures µ by the formula:

sm γ (µ) = µ * N (0, γI),

where * denotes the convolution operator for two measures, and we identify the normal density N (0, γI) with the measure it introduces. It is well-known that the resulting measure has the density given by

x → N (x, γI)(y)dµ(y).

Clearly sm γ (N (0, αI)) = N (0, (α + γ)I)). Moreover, by applying the characteristic function one obtains that if the smoothing of two measures coincide, then the measures also coincide:

EQUATION

We also need to define the transport of the density by the projection x → v T x, where v is chosen from the unit sphere S D . The definition is formulated so that if X is a random vector with density f , then f v is the density of the random vector X v := v T X. Then we have

f v (r) = y:y-rv⊥v f (z)d D-1 (z),

where d D-1 denotes the D -1-dimensional Lebesgue measure. In general, if µ is a measure on R D , then µ v is the measure defined on R by the formula

µ v (A) = µ({x : v T x ∈ A}).

Since, if a random vector X has the density N (a, γI), then the random variable X v has the density N (v T a, α), we directly conclude that

N (a, γI) v = N (v T a, γ).

It is also worth noticing, that due to the fact that the projection of a Gaussian is a Gaussian, the smoothing and projection operators commute, i.e.:

sm γ (µ v ) = (sm γ µ) v .

Given fixed γ > 0, the two above notions allow us to formally define the cw-distance of two measures µ and ν by the formula

EQUATION

First observe that this implies that cw-distance is given by the kernel function

k(µ, ν) = S D sm γ (µ v ), sm γ (ν v ) L2 dσ D (v).

Let us now prove that the function d cw defined by equation 10 is a metric (which, in the kernel function literature means that the kernel is characteristic).

Theorem A.1. The function d cw is a metric.

Proof. Since d cw comes from a scalar product, we only need to show that if a distance of two measures is zero, the measures coincide.

So let µ, ν be given measures such that d cw (µ, ν) = 0. This implies that

sm γ (µ v ) = sm γ (ν v ).

By equation 9 this implies that µ v = ν v . Since this holds for all v ∈ S D , by the Cramer-Wold Theorem we obtain that µ = ν.

Thus we can summarize the above by saying that the Cramer-Wold kernel is a characteristic kernel which has the closed-form the scalar product of two radial Gaussians given by equation 7:

N (x, αI), N (y, βI) cw = 1 √ 2π(α+β+2γ) φ D x-y 2 2(α+β+2γ) .

Remark A.1. Observe, that except for the Gaussian kernel it is the only kernel which has the closed form for the spherical Gaussians, which as we discuss in the next section is important, as the RBF (Gaussian) kernels cannot be successfully applied in AutoEncoder based generative models. The reason is that the derivative of Gaussian decreases to fast, and therefore it does not enforce the proper learning of the model, see also the comments in (Tolstikhin et al., 2017, Section 4, WAE-GAN and WAE-MMD specifics) .

B CWAE VS WAE-MMD:

In this section we are going to compare CWAE model to WAE-MMD. In particular we show that CWAE can be seen as the intersection of the sliced-approach together with MMD-based models.

Since both WAE and CWAE use kernels to discriminate between sample and normal density, to compare the models we first describe the WAE model. where in experiments in (Tolstikhin et al., 2017) a value C = 2Dσ 2 was used, where σ is the hyper-parameter denoting the size of the normal density. Thus the model has hyper-parameters λ and σ, which were chosen to be λ = 10, σ 2 = 1 in MNIST, λ = 100, σ 2 = 2 in CelebA. Observe that the hyper-parameters do not depend on the sample size and that, in general, the WAE-MMD model hyper-parameters have to be chosen by hand. Now let us describe the CWAE model. CWAE cost function for a sample X = (x i ) i=1..n ⊂ R D (in the D-dimensional latent) is given by CWAE cost = M SE + log d 2 cw (X, N (0, I)), where distance between the sample and standard normal distribution is taken with respect to the Cramer-Wold kernel with a regularizing hyperparameter γ given by the Silverman's rule of thumb (the motivation for such a choice of hyper-parameters is explained in Section 2). Thus, we have the following differences:

• Due to the properties of Cramer-Wold kernel, in the distance we are able to substitute the sample estimation of d 2 k (X, N (0, I)) given in WAE-MMD by d 2 cw (X, Y ) by its exact formula.

• CWAE, as compared to WAE, has no hyper-parameters:

1. In our preliminary experiments we have observed that in many situations (like in the case of log-likelihood), taking the logarithm of the nonnegative factors of the cost function, which we aim to minimize to zero, improves the learning process. Motivated by this, instead of taking the additional weighting hyper-parameter λ (as in WAE-MMD), whose aim is to balance the MSE and divergence terms, we take the logarithm of the divergence. Automatically (independently of dimension) balance those terms in the learning process. 2. The choice of regularization hyper-parameter is given by the Silverman's rule of thumb, and depends on the sample size (contrary to WAE-MMD, where the hyper-parameters are chosen by hand, and in general do not depend on the sample size).

Summarizing, in CWAE model, contrary to WAE-MMD, we do not have to choose hyper-parameters. Moreover, since we do not have the noise in the learning process given by the random choice of the sample Y from N (0, I), the learning should be more stable. As a consequence, see Figure 7 , CWAE in generally learns faster then WAE-MMD, and has smaller standard deviation of the cost-function during the learning process.

C COMPUTATION OF φ D

In this section we consider the estimation of values of the function φ D (s) = 1 F 1 ( 1 2 ; D 2 ; -s) for s ≥ 0, which is crucial in the formulation for the Cramer-Wold distance. First we will provide its approximate asymptotic formula valid for dimensions D ≥ 20, and then we shall consider the special case of D = 2 (see Figure 5 )

To do so, let us first recall (Abramowitz & Stegun, 1964, Chapter 13 ) that the Kummer's confluent hypergeometric function 1 F 1 (denoted also by M ) has the following integral representation

1 F 1 (a, b, z) = Γ(b) Γ(a)Γ(b -a) 1 0 e zu u a-1 (1 -u) b-a-1 du, valid for a, b > 0 such that b > a. Since we consider that latent is at least of dimension D ≥ 2, it follows that φ D (s) = Γ( D 2 ) Γ( 1 2 )Γ( D 2 -1 2 ) 1 0 e -su u -1/2 (1 -u) D/2-3/2 du.

By making a substitution u = x 2 , du = 2xdx, we consequently get Proposition C.1. For large7 D we have

EQUATION

EQUATION

Proof. By (11) we have to estimate asymptotics of

φ D (s) = Γ( D 2 ) Γ( 1 2 )Γ( D 2 -1 2 ) 1 -1 e -sx 2 (1 -x 2 ) (D-3)/2 dx.

Since for large D, for all x ∈ [-1, 1] we have

(1 -x 2 ) (D-3)/2 e -sx 2 ≈ (1 -x 2 ) (D-3)/2 • (1 -x 2 ) s = (1 -x 2 ) s+(D-3)/2 , we get φ D (s) ≈ Γ( D 2 ) Γ( D-1 2 ) √ π • 1 -1 (1 -x 2 ) s+(D-3)/2 dx = Γ( D 2 ) Γ( D-1 2 ) √ π • √ π Γ(s + D 2 -1 2 ) Γ(s + D 2 )

.

To simplify the above we apply the formula (1) from (Tricomi & Erdélyi, 1951) :

Γ(z + α) Γ(z + β) = z α-β (1 + (α -β)(α + β -1) 2z + O(|z| -2 )),

with α, β fixed so that α + β = 1 (so only the error term of order O(|z| -2 ) remains), and get

Γ( D 2 ) Γ( D-1 2 ) = Γ(( D 2 -3 4 ) + 3 4 ) Γ(( D 2 -3 4 ) + 1 4 ) ≈ D 2 - 3 4 1 2 and Γ(s + D 2 -1 2 ) Γ(s + D 2 ) ≈ s + D 2 - 3 4 -1 2 . (13) Summarizing, φ D (s) ≈ ( D 2 -3 4 ) 1/2 (s + D 2 -3 4 ) 1/2 = (1 + 4s 2D-3 ) -1/2 .

In general one can obtain the iterative direct formulas for function φ D with the use of erf and modified Bessel functions of the first kind I 0 and I 1 , but for large D they are of little numerical value. We consider here only the special case D = 2 since it is used in the paper for illustrative reasons in the latent for the MNIST data set. Since we have the equality (Gradshteyn & Ryzhik, 2015, (8.406.3 ) and (9.215.3)):

φ 2 (s) = 1 F 1 ( 1 2 , 1, -s) = e -s 2 I 0 s 2 ,

to practically implement φ 2 we apply the approximation of I 0 from (Abramowitz & Stegun, 1964, page 378) given in the following remark.

Remark C.1. Let s ≥ 0 be arbitrary and let t = s/7.5. Then

φ 2 (s) ≈                 

e -s 2 • (1+3.5156229t 2 +3.0899424t 4 +1.2067492t 6 +.2659732t 8 +.0360768t 10 +.0045813t 12 ) for s ∈ [0, 7.5],

2 s • (.39894228+.01328592t -1 +.00225319t -2 -.00157565t -3 +.0091628t -4 -.02057706t -5 +.02635537t -6 -.01647633t -7 +.00392377t -8 ) for s ≥ 7.5.

D COMPARISON OF LEARNING TIMES:

Figure 6 gives comparison of mean learning time for different most frequently used batch-sizes. Time spent on processing a batch is actually smaller for CWAE for a practical range of batch-sizes [32, 512] . For batch-sizes larger than 1024, CWAE is slower due to its quadratic complexity with respect to the batch-size. However, we note that batch-sizes larger even than 512 are relatively rarely used in practice for training autoencoders.

Additionally, to have a direct comparison to WAE-MMD model on CelebA, an identical architecture was used as that in Tolstikhin et al. (2017) utilized for the WAE-MMD model (WAE-GAN architecture is, naturally, different):

encoder four convolution layers with 5 × 5 filters, each layer followed by a batch normalization (consecutively 128, 256, 512, and 1024 channels) and ReLU activation, latent 64-dimensional, decoder dense 1024 neuron layer, three transposed-convolution layers with 5 × 5 filters, and each layer followed by a batch normalization with ReLU activation (consecutively 512, 256, and 128 channels), transposed-convolution layer with 5 × 5 filter and 3 channels, clipped output value.

The results for this architecture for CWAE compared to VAE and WAE-MMD models are given in Table 1 .

Similarly to Tolstikhin et al. (2017) , models were trained using Adam with for 55 epochs, with the same optimizer parameters. 

:

For more information we refer the reader to Appendix, Sections A and B.

For clarity of presentation we provide here the formula for the case of samples of equal size.

We recall that for function (or in particular random vector) X : Ω → R D by image(X) we denote the set consisting of all possible values X can attain, i.e. {X(ω) : ω ∈ Ω}.

We take the logarithm of the Cramer-Wold distance to improve balance between the two terms in the objective function.

Since (3) is valid for dimensions D ≥ 20, to implement CWAE in 2-dimensional latent space we apply equality 1F1(1/2, 1, -s) = e -s 2 I0 s 2 jointly with the approximate formula(Abramowitz & Stegun, 1964, page 378) for the Bessel function of the first kind I0, for more details see Appendix C.

For more information on the statistical tests based on the kernel approach we refer the reader to(Muandet et al., 2017, Subsection 3.5).

In practice we can take D ≥ 20.

Figures and Tables:

Figure 1: Cramer-Wold distance of two sets obtained as the mean squared L 2 -distance of their smoothed projections on all onedimensional lines.

Figure 2: CWAE achieves perceptually similar results to WAE-MMD. Results of WAE-MMD (first row) and CWAE (second row) on CelebA data set of original WAE-MMD architecture. Left: Interpolations between two examples from the test distribution (left to right, in each row). Middle: Reconstruction of examples from the test distribution; odd rows correspond to the real test points. Right: Reconstructed examples from a random samples from the prior distribution.

Figure 3: Latent distribution of CWAE is close to the normal distribution. Each subfigure presents points sampled from two-dimensional latent spaces, AE, VAE, WAE, SWAE, and CWAE (left to right). All trained on the MNIST data set.

Figure 4: Metrics assessing normality of the model output distributions, during training: FID score, Mardia's skewness and kurtosis of models WAE, SWAE and CWAE, on the CelebA test set. The values given are mean from 5 experiments of differently initialised models together with standard deviation confidence intervals. Low standard deviation of CWAE indicates its stability during learning. Optimal value of kurtosis, (i.e. for normal distribution) is given by a dash line.

WAE cost function for a given characteristic kernel k and sample X = (x i ) i=1..n ⊂ R D (in the D-dimensional latent) is given by WAE cost = M SE + λ • d 2 k (X, Y ), where Y = (y i ) i=1..n is a sample from the standard normal density N (0, I), and d 2 k (X, Y ) denotes the kernel-based distance between the probability distributions representing X and Y , that is 1 n i δ xi and 1 n i δ yi , where δ z denotes the atom Dirac measure at z ∈ R D . The inverse multiquadratic kernel k is chosen as default k(x, y) = C C + x -y 2 2 ,

Figure 5: Comparison of φ D value (red line) with the approximation given by equation 12 (green line) in the case of dimensions D = 2, 5, 20. Observe that for D = 20, the functions practically coincide.

Figure 8: CIFAR10 trained CWAE, WAE, and SWAE models with FID score, kurtosis and skewness, as well as CW-, WAE-, and SWAE-distances. Results are given both related to epochs and learning time. All values are mean from 5 models trained for each architecture. Confidence intervals represent the standard deviation. Optimum kurtosis is marked by a dash line.



Table content: <table><tr><td colspan="3">: CWAE achieves similar</td></tr><tr><td colspan="3">FID (lower is better) and sharpness</td></tr><tr><td colspan="3">(higher is better) to WAE-MMD on</td></tr><tr><td colspan="3">the original WAE architecture (see</td></tr><tr><td colspan="2">Appendix E for details).</td><td/></tr><tr><td>Algorithm</td><td colspan="2">FID Sharpness</td></tr><tr><td>SWAE</td><td>72</td><td>0.008</td></tr><tr><td>VAE</td><td>63</td><td>0.003</td></tr><tr><td>WAE-MMD</td><td>55</td><td>0.006</td></tr><tr><td>CWAE</td><td>54</td><td>0.006</td></tr><tr><td>WAE-GAN</td><td>42</td><td>0.006</td></tr><tr><td>True data</td><td>2</td><td>0.020</td></tr></table>

Comparison between different models output distributions and the normal distribution, together with reconstruction error. All models outputs except AE are similarly close to the normal distribution. Normality is assessed by comparing Mardia's skewness, kurtosis, and the reconstruction error. For reference FID scores are provided as well (except for MNIST, where it is not defined).

Table content: <table><tr><td>Data set</td><td>Method</td><td>AE</td><td>VAE</td><td colspan="2">WAE SWAE</td><td>CWAE</td></tr><tr><td>MNIST</td><td>Skewness</td><td>659.67</td><td>0.49</td><td>82.12</td><td>55.59</td><td>52.83</td></tr><tr><td/><td>Kurtosis (normalized)</td><td>749.58</td><td>-410.69</td><td colspan="2">35.61 -37.43</td><td>95.77</td></tr><tr><td/><td>Reconstruction error</td><td>2.10</td><td>4.12</td><td>2.11</td><td>2.11</td><td>2.13</td></tr><tr><td colspan="2">CIFAR10 Skewness</td><td>11444.35</td><td colspan="3">3.07 1893.10 996.01</td><td>171.67</td></tr><tr><td/><td>Kurtosis (normalized)</td><td colspan="5">-2219.50 -4158.33 2346.49 193.15 1943.35</td></tr><tr><td/><td>Reconstruction error</td><td>49.67</td><td>82.82</td><td>45.80</td><td>44.84</td><td>45.52</td></tr><tr><td/><td>FID score error</td><td>400.14</td><td>218.43</td><td colspan="2">146.34 145.04</td><td>121.16</td></tr><tr><td>CelebA</td><td>Skewness</td><td>59770025.50</td><td>22.07</td><td colspan="2">301.71 196.64</td><td>59.22</td></tr><tr><td/><td>Kurtosis (normalized)</td><td>1363931.65</td><td>53.09</td><td colspan="2">942.68 507.39</td><td>307.29</td></tr><tr><td/><td>Reconstruction error</td><td>139.30</td><td>142.46</td><td colspan="2">139.10 138.23</td><td>138.54</td></tr><tr><td/><td>FID score error</td><td>307.70</td><td>95.35</td><td colspan="2">96.30 100.56</td><td>97.22</td></tr></table>

we report for CelebA data set the value of FID score, Mardia's skewness and kurtosis during learning process of AE, VAE, WAE, SWAE and CWAE (measured on the validation data set). WAE, SWAE and CWAE models obtain the best reconstruction error, comparable to AE. VAE model exhibits a sightly worse reconstruction error, but values of kurtosis and skewness indicating their output is closer to normal distribution. As expected, the output of AE is far from normal distribution; its kurtosis and skewness grow during learning. This arguably less standard evaluation, which we hope will find adoption in the community, serves as yet another evidence that CWAE has strong generative capabilities which at least match performance of WAE-MMD. Moreover we observe that VAE model's output distribution is closest to the normal distribution, at the expense of the reconstruction error, which is reflected by the blurred reconstructions typically associated with VAE model.

Table content: <table/>

Peer review (iclr2019):
 The paper introduces a novel regularized auto-encoder architecture called the Cramer-Wold AutoEncoders (CWAE). It*s objective (Eq. 7) consists of two terms: (i) a standard reconstruction term making sure the the encoder-decoder pair aligns nicely to accurately reconstruct all the training images and (ii) the regularizer, which roughly speaking requires the encoded training distribution to look similar to the standard normal (which is a prior used in the generative model being trained). The main novelty of the paper is in the form of this regularizer. The authors introduce what they call "the Cramer-Wold distance" (for definitions see Theorems 3.1 and 3.2) which is defined between two finite sets of D-dimensional points. The authors provide empirical studies showing that the proposed CWAE method achieves the same quality of samples (measured with FID scores) as the WAE-MMD model [1] previously reported in the literature, while running faster (by up to factor of 2 reduction in the training time, as the authors report). While on the AE model / architecture side I feel the contribution is very marginal, I still think that the improvement in the training speed is something useful. Otherwise it is a nicely written and polished piece of work. Detailed comments: (1) My main problem with this paper is that the novel objective proposed by the authors in Eq. 7 is equivalent to the objective of WAEs appearing in Eq. 4 of [1] (up to a heuristic of applying logarithm to the divergence measure, which is not justified but meant to "improve the balance between two terms", see footnote 2), where the authors use the newly introduced Cramer-Wold divergence as a choice of the penalty term in Eq. 4 of [1]. (2) When viewed in this way, CW-distance introduced in Eq (2) closely resembles the unbiased U-statistic estimate of the MMD used in WAE-MMD [1, Algorithm 2]. In other words: it may be the case that there is a choice of a reproducing kernel k such that Eq. 2 of this paper is an estimate of MMD_k between two distributions based on the i.i.d. samples X and Y. Note that if it is indeed the case, this corresponds to the V-statistic and thus biased: in U-statistic the diagonal terms (that is i = i* and j = j* in forst two terms of eq 2) would be omitted. If this all is indeed the case, it is not surprising that the numbers the authors get in the experiments are so similar to WAE-MMD, because CWAE would be exactly WAE-MMD with a specific choice of the kernel. (3) The authors make a big deal out of their proposed divergence measure not requiring samples from the prior as opposed to WAE-MMD. However, WAE-MMD does not necessarily need to sample from the prior when used with Gaussian prior and Gaussian RBF kernel, because in this case the prior-related parts of the MMD can be computed analytically. In other words, if the computational advantage of CWAE compared to WAE-MMD comes from CWAE not sampling Pz, the computational overhead of WAE-MMD can be eliminated at least in the above-mentioned setting. (4) based on the name "CW distance" I would expect the authors to actually prove that it is indeed a distance (i.e. all the main axioms). (5) The authors override the CW distance: first in Theorem 3.1 they define it as a distance between two finite point clouds, and later in Theorem 3.2 they redefine it as a distance between a point cloud and the Gaussian distribution. (6) What is image(X) in Remark 4.1? [1] Tolstikhin et al., Wasserstein Auto-Encoders, 2017.