Title: No title found

Abstract: For distributions p and q with different support, the divergence D(p||q) generally will not exist. We define a spread divergence D(p||q) on modified p and q and describe sufficient conditions for the existence of such a divergence. We give examples of using a spread divergence to train implicit generative models, including linear models (Principal Components Analysis and Independent Components Analysis) and non-linear models (Deep Generative Networks).

1. INTRODUCTION:

A divergence D(p||q) (see, for example Dragomir (2005) ) is a measure of the difference between two distributions p and q with the property D(p||q) ≥ 0, and D(p||q) = 0 ⇔ p = q (1) Some of our results are specific to the f -divergence, defined as

EQUATION

where f (x) is a convex function with f (1) = 0. An important special case of an f -divergence is the well-known Kullback-Leibler divergence KL(p||q) = E p(x) log p(x) q(x) which is widely used to train models using maximum likelihood. We are interested in situations in which the supports of the two distributions are different, supp (p) = supp (q). In this case the divergence may not be defined. For example, for p(x) being an empirical data distribution on continuous dataset x 1 , . . . , x N , p(x) = 1 N N n=1 δ (x -x n ) where δ (•) is the Dirac Delta function. For a model q(x) with support R, then KL(q||p) is not formally defined. This is a challenge since implicit generative models of the form q(x) = δ (x -g θ (z)) p(z)dz only have limited support; in this case maximum likelihood to learn the model parameter θ is not available and alternative approaches are required -see Mohamed & Lakshminarayanan (2016) for a recent survey.

2. SPREAD DIVERGENCES:

The aim is, from q(x) and p(x) to define new distributions q(y) and p(y) that have the same support1 . Using the notation x to denote integration (•) dx for continuous x, and x∈X for discrete x with domain X , we define a random variable y with the same domain as x and distributions p(y) = x p(y|x)p(x), q(y) =

x p(y|x)q(x)

where p(y|x) is a 'noise' process designed to 'spread' the mass of p and q such that p(y) and q(y) have the same support. For example, if we use a Gaussian p(y|x) = N y x, σ 2 , then p and q both have support R. We therefore use noise with the property that, despite D(p||q) not existing, D(p||q) does exist and we define the Spread Divergence D(p||q) ≡ D(p||q) (4)

Note that this satisfies the divergence requirement D(p||q) ≥ 0. The second requirement, D(p||q) = 0 ⇔ p = q, is guaranteed for certain 'noise' processes, as described in section(2.1).

Spread divergences have many potential applications. For example, for a model q(x|θ) with parameter θ and empirical data distribution p(x), maximum likelihood training corresponds to minimising KL(p||q) with respect to θ. However, for implicit models, the divergence KL(p||q) does not exist. However, if a spread divergence exists, provided that the data is distributed according the model p(x) = q(x|θ 0 ) for some unknown parameter θ 0 , the spread divergence D(p(x)||q(x|θ)) has a minimum at θ = θ 0 . That is (for identifiable models) we can correctly learn the underlying data generating process, even when the original divergence is not defined.

2.1. NOISE REQUIREMENTS FOR A SPREAD DIVERGENCE:

Our main interest is in using noise to define a new divergence in situations in which the original divergence D(p||q) is itself not defined. For discrete variables x ∈ {1, . . . , n}, y ∈ {1, . . . , n}, the noise

P ij = p(y = i|x = j) must be a distribution i P ij = 1, P ij ≥ 0 and j P ij p j = j P ij q j ∀i ⇒ p j = q j ∀j (5)

which is equivalent to the requirement that the matrix P is invertible, see appendix(B). There is an additional requirement that the spread divergence exists. In the case of f -divergences, the spread divergence exists provided that p and q have the same support. This is guaranteed if

j P ij p j > 0, j P ij q j > 0 ∀i (6)

which is satisfied if P ij >0. In general, therefore, there is a space of noise distributions p(y|x) that define a valid spread divergence. The 'antifreeze' method of Furmston & Barber (2009) is a special form of spread noise to define a valid Kullback-Leibler divergence (see also Barber (2012) ).

For continuous variables, in order that D(p||q) = 0 ⇒ p = q, the noise p(y|x), with dim(Y ) = dim(X) must be a probability density and satisfy

p(y|x)p(x)dx = p(y|x)q(x)dx ∀y ∈ Y ⇒ p(x) = q(x) ∀x ∈ X (7)

This is satisfied if there exists a transform p -1 such that

EQUATION

where δ (•) is the Dirac delta function. As for the discrete case, the spread divergence exists provided that p and q have the same support, which is guaranteed if p(y|x)>0. A well known example of such an invertible integral transform is the Weierstrass Transform p(y|x) = N y x, σ 2 , which has an explicit representation for p -1 . In general, however, we can demonstrate the existence of a spread divergence without the need for an explicit representation of p -1 . As we will see below, the noise requirements for defining a valid spread divergence such that D(p||q) = 0 ⇔ p = q are analogous to the requirements on kernels such that the Maximum Mean Discrepancy MMD(p, q) = 0 ⇔ p = q, see Sriperumbudur et al. (2011) and Sriperumbudur et al. (2012) .

3. STATIONARY SPREAD DIVERGENCES:

Consider stationary noise p(y|x) = K(y -x) where K(x) is a probability density function with K(x)>0, x ∈ R. In this case p and q are defined as a convolution

p(y) = K(y -x)p(x)dx = (K * p) (y), q(y) = K(y -x)q(x)dx = (K * q) (y) (9)

Since K>0, p and q are guaranteed to have the same support R. A sufficient condition for the existence of the Fourier Transform F {f } of a function f (x) for real x is that f is absolutely integrable. All distributions p(x) are absolutely integrable, so that both F {p} and F {q} are guaranteed to exist. Assuming F {K} exists, we can use the convolution theorem to write

EQUATION

Hence, we can write

D(p||q) = 0 ⇔ p = q ⇔ F {K} F {p} = F {K} F {q} ⇔ F {p} = F {q} ⇔ p = q (11)

where we used the invertibility of the Fourier transform and assumed that F {K} = 0, or equivalently2 , F {K} >0. Hence, provided that K(x)>0 and F {K} >0 then K(x) defines a valid spread divergence. Note that other transforms have a corresponding convolution theorem3 and the above derivation holds, with the requirement that the corresponding transform of K(x) is non-zero. As an example of such a noise process, consider Gaussian noise,

K(y -x) = 1 √ 2πσ 2 e -1 2σ 2 (y-x) 2 (12)

leading to a positive Fourier Transform:

EQUATION

Similarly, for Laplace noise

K(x) = 1 2b e -1 b |x| p(y|x) = K(y -x) = 1 2b e -1 b |y-x| , F {K} (ω) = 2 π b -1 b -2 + ω 2 > 0 (14)

Since K>0 and F {K} >0, this also defines a valid spread divergence over R.

3.1. INVERTIBLE MAPPINGS:

Consider p(y|x) = K(y -f (x)) for strictly monotonic f . Then, using the change of variables

EQUATION

where J is the Jacobian of f . For distributions p(x) with bounded domain, for example x ∈ [0, 1], we can use a logit function, f (x) = -log x -1 -1 , which maps the interval [0, 1] to R. Using then, for example, Gaussian spread noise p(y|x) = N y f (x), σ 2 , both p(y) and q(y) have support R. If D(p||q) is zero then p = q on the domain [0, 1].

3.2. MAXIMISING THE SPREAD:

From the data processing inequality (see appendix(A)), spread noise will always decrease the f -

divergence D f (p(y)||q(y)) ≤ D f (p(x)||q(x)

). If we are to use a spread divergence to train a model, there is the danger that adding too much noise may make the spreaded empirical distribution and spreaded model distribution so similar that it becomes difficult to numerically distinguish them, impeding training. In general, therefore, it would be useful to add noise such that we define a valid spread divergence, but can maximally still discern the difference between the two distributions. To gain intuition, we define p and q to generate data in separated linear subspaces,

p(x) = δ (x -a -Az) p(z)dz, q(x) = δ (x -b -Bz) p(z)dz, p(z) = N (z 0, I Z ). Using Gaussian spread, p(y|x) = N (y µ, Σ)

, what is the optimal µ, Σ that maximises the divergence? Clearly, as Σ tends to zero, the divergence increases to infinity, meaning that we must at least constrain the entropy of p(y|x) to be finite. In this case the spreaded distributions are given by

p(y) = N y µ + a, AA T + Σ , q(y) = N y µ + b, BB T + Σ (16)

We define a simple Factor Analysis noise model with Σ = σ 2 I + uu T , where σ 2 is fixed and u T u = 1. The entropy of p(y|x) is then fixed and independent of u. Also, for simplicity, we assume A = B. It is straightforward to show that the spread divergence KL(p||q) is maximised for u pointing orthogonal to the vector AA T + σ 2 I -1 (b -a). Then u optimally points along the direction in which the support lies. The support of p(y|x) must be the whole space but to maximise the divergence the noise preferentially spreads along directions defined by p and q, see figure(1). the linear subspace defined by the origin a and direction A. The upper dotted line denotes Gaussian distributed data q(x) with support different from p(x). Optimally, to maximise the spread divergence between the two distributions, for fixed noise entropy, we should add noise that preferentially spreads out along the directions defined by p and q, as denoted by the ellipses.

4. MERCER SPREAD DIVERGENCE:

We showed in section(3) how to define one form of spread divergence, with the result that stationary noise distributions must have strictly positive Fourier Transforms. A natural question is whether, for continuous x, there are other easily definable noise distributions that are non-stationary. To examine this question, let x ∈ [a, b], y ∈ [a, b] and K(x, y) = K(y, x) be square integrable, K(x, y) ∈ L 2 . We define Mercer noise p(y|x) = K(x, y)/K(x), where K(x) = K(x, y)dy. For strictly positive definite K, by Mercer's Theorem, it admits an expansion

EQUATION

where the eigenfunctions φ n form a complete orthogonal set of L 2 [a, b] and all λ n >0, see for example Sriperumbudur et al. (2011) . Then

EQUATION

and p(y) = q(y) is equivalent to the requirement

EQUATION

Multiplying both sides by φ m (y) and integrating over y we obtain

EQUATION

If p(x)/K(x) and q(x)/K(x) are in L 2 [a, b] then, from Mercer's Theorem, they can be expressed as orthogonal expansions

EQUATION

Then, equation( 20) is

EQUATION

which reduces to (using orthonormality), γ p m = γ q m ⇒ p = q. Hence, provided K(x, y) = K(y, x) is square integrable on [a, b] and strictly positive definite, then K(x, y)/ K(x, y)dy defines valid spread noise. For example, K(x, y) = exp -λ 1 (x 2 + y 2 ) +exp -λ 2 (x 2 + y 2 ) defines a strictly positive non-stationary square integrable kernel on [a, b] . Provided p(x)/K(x) and q(x)/K(x) are in L 2 [a, b] then the spread noise p(y|x) = K(x, y)/K(x) defines a valid spread divergence.

5. APPLICATIONS:

We demonstrate using a spread divergence to train implicit models

EQUATION

)

where θ are the parameters of the encoder g. We show that, despite the likelihood not being defined, we can nevertheless successfully train the models using an EM style algorithm, see for example Barber (2012) . We then show how to train a deterministic non-linear generative model using a variational approximation.

5.1. DETERMINISTIC LINEAR LATENT MODEL:

For observation noise γ, the Probabilistic PCA model (Tipping & Bishop, 1999) for X-dimensional observations and Z-dimensional latent is

x = F z + γ , z ∼ N (0, I Z ), ∼ N (0, I X ), p θ (x) = N y 0, F F T + γ 2 I X (24)

When γ = 0, the generative mapping from z to x is deterministic and the model p θ (x) has support only on a subset of R X and the data likelihood is in general not defined. In the following we consider general γ, setting γ to zero at the end of the calculation. To fit the model to iid data {x 1 , . . . , x N } using maximum likelihood, the only information required from the dataset is the data covariance Σ. The maximum likeihood solution for PPCA is then

F = U Z Λ Z -γ 2 I Z 1 2 R

, where Λ Z , U Z are the Z largest eigenvalues, eigenvectors of Σ; R is an arbitrary orthogonal matrix. Using spread noise p(y|x) = N y x, σ 2 I X , the spreaded distribution pθ (y) is a Gaussian

EQUATION

Thus, pθ (y) is of the same form as PPCA, albeit with an inflated covariance matrix. Adding Gaussian spread noise to the data also simply inflates the sample covariance to Σ = Σ + σ 2 I X . Since the eigenvalues of Σ ≡ Σ+σ 2 I X are simply Λ = Λ+σ 2 I X , with unchanged eigenvectors, the optimal deterministic (γ = 0) latent linear model has solution

F = U Z Λ Z -σ 2 I Z 1 2 R = U Z Λ 1 2

Z R. Unsurprisingly, this is the standard PCA solution; however, the derivation is non-standard since the likelihood of the deterministic latent linear model is not defined. Nevertheless, using the spread divergence, we learn a sensible model and recover the true data generating process if the data were exactly generated according to the deterministic model.

5.2. DETERMINISTIC INDEPENDENT COMPONENTS ANALYSIS:

ICA corresponds to the model

EQUATION

where the independent components z i follow a non-Gaussian distribution. For Gaussian noise ICA an observation x is assumed to be generated by the process

EQUATION

where g i (z) mixes the independent latent process z. In standard linear ICA, g j (z) = a T j z where a j is the j th column on the mixing matrix A. For small observation noise γ 2 , the EM algorithm (Bermond & Cardoso, 1999) becomes ineffective. To see this, consider X = Z and invertible mixing matrix A, x = Az. At iteration k the EM algorithm has an estimate A k of the mixing matrix. The M-step updates A k to datapoints from the model x = Az, for independent zero mean unit variance Laplace components on z. The elements of A used to generate the data are uniform random ±1. We use S y = 1, S z = 1000 samples and 2000 EM iterations to estimate the mixing matrix. The relative error is averaged over all i, j and 10 random experiments. We also plot standard errors around the mean relative error. In blue we show the error in learning the underlying parameter using the standard EM algorithm. As expected, as γ → 0, the error blows up as the EM algorithm 'freezes'. In orange we plot the error for EM using spread noise, as described in section(5.2.1); no slowing down appears as the model noise γ decreases. As the model noise increases, the quality of the learned model under spread noise decreases gradually. In (b) we show that, apart from very small N , the error for the spread EM algorithm is lower than for the standard EM algorithm. Here Z = 5, X = 10, S = 1000, γ = 0.2, with 500 EM updates used. Results are averaged over 50 runs of randomly drawn A.

EQUATION

where, for noiseless data (γ = 0),

EQUATION

)

where Ŝ ≡ 1 N n x n x T n is the moment matrix of the data. Thus,

A k+1 = ŜA -T k A -1 k ŜA -T k -1 = A k

. and the algorithm 'freezes'. Similarly, for low noise γ 1 progress critically slows down. Whilst over-relaxation methods, see for example Winther & Petersen (2007) can help in the case of small noise, for zero noise γ = 0, over-relaxation is of no benefit.

5.2.1. HEALING CRITICAL SLOWING DOWN:

To deal with small noise and the limiting case of a deterministic model (γ = 0), we consider Gaussian spread noise p(y|x) = N y x, σ 2 I X to give

p(y, z) = p(y|x)p(x, z)dx = i N y g i (z), γ 2 + σ 2 I X i p(z i ) (30)

The empirical distribution is replaced by the spreaded empirical distribution

EQUATION

)

The M-step has the same form as equation( 28) but with modified statistics

EQUATION

E zz T = 1 N n N y x n , σ 2 p(z|y)zz T dzdy (33)

The E-step optimally sets where Z q (y) is a normaliser and

p(z|y) = 1 Z q (y) N (z µ(y), Σ) i p(z i ), Z q (y) = N (z µ(y), Σ) i p(z i )dz (34)

Σ = (γ 2 + σ 2 ) A T A -1 , µ(y) = A T A -1 Ay (35)

We can rewrite the expectations required for the E-step of the EM algorithm as

E yz T = 1 N n N y x n , σ 2 N (z µ(y), Σ) i p(z i ) Z q (y) yz T dzdy (36) E zz T = 1 N n N y x n , σ 2 N (z µ(y), Σ) i p(z i ) Z q (y) zz T dzdy (37)

Generally the posterior p(z|y) will be peaked around N (z µ(y), Σ) and writing the expectations with respect to N (z µ(y), Σ) allows for an effective sampling approximation focussed on regions of high probability. We implement this update by drawing S y samples from N y x n , σ 2 I and, for each y sample, we draw S z samples from N (z µ(y), Σ). This scheme has the advantage over more standard variational approaches, see for example Winther & Petersen (2007) , in that we obtain a consistent estimator of the M-step update for A4 . We show results for a toy experiment in figure(2), learning the underlying mixing matrix in a deterministic non-square setting. Note that standard algorithms such as FastICA (Hyvärinen, 1999) fail in this setting. The noise value is set to σ = max(0.001, 2.5 * sqrt(mean(AA T ))), for estimated mixing matrix A of the underlying deterministic model x n = Az n , n = 1, . . . , N . The EM algorithm learns a good approximation of the unknown mixing matrix and latent components z n , with no critical slowing down.

5.3. TRAINING IMPLICIT NON-LINEAR MODELS:

For a deterministic non-linear implicit model, we set p(z) = N (z 0, I) and parameterise g θ (x) by a deep neural network. The likelihood equation( 23) is in general intractable and it is natural to consider a variational approximation (Kingma & Welling, 2013) ,

EQUATION

However, since p θ (x|z) = δ (x -g θ (z)) this bound is not well defined. Instead, we minimise the spread divergence KL(p(y)||p θ (y)). The approach is a straightforward extension of the standard variational autoencoder and in appendix(C) we provide details of how to do this, along with higher resolution images of samples from the generative model. We dub this model and associated spread divergence training the 'δVAE'. As a demonstration, we trained a generative network on the MNIST dataset, see figure(3) and appendix(D). We used Gaussian spread noise σ = 1 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE. The network g θ (x) contains 8 layers, each layer with 400 units and relu activation function and latent dimension Z = 64. We also trained a deep convolutional generative model on the CelebA dataset (Liu et al., 2015) , see figure(4) and appendix(E). We pre-process CelebA images by first taking 140x140 centre crops and then resizing to 64x64. Pixel values were then rescaled to lie in [0, 1]. We use Gaussian spread noise σ = 0.5 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE.

6. SUMMARY:

We described an approach to defining a divergence, even when two distributions to not have the same support. The method introduces a 'noise' variable to 'spread' mass from each distribution to cover the same domain. Previous approaches (Furmston & Barber, 2009; Sønderby et al., 2016) can be seen as special cases. We showed that defining divergences this way enables us to train deterministic generative models using standard 'likelihood' based approaches. Indeed, for simple models such as Independent Components Analysis, we showed how we can implement a principled learning method based on classical EM training, without the standard difficulty of critical slowing down in the case of small (or zero) observation noise.

Introducing noise means that an additional expectation is required. This can be carried out, in part, exactly, although additional approximations using perturbation theory are possible, similar to Roth et al. (2017) . Spread divergences have deep connections to other approaches to define measures of disagreement between distributions. In particular, one can view the spread divergence as the probabilistic analogue of MMD, with conditions required for the existence of the spread divergence closely related to the universality requirement on MMD kernels (Micchelli et al., 2006) .

Theoretically, we can learn the underlying true data generating process by the use of any valid spread divergence -for example for fixed Gaussian spread noise. In practice, however, the quality of the learned model can depend on the choice of spread noise. In this work we fixed the spread noise, but showed that if we were to learn the spread noise, it would preferentially spread mass across the manifolds defining the two distributions. In future work, we will investigate learning spread noise to maximally discriminate two distributions, which would involve a minimax model training objective, with an inner maximisation over the spread noise and an outer maximisation over the model parameters. This would bring our work much closer to adversarial training methods (Goodfellow, 2017) .

A SPREAD NOISE MAKES DISTRIBUTIONS MORE SIMILAR:

The data processing inequality for f -divergences (see for example Gerchinovitz et al. (2018) ) states that D f (p(y)||q(y)) ≤ D f (p(x)||q(x)). For completeness, we provide here an elementary proof of this result. We consider the following joint distributions q(y, x) = p(y|x)q(x), p(y, x) = p(y|x)p(x)

whose marginals are the spreaded distributions

EQUATION

The divergence between the two joint distributions is

EQUATION

The f -divergence between two marginal distributions is no larger than the f -divergence between the joint (see also Zhang et al. (2018) ). To see this, consider

EQUATION

Intuitively, spreading two distributions increases their overlap, reducing the divergence. When p and q do not have the same support, D f (q(x)||p(x)) can be infinite or not well-defined.

B INJECTIVE LINEAR MAPPINGS:

Consider an injective linear mapping T from space V to W . From the rank nullity theorem for finite dimensional spaces,

EQUATION

Since image(T ) ⊆ W , it must be that image(T ) = W . Hence, injective linear maps between between two (finite dimensional) spaces of the same dimension are surjective; equivalently, they are invertible.

In the context of spread noise, since the domain of x and y are equal and p(y) is defined through a linear transformation of p(x), the requirement in (5) that the mapping is injective is equivalent to the requirement that the mapping is invertible.

C SPREAD DIVERGENCE FOR DETERMINISTIC DEEP GENERATIVE MODELS:

Instead of minimising the likelihood, we train an implicit generative model by minimising the spread divergence

EQUATION

We note that for Σ φ independent of y, we can partially integrate equation( 54 Similar to Roth et al. (2017) , in principle, one can form a perturbation approximation of the above to second order in x and express the integral over the spread noise as a form of regularisation; however, in our experiments we found that the above works well -we therefore leave such analysis for future work.

D MNIST EXPERIMENT:

We first scaled the MNIST data to lie in [0.05, 0.95] and then transformed using the logit (inverse logistic sigmoid) of the pixel value in order to use Gaussian spread noise. We use Gaussian spread noise σ = 1 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE. The network g θ (x) contains 8 layers, each layer with 400 units and relu activation function and latent dimension Z = 64. The variational inference network q φ (z|y) = N z µ φ (y), σ 2 φ I Z has a similar structure for the mean network µ φ (y). Learning was done using the Adam optimiser with learning rate 10 -4 and exponential decay rate of 0.96 every 10000 iterations.

E CELEBA EXPERIMENT:

Both encoder and decoder used fully convolutional architectures with 5x5 convolutional filters and used vertical and horizontal strides 2 except the last deconvolution layer we used stride 1. Here Conv k stands for a convolution with k filters, DeConv k for a deconvolution with k filters, BN for the batch normalization Ioffe & Szegedy (2015) , ReLU for the rectified linear units, and FC k for the fully connected layer mapping to R k .

x 

:

For simplicity, we use univariate x, with the extension to the multivariate setting being straightforward.

If F {K} can change sign, by continuity, there must exist a point at which F {K} = 0.

This includes the Laplace, Mellin and Hartley transforms.

We focus on demonstrating how the spread divergences heals critical slowing down, rather than deriving a state-of-the-art approximation of p(z|y). The importance sampling approach has fast run time and works well, even for large latent dimensions, Z =

We also implemented a variational factorised approximation of p(z|y) but found this to be relatively slow and ineffective. A variational Gaussian approximation of p(z|y) improves on the factorised approximation, but is still slow compared to the importance sampling scheme.

Figures and Tables:

Figure1: Left: The lower dotted line denotes Gaussian distributed data p(x) with support only along the linear subspace defined by the origin a and direction A. The upper dotted line denotes Gaussian distributed data q(x) with support different from p(x). Optimally, to maximise the spread divergence between the two distributions, for fixed noise entropy, we should add noise that preferentially spreads out along the directions defined by p and q, as denoted by the ellipses.

number of datapoints N .

Figure2: (a) For X = 20 observations and Z = 10 latent variables, we generate N = 20000 datapoints from the model x = Az, for independent zero mean unit variance Laplace components on z. The elements of A used to generate the data are uniform random ±1. We use S y = 1, S z = 1000 samples and 2000 EM iterations to estimate the mixing matrix. The relative error is averaged over all i, j and 10 random experiments. We also plot standard errors around the mean relative error. In blue we show the error in learning the underlying parameter using the standard EM algorithm. As expected, as γ → 0, the error blows up as the EM algorithm 'freezes'. In orange we plot the error for EM using spread noise, as described in section(5.2.1); no slowing down appears as the model noise γ decreases. As the model noise increases, the quality of the learned model under spread noise decreases gradually. In (b) we show that, apart from very small N , the error for the spread EM algorithm is lower than for the standard EM algorithm. Here Z = 5, X = 10, S = 1000, γ = 0.2, with 500 EM updates used. Results are averaged over 50 runs of randomly drawn A.

Figure 3: Comparison of deep generative models trained on the MNIST digits after 300K iterations. (a) Samples from the trained deterministic model. (b) Means from a standard (noisy) VAE with fixed observation noise. (c) Samples from the standard noisy VAE model.

Figure 4: Comparison of training approaches for the CelebA dataset. All models had the same structure and were trained using the same Adam settings, as in the MNIST experiment.

) over y to give the boundN y x, σ 2 I X log pθ (y) ≥ H(Σ φ ) + E N ( 0,I) [log q z (z = µ φ (y) + C φ )] E N ( 0,I) E N (y x,σ 2 I X ) (y -g θ (µ φ (y) + C φ )) y x,σ 2 I X ) (y -g θ (µ φ (y) + C φ )) 2 = σ 2 -2E N ( x 0,I X ) [ x g θ (µ φ (x + σ x ))] + E N ( x 0,I X ) (x -g θ (µ φ (x + σ x )))

Figure 5: δVAE samples

Figure 7: Noisy VAE samples

∈ R 64×64×3 → Conv 128 → BN → Relu → Conv 256 → BN → Relu → Conv 512 → BN → Relu → Conv 1024 → BN → Relu → FC 64 z ∈ R 64 → FC 8×8×1024 → DeConv 512 → BN → Relu → DeConv 256 → BN → Relu → DeConv 128 → BN → Relu → DeConv 64 → BN → Relu → DeConv 3

Table content: <table/>

Peer review (iclr2019):
 Pros: - interesting idea. Cons: - the paper forgets the state of the art for comparisons (optimal transport, data processing inequalities) - the paper formally shows little as most results are in fact buried in the text and it is hard to tell the formal from the informal. - experiments fall short of really using the setting proposed - the paper focuses too much on keeping the identity of the indiscernibles and forgets the study of other properties (including downsides, such as variance increase) Detail: * The paper claims to propose a "theory" for spread divergences (conditioning a f-divergence by a "third-party" conditional distribution on supports which makes supports match) still keeping the identity of indiscernibles. * The paper recycles the notion of Spread f-divergences from Zhang et al. (which makes a circular reference to this paper for the introduction of these divergences). * The paper motivates the notion of spread divergences by the fact that f-divergences impose matching supports (Section 1), not mentioning that optimal transport theory is a much natural fit for any such kind of setting (e.g. Wasserstein distances). This is a big omission and a missed occasion for a potentially interesting discussion. * The paper then claims that "spread noise makes distributions more similar" (Section 2.1), not mentioning that equation (8), which it claims to have been shown by Zhang et al. paper (see below), is in fact a data processing inequality *long known*. They will find it, along with a huge number of other useful properties, in series of IEEE T. IT papers, among which Pardo and Vajda*s "About distances of discrete distributions satisfying the data processing theorem of information theory", Van Erven and Harremoes, "Re ́nyi Divergence and Kullback-Leibler Divergence" (for the KL / Rényi divergence, but you have more references inside), etc. . * The paper then goes on "showing" (a word used often, even when there is not a single Theorem, Lemma or the like ever stated in the paper...) several properties (Section 2.2). The first states that (9) is equivalent to P being invertible. It is wrong because it is just in fact stating (literally) that P defines an injective mapping. The second states that (11) is equivalent to (12), without the beginning of a proof. I do need to see a proof, and in particular how you "define" an invertible transform "p^-1". * The paper then gives two examples (Sections 3, 4). In Section 3, I am a bit confused because it seems that p and q must have supports in IR, which limits the scope of the example. The same limitation applies to Section 4, even when it is a bit more interesting. In all cases, the authors must properly state a Lemma in each Section that states and shows what is claimed before Section 3. * The paper then makes several experiments. Unless I am mistaken, it seems that Section 5.1 relies on a trick that does not change the support from x to y. Therefore, what is the interest of the approach in this case ? In Section 5.2, isn’t the trick equivalent to considering ICA with a larger .gamma ? * A concern is that the paper says little about the reason why we should pick one p(y|x) instead of another one. The focus is on the identity of indiscernibles. The paper also forgets some potential drawbacks of the technique, including the fact that variance increases — the increase can be important with bad choices, which is certainly not a good thing.