Year: 2018

Abstract:
For distributions p and q with different support, the divergence D(p||q) generally will not exist. We define a spread divergence D̃(p||q) on modified p and q and describe sufficient conditions for the existence of such a divergence. We give examples of using a spread divergence to train implicit generative models, including linear models (Principal Components Analysis and Independent Components Analysis) and non-linear models (Deep Generative Networks).

heading: 1 INTRODUCTION
text: A divergence D(p||q) (see, for example Dragomir (2005)) is a measure of the difference between two distributions p and q with the property
D(p||q) ≥ 0, and D(p||q) = 0 ⇔ p = q (1) Some of our results are specific to the f -divergence, defined as
Df (p||q) = Eq(x) [ f ( p(x)
q(x)
)] (2)
where f(x) is a convex function with f(1) = 0. An important special case of an f -divergence is the well-known Kullback-Leibler divergence KL(p||q) = Ep(x) [ log p(x)q(x) ] which is widely used to train models using maximum likelihood. We are interested in situations in which the supports of the two distributions are different, supp (p) 6= supp (q). In this case the divergence may not be defined. For example, for p(x) being an empirical data distribution on continuous dataset x1, . . . , xN , p(x) = 1 N ∑N n=1 δ (x− xn) where δ (·) is the Dirac Delta function. For a model q(x) with support R, then KL(q||p) is not formally defined. This is a challenge since implicit generative models of the form q(x) = ∫ δ (x− gθ(z)) p(z)dz only have limited support; in this case maximum likelihood to learn the model parameter θ is not available and alternative approaches are required – see Mohamed & Lakshminarayanan (2016) for a recent survey.

heading: 2 SPREAD DIVERGENCES
text: The aim is, from q(x) and p(x) to define new distributions q̃(y) and p̃(y) that have the same support1. Using the notation ∫ x to denote integration ∫ (·) dx for continuous x, and ∑ x∈X for discrete x with domain X , we define a random variable y with the same domain as x and distributions
p̃(y) = ∫ x p(y|x)p(x), q̃(y) = ∫ x p(y|x)q(x) (3)
where p(y|x) is a ‘noise’ process designed to ‘spread’ the mass of p and q such that p̃(y) and q̃(y) have the same support. For example, if we use a Gaussian p(y|x) = N ( y x, σ2 ) , then p̃ and q̃ both have support R. We therefore use noise with the property that, despite D(p||q) not existing, D(p̃||q̃) does exist and we define the Spread Divergence
D̃(p||q) ≡ D(p̃||q̃) (4)
Note that this satisfies the divergence requirement D̃(p||q) ≥ 0. The second requirement, D̃(p||q) = 0⇔ p = q, is guaranteed for certain ‘noise’ processes, as described in section(2.1).
1For simplicity, we use univariate x, with the extension to the multivariate setting being straightforward.
Spread divergences have many potential applications. For example, for a model q(x|θ) with parameter θ and empirical data distribution p(x), maximum likelihood training corresponds to minimising KL(p||q) with respect to θ. However, for implicit models, the divergence KL(p||q) does not exist. However, if a spread divergence exists, provided that the data is distributed according the model p(x) = q(x|θ0) for some unknown parameter θ0, the spread divergence D(p(x)||q(x|θ)) has a minimum at θ = θ0. That is (for identifiable models) we can correctly learn the underlying data generating process, even when the original divergence is not defined.

heading: 2.1 NOISE REQUIREMENTS FOR A SPREAD DIVERGENCE
text: Our main interest is in using noise to define a new divergence in situations in which the original divergence D(p||q) is itself not defined. For discrete variables x ∈ {1, . . . , n}, y ∈ {1, . . . , n}, the noise Pij = p(y = i|x = j) must be a distribution ∑ i Pij = 1, Pij ≥ 0 and∑
j Pijpj = ∑ j Pijqj ∀i ⇒ pj = qj ∀j (5)
which is equivalent to the requirement that the matrix P is invertible, see appendix(B). There is an additional requirement that the spread divergence exists. In the case of f -divergences, the spread divergence exists provided that p̃ and q̃ have the same support. This is guaranteed if∑
j Pijpj > 0, ∑ j Pijqj > 0 ∀i (6)
which is satisfied if Pij>0. In general, therefore, there is a space of noise distributions p(y|x) that define a valid spread divergence. The ‘antifreeze’ method of Furmston & Barber (2009) is a special form of spread noise to define a valid Kullback-Leibler divergence (see also Barber (2012)).
For continuous variables, in order that D̃(p||q) = 0 ⇒ p = q, the noise p(y|x), with dim(Y ) = dim(X) must be a probability density and satisfy∫
p(y|x)p(x)dx = ∫ p(y|x)q(x)dx ∀y ∈ Y ⇒ p(x) = q(x) ∀x ∈ X (7)
This is satisfied if there exists a transform p−1 such that∫ p−1(x′|y)p(y|x)dy = δ (x′ − x) (8)
where δ (·) is the Dirac delta function. As for the discrete case, the spread divergence exists provided that p̃ and q̃ have the same support, which is guaranteed if p(y|x)>0. A well known example of such an invertible integral transform is the Weierstrass Transform p(y|x) = N ( y x, σ2 ) , which has an explicit representation for p−1. In general, however, we can demonstrate the existence of a spread divergence without the need for an explicit representation of p−1. As we will see below, the noise requirements for defining a valid spread divergence such that D(p̃||q̃) = 0⇔ p = q are analogous to the requirements on kernels such that the Maximum Mean Discrepancy MMD(p, q) = 0 ⇔ p = q, see Sriperumbudur et al. (2011) and Sriperumbudur et al. (2012).

heading: 3 STATIONARY SPREAD DIVERGENCES
text: Consider stationary noise p(y|x) = K(y − x) where K(x) is a probability density function with K(x)>0, x ∈ R. In this case p̃ and q̃ are defined as a convolution
p̃(y) = ∫ K(y−x)p(x)dx = (K ∗ p) (y), q̃(y) = ∫ K(y−x)q(x)dx = (K ∗ q) (y) (9)
Since K>0, p̃ and q̃ are guaranteed to have the same support R. A sufficient condition for the existence of the Fourier Transform F {f} of a function f(x) for real x is that f is absolutely integrable. All distributions p(x) are absolutely integrable, so that both F {p} and F {q} are guaranteed to exist. Assuming F {K} exists, we can use the convolution theorem to write
F {p̃} = F {K}F {p} , F {q̃} = F {K}F {q} (10)
Hence, we can write
D(p̃||q̃) = 0⇔ p̃ = q̃ ⇔ F {K}F {p} = F {K}F {q} ⇔ F {p} = F {q} ⇔ p = q (11)
where we used the invertibility of the Fourier transform and assumed that F {K} 6= 0, or equivalently2, F {K}>0. Hence, provided that K(x)>0 and F {K}>0 then K(x) defines a valid spread divergence. Note that other transforms have a corresponding convolution theorem3 and the above derivation holds, with the requirement that the corresponding transform of K(x) is non-zero. As an example of such a noise process, consider Gaussian noise,
K(y − x) = 1√ 2πσ2 e− 1 2σ2 (y−x)2 (12)
leading to a positive Fourier Transform:
F {K} (ω) = 1√ 2πσ2 ∫ ∞ −∞ eiωxe− 1 2σ2 x2dx = e− σ2ω2 2 > 0 (13)
Similarly, for Laplace noise K(x) = 12be − 1b |x|
p(y|x) = K(y − x) = 1 2b e− 1 b |y−x|, F {K} (ω) =
√ 2
π
b−1
b−2 + ω2 > 0 (14)
Since K>0 and F {K}>0, this also defines a valid spread divergence over R.

heading: 3.1 INVERTIBLE MAPPINGS
text: Consider p(y|x) = K(y − f(x)) for strictly monotonic f . Then, using the change of variables
p̃(y) = ∫ K(y − z)pz(z)dz, pz(z) = px(f−1(z)) ( |J ( x = f−1(z) ) | )−1
(15)
where J is the Jacobian of f . For distributions p(x) with bounded domain, for example x ∈ [0, 1], we can use a logit function, f(x) = − log ( x−1 − 1 ) , which maps the interval [0, 1] to R. Us-
ing then, for example, Gaussian spread noise p(y|x) = N ( y f(x), σ2 ) , both p̃(y) and q̃(y) have support R. If D(p̃||q̃) is zero then p = q on the domain [0, 1].

heading: 3.2 MAXIMISING THE SPREAD
text: From the data processing inequality (see appendix(A)), spread noise will always decrease the f - divergence Df (p̃(y)||q̃(y)) ≤ Df (p(x)||q(x)). If we are to use a spread divergence to train a model, there is the danger that adding too much noise may make the spreaded empirical distribution and spreaded model distribution so similar that it becomes difficult to numerically distinguish them, impeding training. In general, therefore, it would be useful to add noise such that we define a valid spread divergence, but can maximally still discern the difference between the two distributions. To gain intuition, we define p and q to generate data in separated linear subspaces, p(x) = ∫ δ (x− a−Az) p(z)dz, q(x) = ∫ δ (x− b−Bz) p(z)dz, p(z) = N (z 0, IZ). Using Gaussian spread, p(y|x) = N (y µ,Σ), what is the optimal µ, Σ that maximises the divergence? Clearly, as Σ tends to zero, the divergence increases to infinity, meaning that we must at least constrain the entropy of p(y|x) to be finite. In this case the spreaded distributions are given by
p̃(y) = N ( y µ+ a,AAT + Σ ) , q̃(y) = N ( y µ+ b, BBT + Σ ) (16)
We define a simple Factor Analysis noise model with Σ = σ2I + uuT, where σ2 is fixed and uTu = 1. The entropy of p(y|x) is then fixed and independent of u. Also, for simplicity, we assume A = B. It is straightforward to show that the spread divergence KL(p̃||q̃) is maximised for u pointing orthogonal to the vector ( AAT + σ2I )−1 (b− a). Then u optimally points along the direction in which the support lies. The support of p(y|x) must be the whole space but to maximise the divergence the noise preferentially spreads along directions defined by p and q, see figure(1).
2If F {K} can change sign, by continuity, there must exist a point at which F {K} = 0. 3This includes the Laplace, Mellin and Hartley transforms.

heading: 4 MERCER SPREAD DIVERGENCE
text: We showed in section(3) how to define one form of spread divergence, with the result that stationary noise distributions must have strictly positive Fourier Transforms. A natural question is whether, for continuous x, there are other easily definable noise distributions that are non-stationary. To examine this question, let x ∈ [a, b], y ∈ [a, b] and K(x, y) = K(y, x) be square integrable, K(x, y) ∈ L2. We define Mercer noise p(y|x) = K(x, y)/K(x), whereK(x) = ∫ K(x, y)dy. For strictly positive definite K, by Mercer’s Theorem, it admits an expansion
K(x, y) = ∑ n λnφn(x)φn(y) (17)
where the eigenfunctions φn form a complete orthogonal set of L2[a, b] and all λn>0, see for example Sriperumbudur et al. (2011). Then
p̃(y) = ∑ n ∫ λnφn(x)φn(y) p(x) K(x) dx, q̃(y) = ∑ n ∫ λnφn(x)φn(y) q(x) K(x) dx (18)
and p̃(y) = q̃(y) is equivalent to the requirement∑ n ∫ λnφn(x)φn(y) p(x) K(x) dx = ∑ n ∫ λnφn(x)φn(y) q(x) K(x) dx (19)
Multiplying both sides by φm(y) and integrating over y we obtain∫ φm(x) p(x)
K(x) dx =
∫ φm(x) q(x)
K(x) dx (20)
If p(x)/K(x) and q(x)/K(x) are in L2[a, b] then, from Mercer’s Theorem, they can be expressed as orthogonal expansions
p(x) K(x) = ∑ n γpnφn(x), q(x) K(x) = ∑ n γqnφn(x) (21)
Then, equation(20) is∫ φm(x) ∑ n γpnφn(x)dx = ∫ φm(x) ∑ n γqnφn(x)dx (22)
which reduces to (using orthonormality), γpm = γ q m ⇒ p = q. Hence, provided K(x, y) = K(y, x) is square integrable on [a, b] and strictly positive definite, then K(x, y)/ ∫ K(x, y)dy defines valid
spread noise. For example,K(x, y) = exp ( −λ1(x2 + y2) ) +exp ( −λ2(x2 + y2) ) defines a strictly positive non-stationary square integrable kernel on [a, b]. Provided p(x)/K(x) and q(x)/K(x) are in L2[a, b] then the spread noise p(y|x) = K(x, y)/K(x) defines a valid spread divergence.

heading: 5 APPLICATIONS
text: We demonstrate using a spread divergence to train implicit models
pθ(x) = ∫ δ (x− gθ(z)) p(z)dz (23)
where θ are the parameters of the encoder g. We show that, despite the likelihood not being defined, we can nevertheless successfully train the models using an EM style algorithm, see for example Barber (2012). We then show how to train a deterministic non-linear generative model using a variational approximation.

heading: 5.1 DETERMINISTIC LINEAR LATENT MODEL
text: For observation noise γ, the Probabilistic PCA model (Tipping & Bishop, 1999) for X-dimensional observations and Z-dimensional latent is
x = Fz+γ , z ∼ N(0, IZ), ∼ N(0, IX), pθ(x) = N ( y 0, FF T + γ2IX ) (24)
When γ = 0, the generative mapping from z to x is deterministic and the model pθ(x) has support only on a subset of RX and the data likelihood is in general not defined. In the following we consider general γ, setting γ to zero at the end of the calculation. To fit the model to iid data {x1, . . . , xN} using maximum likelihood, the only information required from the dataset is the data covariance
Σ̂. The maximum likeihood solution for PPCA is then F = UZ ( ΛZ − γ2IZ ) 1 2 R, where ΛZ , UZ
are the Z largest eigenvalues, eigenvectors of Σ̂; R is an arbitrary orthogonal matrix. Using spread noise p(y|x) = N ( y x, σ2IX ) , the spreaded distribution p̃θ(y) is a Gaussian
p̃θ(y) = N ( y 0, FF T + (γ2 + σ2)IX ) (25)
Thus, p̃θ(y) is of the same form as PPCA, albeit with an inflated covariance matrix. Adding Gaussian spread noise to the data also simply inflates the sample covariance to Σ̂′ = Σ̂ + σ2IX . Since the eigenvalues of Σ̂′ ≡ Σ̂+σ2IX are simply Λ′ = Λ+σ2IX , with unchanged eigenvectors, the optimal deterministic (γ = 0) latent linear model has solution F = UZ ( Λ′Z − σ2IZ ) 1 2 R = UZΛ 1 2
ZR. Unsurprisingly, this is the standard PCA solution; however, the derivation is non-standard since the likelihood of the deterministic latent linear model is not defined. Nevertheless, using the spread divergence, we learn a sensible model and recover the true data generating process if the data were exactly generated according to the deterministic model.

heading: 5.2 DETERMINISTIC INDEPENDENT COMPONENTS ANALYSIS
text: ICA corresponds to the model p(x, z) = p(x|z) ∏ i p(zi) (26)
where the independent components zi follow a non-Gaussian distribution. For Gaussian noise ICA an observation x is assumed to be generated by the process
p(x|z) = ∏ j N ( xj gj(z), γ 2 )
(27)
where gi(z) mixes the independent latent process z. In standard linear ICA, gj(z) = aTj z where aj is the jth column on the mixing matrixA. For small observation noise γ2, the EM algorithm (Bermond & Cardoso, 1999) becomes ineffective. To see this, consider X = Z and invertible mixing matrix A, x = Az. At iteration k the EM algorithm has an estimate Ak of the mixing matrix. The M-step updates Ak to
Ak+1 = E [ xzT ] E [ zzT ]−1
(28)
where, for noiseless data (γ = 0), E [ xzT ] = 1
N ∑ n xn ( A−1k x T n ) = ŜA−Tk , E [ zzT ] = A−1k ŜA −T k (29)
where Ŝ ≡ 1N ∑ n xnx T n is the moment matrix of the data. Thus,Ak+1 = ŜA −T k ( A−1k ŜA −T k )−1 = Ak. and the algorithm ‘freezes’. Similarly, for low noise γ 1 progress critically slows down. Whilst over-relaxation methods, see for example Winther & Petersen (2007) can help in the case of small noise, for zero noise γ = 0, over-relaxation is of no benefit.

heading: 5.2.1 HEALING CRITICAL SLOWING DOWN
text: To deal with small noise and the limiting case of a deterministic model (γ = 0), we consider Gaussian spread noise p(y|x) = N ( y x, σ2IX ) to give
p(y, z) = ∫ p(y|x)p(x, z)dx = ∏ i N ( y gi(z), ( γ2 + σ2 ) IX )∏ i p(zi) (30)
The empirical distribution is replaced by the spreaded empirical distribution
p̂(y) = 1
N ∑ n N ( y xn, σ2IX ) (31)
The M-step has the same form as equation(28) but with modified statistics E [ yzT ] = 1
N ∑ n ∫ N ( y xn, σ2 ) p(z|y)yzTdzdy, (32)
E [ zzT ] = 1
N ∑ n ∫ N ( y xn, σ2 ) p(z|y)zzTdzdy (33)
The E-step optimally sets
p(z|y) = 1 Zq(y) N (z µ(y),Σ) ∏ i p(zi), Zq(y) = ∫ N (z µ(y),Σ) ∏ i p(zi)dz (34)
where Zq(y) is a normaliser and Σ = (γ2 + σ2) ( ATA )−1 , µ(y) = ( ATA )−1 Ay (35)
We can rewrite the expectations required for the E-step of the EM algorithm as E [ yzT ] = 1
N ∑ n ∫ N ( y xn, σ2 ) N (z µ(y),Σ) ∏ i p(zi) Zq(y) yzTdzdy (36)
E [ zzT ] = 1
N ∑ n ∫ N ( y xn, σ2 ) N (z µ(y),Σ) ∏ i p(zi) Zq(y) zzTdzdy (37)
Generally the posterior p(z|y) will be peaked around N (z µ(y),Σ) and writing the expectations with respect to N (z µ(y),Σ) allows for an effective sampling approximation focussed on regions of high probability. We implement this update by drawing Sy samples fromN ( y xn, σ 2I )
and, for each y sample, we draw Sz samples fromN (z µ(y),Σ). This scheme has the advantage over more standard variational approaches, see for example Winther & Petersen (2007), in that we obtain a consistent estimator of the M-step update for A4. We show results for a toy experiment in figure(2), learning the underlying mixing matrix in a deterministic non-square setting. Note that standard algorithms such as FastICA (Hyvärinen, 1999) fail in this setting. The noise value is set to σ = max(0.001, 2.5∗ sqrt(mean(AAT))), for estimated mixing matrix A of the underlying deterministic model xn = Azn, n = 1, . . . , N . The EM algorithm learns a good approximation of the unknown mixing matrix and latent components zn, with no critical slowing down.

heading: 5.3 TRAINING IMPLICIT NON-LINEAR MODELS
text: For a deterministic non-linear implicit model, we set p(z) = N (z 0, I) and parameterise gθ(x) by a deep neural network. The likelihood equation(23) is in general intractable and it is natural to consider a variational approximation (Kingma & Welling, 2013),
log pθ(x) ≥ − ∫ qφ(z|x) (log qφ(z|x) + log (pθ(x|z)p(z))) dz (38)
However, since pθ(x|z) = δ (x− gθ(z)) this bound is not well defined. Instead, we minimise the spread divergence KL(p̃(y)||p̃θ(y)). The approach is a straightforward extension of the standard
4We focus on demonstrating how the spread divergences heals critical slowing down, rather than deriving a state-of-the-art approximation of p(z|y). The importance sampling approach has fast run time and works well, even for large latent dimensions, Z = 50. We also implemented a variational factorised approximation of p(z|y) but found this to be relatively slow and ineffective. A variational Gaussian approximation of p(z|y) improves on the factorised approximation, but is still slow compared to the importance sampling scheme.
variational autoencoder and in appendix(C) we provide details of how to do this, along with higher resolution images of samples from the generative model. We dub this model and associated spread divergence training the ‘δVAE’. As a demonstration, we trained a generative network on the MNIST dataset, see figure(3) and appendix(D). We used Gaussian spread noise σ = 1 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE. The network gθ(x) contains 8 layers, each layer with 400 units and relu activation function and latent dimension Z = 64. We also trained a deep convolutional generative model on the CelebA dataset (Liu et al., 2015), see figure(4) and appendix(E). We pre-process CelebA images by first taking 140x140 centre crops and then resizing to 64x64. Pixel values were then rescaled to lie in [0, 1]. We use Gaussian spread noise σ = 0.5 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE.

heading: 6 SUMMARY
text: We described an approach to defining a divergence, even when two distributions to not have the same support. The method introduces a ‘noise’ variable to ‘spread’ mass from each distribution to cover the same domain. Previous approaches (Furmston & Barber, 2009; Sønderby et al., 2016) can be seen as special cases. We showed that defining divergences this way enables us to train deterministic generative models using standard ‘likelihood’ based approaches. Indeed, for simple models such as Independent Components Analysis, we showed how we can implement a principled learning method based on classical EM training, without the standard difficulty of critical slowing down in the case of small (or zero) observation noise.
Introducing noise means that an additional expectation is required. This can be carried out, in part, exactly, although additional approximations using perturbation theory are possible, similar to Roth et al. (2017). Spread divergences have deep connections to other approaches to define measures of disagreement between distributions. In particular, one can view the spread divergence as the probabilistic analogue of MMD, with conditions required for the existence of the spread divergence closely related to the universality requirement on MMD kernels (Micchelli et al., 2006).
Theoretically, we can learn the underlying true data generating process by the use of any valid spread divergence — for example for fixed Gaussian spread noise. In practice, however, the quality of the learned model can depend on the choice of spread noise. In this work we fixed the spread noise, but showed that if we were to learn the spread noise, it would preferentially spread mass across the manifolds defining the two distributions. In future work, we will investigate learning spread noise to maximally discriminate two distributions, which would involve a minimax model training objective, with an inner maximisation over the spread noise and an outer maximisation over the model parameters. This would bring our work much closer to adversarial training methods (Goodfellow, 2017).

heading: A SPREAD NOISE MAKES DISTRIBUTIONS MORE SIMILAR
text: The data processing inequality for f -divergences (see for example Gerchinovitz et al. (2018)) states that Df (p̃(y)||q̃(y)) ≤ Df (p(x)||q(x)). For completeness, we provide here an elementary proof of this result. We consider the following joint distributions
q(y, x) = p(y|x)q(x), p(y, x) = p(y|x)p(x) (39)
whose marginals are the spreaded distributions
p̃(y) = ∫ x p(y|x)p(x), q̃(y) = ∫ x p(y|x)q(x) (40)
The divergence between the two joint distributions is Df (p(y, x)||q(y, x)) = ∫ x,y q(y, x)f ( p(y|x)p(x) p(y|x)q(x) ) = Df (p(x)||q(x)) (41)
The f -divergence between two marginal distributions is no larger than the f -divergence between the joint (see also Zhang et al. (2018)). To see this, consider
Df (p(u, v)||q(u, v)) = ∫ q(u) ∫ q(v|u)f ( p(u, v)
q(u, v)
) dydu
≥ ∫ q(u)f (∫ q(v|u) p(u, v)
q(v|u)q(u) dv
) du
= ∫ q(u)f ( p(u)
q(u)
) du = Df (p(u)||q(u))
Hence,
Df (p̃(y)||q̃(y)) ≤ Df (p(y, x)||q(y, x)) = Df (p(x)||q(x)) (42)
Intuitively, spreading two distributions increases their overlap, reducing the divergence. When p and q do not have the same support, Df (q(x)||p(x)) can be infinite or not well-defined.
B INJECTIVE LINEAR MAPPINGS
Consider an injective linear mapping T from space V to W . From the rank nullity theorem for finite dimensional spaces,
dim(image(T )) + dim(kernel(T )) = dim(V ) (43)
If T is injective, then dim(kernel(T )) = 0. If dim(V ) = dim(W ) then
dim(image(T )) = dim(W ) (44)
Since image(T ) ⊆ W , it must be that image(T ) = W . Hence, injective linear maps between between two (finite dimensional) spaces of the same dimension are surjective; equivalently, they are invertible.
In the context of spread noise, since the domain of x and y are equal and p̃(y) is defined through a linear transformation of p(x), the requirement in (5) that the mapping is injective is equivalent to the requirement that the mapping is invertible.

heading: C SPREAD DIVERGENCE FOR DETERMINISTIC DEEP GENERATIVE MODELS
text: Instead of minimising the likelihood, we train an implicit generative model by minimising the spread divergence
min θ
KL(p̃(y)||p̃θ(y)) (45)
where
p̃(y) = 1
N N∑ n=1 N ( y xn, σ 2IX )
(46)
and
p̃θ(y) = ∫ p(y|x)pθ(x)dx = ∫ N ( y gθ(z), σ 2IX ) p(z)dz = ∫ pθ(y|z)p(z)dz (47)
According to our general theory,
min θ
KL(p̃(y)||p̃θ(y)) = 0 ⇔ p(x) = pθ(x) (48)
Here
KL(p̃(y)||p̃θ(y)) = 1
N N∑ n=1 ∫ N ( y xn, σ 2IX ) log q̃(y)dy + const. (49)
Typically, the integral over y will be intractable and we resort to an unbiased sampled estimate (though see below for Gaussian q). Neglecting constants, the KL divergence estimator is
1
NS N∑ n=1 S∑ s=1 log q̃(yns ) (50)
where yns is a noisy sample of xn, namely y n s ∼ N ( yns xn, σ 2IX ) . In most cases of interest, with non-linear g, the distribution q̃(y) is intractable. We therefore use the variational lower bound
log p̃θ(y) ≥ ∫ qφ(z|y) (− log qφ(z|y) log (pθ(y | z)p(z))) dz (51)
Parameterising the variational distribution as a Gaussian,
qφ(z|y) = N (z µφ(y),Σφ(y)) (52)
then we can reparameterise and write
log p̃θ(y) ≥ H(Σφ) + EN ( 0,I) [log (pθ(y|z = µφ + Cφ )qz(z = µφ + Cφ ))] (53)
where H is the entropy of a Gaussian with covariance Σφ. For Gaussian spread noise in D dimensions, this is (ignoring constants)
log p̃θ(y) ≥ H(Σφ)+EN ( 0,I)
[ − 1
(2σ2) D/2
(y − gθ (µφ(y) + Cφ ))2 + log qz(z = µφ(y) + Cφ ) ] (54)
where Cφ is the Cholesky decomposition of Σφ.
The overall procedure is therefore a straightforward modification of the standard VAE method Kingma & Welling (2013) in which both the model and data are corrupted by noise:
1. Choose a noise corruption variance σ2.
2. Choose a tractable family for the variational distribution, for example qφ(z|y) = N (z µφ(y),Σφ(y)) and initialise φ.
3. We then sample a noisy version yn for each datapoint (if we’re using S = 1 samples)
4. Draw samples to estimate log p̃θ(yn), equation(54)
5. Do a gradient ascent step in (θ, φ).
6. Go to 3 and repeat until convergence.
We note that for Σφ independent of y, we can partially integrate equation(54) over y to give the bound ∫
N ( y x, σ2IX ) log p̃θ(y) ≥ H(Σφ) + EN ( 0,I) [log qz(z = µφ(y) + Cφ )] (55)
− 1 (2σ2) D/2 EN ( 0,I)
[ EN (y x,σ2IX) [ (y − gθ (µφ(y) + Cφ ))2 ]] (56)
where EN (y x,σ2IX) [ (y − gθ (µφ(y) + Cφ ))2 ] = σ2 − 2EN ( x 0,IX) [ xgθ(µφ(x+ σ x))] + EN ( x 0,IX) [ (x− gθ(µφ (x+ σ x)))2
] (57)
Similar to Roth et al. (2017), in principle, one can form a perturbation approximation of the above to second order in x and express the integral over the spread noise as a form of regularisation; however, in our experiments we found that the above works well – we therefore leave such analysis for future work.

heading: D MNIST EXPERIMENT
text: We first scaled the MNIST data to lie in [0.05, 0.95] and then transformed using the logit (inverse logistic sigmoid) of the pixel value in order to use Gaussian spread noise. We use Gaussian spread noise σ = 1 for the δVAE and observation noise σ = 0.5 for the standard noisy VAE. The network gθ(x) contains 8 layers, each layer with 400 units and relu activation function and latent dimension Z = 64. The variational inference network qφ(z|y) = N ( z µφ(y), σ 2 φIZ ) has a similar structure for the mean network µφ(y). Learning was done using the Adam optimiser with learning rate 10−4 and exponential decay rate of 0.96 every 10000 iterations.

heading: E CELEBA EXPERIMENT
text: Both encoder and decoder used fully convolutional architectures with 5x5 convolutional filters and used vertical and horizontal strides 2 except the last deconvolution layer we used stride 1. Here Convk stands for a convolution with k filters, DeConvk for a deconvolution with k filters, BN for the batch normalization Ioffe & Szegedy (2015), ReLU for the rectified linear units, and FCk for the fully connected layer mapping to Rk.
x ∈ R64×64×3 → Conv128 → BN→ Relu → Conv256 → BN→ Relu → Conv512 → BN→ Relu → Conv1024 → BN→ Relu→ FC64
z ∈ R64 → FC8×8×1024 → DeConv512 → BN→ Relu → DeConv256 → BN→ Relu → DeConv128 → BN→ Relu → DeConv64 → BN→ Relu→ DeConv3

Review summaries:
This paper introduces spread divergences. Spread divergences are obtained by taking the divergence between smoothed/noisy versions of two distributions. A spread divergence between two distributions of non-overlapping support can be defined even when the corresponding divergence is not. The authors discuss conditions under which the data generating process can be identified by minimizing spread divergences and apply spread divergences to the examples of PCA, ICA, and noiseless VAE.