SPREAD  DIVERGENCE
Authors: Mingtian Zhang
David Barber
Thomas Bird
Peter Hayes
Raza Habib
Venue:  (2020)
Keywords: divergence minimization
generative model
variational inference

Abstract: For distributions $p$ and $q$ with different supports, the divergence $\div{p}{q}$ may not exist. We define a spread divergence $\sdiv{p}{q}$ on modified $p$ and $q$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a spread divergence to train and improve implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).

Under review as a conference paper at ICLR 2020
SPREAD DIVERGENCE
Anonymous authors Paper under double-blind review
ABSTRACT
For distributions p and q with different supports, the divergence D(p [g) may not exist. We define spread divergence D(pl(g) on modified p and and describe sufficient conditions for the existence of such a divergence. We demonstrate how maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using spread divergence to train and improve implicit generative models, including linear models (Independent Components Analysis) and non-linen oon-iinear models (Deep Generative Networks).
INTRODUCTION
divergence D(pllg) (see, for example Dragomir (2005) a measure of the difference between two distributions p and q with the property
D(plla)
and D(pllq)
p=
(1)
We are interested in situations in which the supports of the two distributions are different, supp (p) + supp (g). An important class is the f-divergence, defined as
(o())) g(x))
Dy(pllg) Exg(=)
where f(z) is convex function with f(1) 0. A special case of an a f-divergence is the well-known Kellback-Leiblee divergence KL(pllq) Ep(r) llog p(z)
q(*) By setting p(z) to the empirical data
distribution, maximum likelihood training of a model ((x) corresponds to minimising KL(pllg).
However, this divergence may not be defined since the ratio p(x)//(2)) can cause a division by zero.
This is a challenge since popular implicit generative models (Mohamed & Lakshminarayanan (2016)
of the form q(x) = f (x (=)) p(2)dz only have limited support. In this case, maximum likelihood to learn the model parameters is not available and alternative approaches to measure the difference between distributions such as Maximum Mean Discrepancy (Gretton al. (2012) or
Wasserstein distance (Peyré et al. (2019) are required.
2 SPREAD DIVERGENCE
From g(x) and p(x) we define new distributions 7(v) and p(y) that have the same suppor Using the notation f, to denote integration (-) dx for continuous 2, and Evex for discrete x with domain X, we define random variable y with the same domain as x and distributions
p(v)= p(yl))p(a)) a(v) (()))))(())
(3)
where p(y(ær) is "noise" designed to 'spread" the mass of p and such that ply) and 7(v) have the same support. For example, if we use a Gaussian p(ylæ) A (y x, o²), then ş and q both have support R. We also impose an additional requirement on the noise p(y|x), namely that D(pllg) = 0 + p p= q. As show in section 2.1) this is guaranteed for certain 'noise' distributions. Given these requirements, can define the Spread Divergence D(pllg) D(āll7). This satisfies the divergence requirement D(pllg) 2 0 and D(pllg) 0 + p= q.
For simplicity, we univariate v, with the extension the multivariate setting being straightforward.

---

Under review as a conference paper at ICLR 2020
For example, given two delta distributions (o())) 8(2 Mu), P1 (z) 5(2 p1). the KL divergence (or f-divergence) between them is not defined. However, the spread KL divergence (or f-divergence) is defined. Assume a Gaussian noise distribution p(y(z) N (yl (0122 o²) where o² 0.5, the "spreaded" delta distributions have the form: po(y) = J. 8(x - No) (v/x, = N (yluo, o²), (y()) = ʃ2 6(x W1)N (ylæ, o²) = N (ulu), Therefore, the spread KL divergence (with Gaussian noise) between two delta distributions is equivalent to the KL divergence between two Gaussian distributions with the same variance, which has closed form (see appendix (D) for a derivation):
(=(pl(=))ll (z)) = (p (v) |0 (v)) 0| 11ll
(4)
It's worth noticing that in the case of two delta distributions, the spread KI. divergence is equal the squared 2- S-Wasserstein distance (see Peyré et al. (2019); [eebbric|| (1900))
2.1 NOISE REQUIREMENTS FOR A SPREAD DIVERGENCE
Our main interest is in using noise to define a new divergence in situations in which the original divergence (ppl 44 is itself not defined. For discrete variables E {l...., n), y E {I,..., n). the
noise P. = pyy = i| j) must be a distribution Σ, Piy = 1, P.j 2 0 and
Σ PopPy = > Pigq
Vi
Pj 9
Vj
(5)
which is equivalent to the requirement that the matrix P is invertible. There is additional requirement that the spread divergence exists. In the case of f-divergences, the spread divergence exists provided that ğ and q have the same support. This is guaranteed if
Σ Poppy >0 O,
P > 0 Vi
(6)
which is satisfied if Pty>0. O. In general, therefore, there a space of noise distributions p(yl#) that define valid spread divergence. For example, the 'antifreeze' method of Furmston & Barber (2009)
a special form of spread noise to define a valid KL divergence (see also Barber (2012)).
For continuous variables, in order for D(pllq) 0 > p = q: the noise p(y]æ), with dim(Y) dim(X) must be probability density and satisfy
/ p(ul=)p(z)dr= = f (())
Ay E
+ p(x) = g(x)
Vz E X
In the following section we discuss the special case of stationary noise for continuous systems.
STATIONARY SPREAD DIVERGENCES
Consider stationary noise p(y]æ)) = Kạy x) where K(x) is probability density function with K(x)>0,, x E R. In this case ` and ; are defined as a convolution
p(y) = / ((y--a)ai x))(x) da = (K p) (v), a(v) = f K -(y--)g(a))) x)((x)dx (K* +g) (v)
p(y)=
(8)
Since K>0, p and q are guaranteed to have the same support R. A sufficient condition for the existence of the Fourier Transform. FF {ʃ} of a function f(x) for real x is that f is absolutely integrable. Since all distributions p(x) are absolutely integrable, both F (p) and F (9) are guaranteed to exist. Assuming F. {K} exists, we can then use the convolution theorem to write
F F = F (K) (p),
F F = F (K) F {q)
(9)
Let F(K) or = on at most a countable set. Then
= [(K){()
+ F(p)
=(p))=(4))
(10)
The proof is given in appendix A
Using this write
D(p
0
(11)
* # (K)
(12)
* F (p) = (g) -ア=リ
(13)
2

---

Under review as a conference paper at ICLR 2020
where used the invertibility of the Fourier transform. Hence, for stationary noise p(y[z) K I). we can define a valid spread divergence provided (i) (x) ) is a probability density function and (ii }(K} # 0 or } = 0 on at most countable set. Interestingly, the sufficient conditions for defining a valid spread divergence such that D = 0 + p = are analogous to the characteristic condition kernels such that the Maximum Mean Discrepancy MMD(p. Q) 0 p q. see Sriperumbudur et al. (2011 2012); Gretton et al. (2012). As an example of such noise process, consider Gaussian noise,
(x)) 2TT2
F(K)(w)
jun E 2012 dr = E -- > (14)
V2T02
Similarly, for Laplace noise
1 Hal
2 b- [(K]
(xx)
>0
(15)
2b
π b w
Since in both cases K>0 and F [(K} >0, Gaussian and Laplace noise can be used to define valid spread divergence.
4 MAXIMISING DISCRIMINATORY POWER
From the data processing inequality (see appendix B adding spread noise will always decrease the f-divergence (f(()) ([())) S Dy (p(()))l(())) Intuitively, spreading out distributions makes them more similar. If we are to use spread divergence to train model using maximum likelihood (see section s)c there is the danger that adding too much noise may make the spreaded empirical distribution and spreaded model distribution similar that it becomes difficult to numerically distinguish them, impeding training. discerns the difference between the two distributions max, D(p(v) [l)( for spread noise Pv (vl==)
useful therefore to define spread noise that maximally
parameterised by vb. In general we need to constrain the spread noise to ensure that the divergence remains bounded.
We discuss below two complementary approaches to adjust p(ylær during training. The first approach adjusts the dimension- wise correlations (this corresponds adjusting the covariance structure for Gaussian p(y) (z)) and the second forms a mean transformation. In principle, both approaches can be combined and easily generalized other noise distributions, such as Laplace noise.
4.1 LEARNING COVARIANCE STRUCTURE
I Learning the covariance adjusts the shape of noise centered around the original model manifold. When we maximize the divergence between two spreaded distributions max, D(p(v)ll((x)) the learned noise will discourage overlap between the two distributions. Hence, if the data p and model g lie on the same manifold, the noise will be orthogonal to the manifold.
In learning the Gaussian spread distribution p(y|z) . (vl x E), the number of parameters in the covariance matrix Σ scales quadratically with the data dimension D. We thus define Σ. = o² T + LLT where x² > 0 is fixed (to ensure bounded spread divergence) and L is a learnable D R matrix with R < D. Calculating the log likelihood and sampling then be performed efficiently using standard Woodberry identities, see appendix
4.2 LEARNING THE MEAN TRANSFORM
Consider p(yla)) K(y f()) for injective and stationary K. Then, we define
p(y) - / K(y - (z))
(16)
Note that this is valid spread divergence since, using change of variables,
p(v) / K(v ())=2, ==()==)((~(\))]]00==>=>=>
(17)
Since the codomain of f is determined by its range, injective indicates invertible in this case.
3

---

Under review as a conference paper at ICLR 2020
where J is the absolute Jacobian of f. Hence, D(pyll =0 p= + p==9-- Each injective fo gives a different noise p(y(z), we can thus search for the best noise implicitly by learning fø-
In our experiments we use the invertible residual network Behrmann et al. (2018) fv IR RD with l = (/z ..... o f denotes a ResNet with blocks 1/ = I(.) + Dve (-). Then lo is invertible the Lipschitz-constant, Lip(ge, <<1 for all E (I....,T). Note that when using the spread divergence for training (see section 5.2.2 we only need samples from p(y) which can be obtained from equation by first sampling x from Pz (x) and then from p(y|z) = K(y fx))); this does not require computing the Jacobian or inverse fg
SPREAD MAXIMUM LIKELIHOOD ESTIMATION
Minimising the forward KL divergence between the empirical data distribution p(x)) and model pa (ar) is equivalent to Maximum Likelihood Estimation (MLE) of the parameters 0 of the model. Minimising instead the forward spread KL. divergence, (L.((<))]) (ox))) = En n= log po (v.) + const., where Un are sampled i.i.d from p(y) J. p(y ()]==)((m)))) results in new type of estimation, namely spread MLE. In what follows, we will discuss the statistical properties of spread MLE and demonstrate how it enables the training of models where maximum likelihood is not suited.
5.1 STATISTICAL PROPERTIES
Maximum likelihood is cherished criterion because it exhibits many favourable statistical properties, mainly consistency (convergence to the true parameters in the large data limit) and asymptotic efficiency (achieves the Cramér-Rao Lower Bound, which a lower bound on the variance of any unbiased estimators)- see Casella & Berger (2002) for introduction. A key desideratum for spread MLE is to analyse how these properties are affected. In appendix(E) we demonstrate that spread MLE (for certain family of spread noise) needs weaker sufficient conditions than MLE for both consistency and asymptotic efficiency. Furthermore, a sufficient condition for the existence of MLE is that the likelihood function is continuous over compact parameter space O. We provide example in appendix E.I) where the compactness requirement is violated, but spread MLE is still well defined.
5.2 APPLICATIONS
As an application to show the effectiveness of spread MLE, we use it to train implicit models
po(x)) = / (x: (o(o=)()))))))))) p(==))((=)d=
(18)
where / are the parameters of the encoder g. We show that, despite the likelihood not being defined (see also section(K) for simple linear model example), we can nevertheless successfully train such models using modified EM/variational algorithms (Barber (2012))
5.2.1 TRAINING IMPLICIT LINEAR MODELS: DETERMINISTIC ICA
ICA (Independent Components Analysis) corresponds to the model p(a, 2) p(x|2) II. p(z.). where the independent components z; follow non-Gaussian distribution. For Gaussian noise ICA an observation & assumed be generated by the process p(xlz) = II, (xn]( (g)(=), 7²) where g: mixes the independent latent process z. In linear ICA, (z) = a. where aj is the jth column on the mixing matrix A. For small observation noise 72, it is well known that the maximum likelihood EM algorithm to learn from observed data is ineffective (Bermond & Cardoso 1999; Winther & Petersen 2007). To see this, consider X Z (where X and Z are the dimension of the data and latents respectively) and invertible A, x = Az. At iteration i the EM algorithm has an estimate A of the mixing matrix. The M-step updates A to
A = E
rz
E
22
(19)
where, for zero observation noise (7 = 0),
E x2
N Σ Fn (A^1zT)) -.))) SA.
E
122
A
SAA
(20)
4

---

Under review as a conference paper at ICLR 2020
BM
FMM
sproaa 234
spesad EM
35-
30
0.5
23
20
0.3
4
011
0.
0.3 0.4
230
503 800
1000
(a) Error versus observation noise Ty.
(b) Error versus number of training points.
Figure 1: Relative error | Afrue Atuue versus observation noise (a) and number of training
points (b). (a) For X =20 observations and 2=10 latent variables, we generate N datapoints from the model A=Az, for independent zero mean unit variance Laplace components on z. The elements of A used generate the data are uniform random =1. We use S, -, S : 1000 samples and 2000 EM iterations to estimate A. The error is averaged over all i, and 10 experiments. We also plot standard errors around the mean relative error. In blue we show the error learning the underlying parameter using the standard EM algorithm. As expected, as 7-0, the error blows up as the EM algorithm 'freezes'. In orange we plot the error for EM using spread noise; no slowing down appears as the observation noise y decreases. In (b), apart from very small N. the error for the spread EM algorithm is lower than for the standard EM algorithm. Here Z=5, X=10, S,=1, 5001000, 7 0.2, with 500 EM updates used. Results averaged over 50 runs of randomly drawn A.
and S= = En En I is the moment matrix the data. Thus, Ak+1 = SA T (A and the algorithm 'freezes'. Similarly, for low noise 7 progress critically slows down.
Ak
To deal with small noise and the limiting case of deterministic model (- = 0), we consider Gaussian spread noise ply) x) = N (y x, x²Ix) to give
p(u.2) = / p(yl=)) 2) = II N (us (z))), (72 + 0²) Ix I I(4))
(21)
Using spread noise, the empirical distribution replaced by the spreaded empirical distribution p(y) N Σn N (ylæ", x²Ix) The M-step has the same form as equation 19 but with modified statistics
Σ / N (ul=",0²))) -z) p(2lu)yz Jddyy,
E
y2
N n
N Σ / N (ylæ", :0l p(zlu)ze dady.
E
22
(22)
The E-step optimally sets
p(=ly)= = (v(x) N (=lm()) Σ) II p(z)),
Zu(1) = [[ p(2))dz
p((()d
(23)
where (v) normaliser and
Σ - (72 + 0²) (ATA)-
p(y) = (ATA) Ay.
(24)
Since the posterior p(zly) peaks around N (=|p(y), E), we rewrite equation 22
N EEN(Nle ( Σ) (f
Il,((2)) Z₂(2)
E
yz dady
and similarly for E [22T]. Writing the expectations with respect to N (2 p(y), Σ) allows for a simple but effective importance sampling approximation focused on regions of high probability.
5

---

Under review as a conference paper at ICLR 2020
3
2
7
4
8
6
4
/ 9
0.0
(a) Fixed Laplace
10
0
1
7
7
8
6
4 4
9
(b) Fixed Gaussian
032
3
0
7 4
8
4 9 6
081
15
(c) Learned Gaussian
(d) Covariance
Figure 2: Samples from a generative model (deterministic output) trained using 8-VAE with (a) fixed Laplace covariance, (b) fixed Gaussian covariance and (c) learned Gaussian covariance. We first train with one epoch standard VAE as initialization to all models, and keep latent code z ~ N (=10, Iz) fixed when sampling from these models, so we can more easily compare the sample quality. Figure (d) visualizes the absolute mean of the leading 20 eigenvectors of the learned covariance.
We implement this update by drawing Sy samples from '' (y In, T² and, for each y sample, we draw S₂ samples from N (z ply), Σ) This scheme has the advantage over more standard variational approaches, see for example Winther & Petersen (2007). in that we obtain consistent estimator the M-step update for We show results for a toy experiment figure TT. learning the underlying mixing matrix in deterministic non-square setting. Note that standard algorithms such as FastICA (Hyvärinen 1999) fail this setting. The spread noise is set to = max(0.00) (0.001, 2.5 sqrt(mean( (rrean(AAA))) This modified EM algorithm thus learns a good approximation of the underlying A, with critical slowing down. Other applications of EM that suffer from this slow down phenomenon, such as MLE for Slow Feature Analysis (Turner & Sahani (2007)), or other probabilistic matrix factorization algorithms (Barber (2012)) also benefit from spread divergence.
5.2.2 TRAINING IMPLICIT NON-LINEAR MODELS: 8- V-VAE
A standard way train a deep generative model po f po (2r p) (=)2d2 is maximum likelihood (minimizing ((p(x)][po(() is common to use the variational lower bound (see (Kingma & Welling 2013)). However, for deterministic observation model pa (=lz)= (1- (z))) and ZxXX, this generative model describes only a low dimensional manifold in the data space and the divergence D(p(z)) po(x)) not well defined. Additionally the above bound is not well defined (due to log of a delta function) and the variational EM approach fails, as the deterministic ICA setting. minimize the spread divergence (l)(()))loo))))) For Gaussian noise with fixed diagonal noise p(y|x))N (ylær, o² Ix),
The likelihood equation 18 is in general intractable and
address this, we instead
we can write p(y) + > N (ylxm, x²1x) and NN
(v( = p(y)=))o(x)dd= / (vlgo(2), (2 p(z)d= po(vl=)p(z)de. (25)
We then minimize the divergence
(L(T())lbo(U)) = f p(y) log (y + const.
(26)
Typically, the integral over y intractable, in which case we resort to a sampling estimation.
Neglecting constants, the divergence estimator is 1 EN E log pa (o""), where y is spread noise sample from p(yn (z,); for example yr ~ 'N' (y" xn, o² For non-linear g. the distribution (y) usually intractable and we therefore use the variational lower bound
NS
log (v( 2
(o (- log go + log (p<(vl=)p(2))) dz.
(27)
The approach is straightforward extension of the standard variational autoencoder and in ap- pendix(G) we also derive a lower variance objective and detail how to learn the spread noise (also see appendix(F)) We dub this model and associated spread divergence training the 'A-A

---

Under review as a conference paper at ICLR 2020
(a) 5 Fixed spread noise
(b) 8 Learned spread noise
Figure 3: Samples from a generative model with deterministic output trained using ò- VAE with (a) fixed and (b) learned spread with injective function. We use similar sampling strategy in the MNIST experiment to facilitate sample comparison between the different models see section
Encuder Decoder Models
FID
GAN Models
Table 1: CelebA FID Scores. The 5-VAE re-
VAE
63.0
WGAN GP
30.0
sults are the average over independent mea- surements. The scores of GAN-based mod- els are based on a large-scale hyperparameter search and take the best FID obtained Lucic
6-VAE with fixed spread
52.7
BEGAN
38.9
6-VAE with learned spread
46.5
WGAN
41.3
DRAGAN
42.3
WAE-MMD WAE-GAN
55.0
LSGAN
$3.9
42.0
NS GAN
55.0
et al. (2018). The results of VAE and WAE- based model are from Tolstikhin et al. 2017).
MM GAN
65.6
MNIST Experiment: We trained a 6- VAE on MNIST (LeCun et al. (2010) with (i) fixed Laplace spread noise, equation 15] (ii) fixed Gaussian spread noise, equation 14 and (iii) Gaussian noise with learned covariance, [ection(4.] with rank R = 20; see appendix H) for details. Figures a..b.c show samples from po (a=) for these models; MNIST sufficiently easy that it hard to distinguish between the quality of the fixed and learned noise samples. However, qualitatively, the sharpness of the Laplace spread noise trained model is higher than for the Gaussian noise and motivates that the spread noise can affect the quality of the learned model. We speculate that Laplace noise improves image sharpness since the noise focuses attention on discriminating between points close to the data manifold (since the Laplace distribution is leptokurtic and has higher probability of generating points close to the data manifold than the Gaussian distribution). Figure dd) visualizes the Gaussian learned covariance and shows that the learned noise is largely orthogonal the data manifold.
CelebA Experiment: We trained a 8- 5-VAE on the CelebA dataset (Liu et al. 2015) with (i) fixed and (ii) learned spread with injective function, see appendix(]]. We compared to results from standard VAE with fixed Gaussian noise p(x)=) =) = A (ar (z()), 0.5lx) x Tolstikhin et al. (017)) For (i) the fixed spread divergence uses Gaussian noise N (ylær, x For (ii) use Gaussian noise with learned injective function ResNet (-( = I(-) + (-); see appendix[ for more details. Figure shows samples from 8- VAE trained using Gaussian spread divergence with both fixed and learned spread noise (with go (z) initialised to the fixed-noise setting). It is notable how the 'sharpness' of the image samples substantially increases when learning the spread noise. Table [ shows FIDD (Heusel et al. (2017)) score comparisons between different algorithm: The 5-VAE significantly improves on the standard VAE result; d- V-V with injective function learning also improves on the fixed-noise V-VAE. Indeed the injective 5-VAE results are comparable to popular GAN and WAE models (Gulrajani et al. (2017); Berthelot et al. 2017); Arjovsky et al. (0177) Kodali et al. (2017 7); Mao et al. (2017) ; Fedus et al. (2017); 7): Tolstikhin et al. (2017). Whilst the 8- VAE results are not fully state-of-the-art, we believe it is the first time that implicit models have been trained using a principled maximum likelihood based approach. Our expectation is that by increasing the complexity of the generative model ge and injective function fv. or using different noise such as Laplace distribution, the results will become competitive with state-of-the-art GAN models.
'FIDD scores were computed using cow.b...oom/iiiiooo//. fuinnnfjjjkuuuuuuu based based 10000 samples.

---

Under review as a conference paper at ICLR 2020
6
RELATED WORK
MMD versus spread f-divergence: In spite of the conditions required for defining the spread divergence being closely related the kernel requirement of MMD (Gretton al. 2012), also show that MMD and spread Total Variation distance" can be written as different norms (L2, L respectively) of a common objective (see appendix(().
Instance noise: The instance noise trick to stabilize GAN training Roth et al. (2017); Sonderby al. (2016) is a special case of spread divergence using fixed Gaussian noise. Whilst other similar tricks (for example Furmston & Barber (2009) have been proposed previously, we believe that it is important to state the general utility of the spread noise approach.
The Wasserstein auto-encoder Tolstikhin et al. (201)) is another implicit generative model that uses an mucoder-decoder architecture. The difference that 8- 6-VAE is based on
5-VAB versus WAE:
KL divergence which is corresponding to MLE but WAE uses the Wasserstein distance.
6- VAE versus denoising VAT: The Denoising VAE Im al. (0017) uses VAE with noise added to the data only. In contrast, the 5-VA GAV adds noise to both the data and model. Since the denoising VAE model only adds noise to the model, it cannot recover the true data distribution.
MMD GAN with kernel learning: The idea of learning kernel to increase discrimination is also used MMD GAN (Li et al. 2017). Similar to ours, the kernel in MMD GAN constructed by k = fofo where & a fixed kernel and f is a neural network. To ensure Moof, (p. q) = 0 + p q. this requires f to be injective (Gretton al. (2012)) However, in the MMD GAN framework, (I) usually maps x to a lower dimension. This is crucial for MMD because the amount of data required to produce reliable estimator grows with the data dimension (Ramdas et al.) (2015) and the computation cost of MMD scales quadratically with the amount of data. Whilst using a lower- dimensional mapping makes MMD more practical it also makes it difficult to construct an injective function f. For this reason, heuristics such as the auto-ena untt--encodrr regularizer (Li et al. (2017) are considered. In contrast, for the 8- VAE, the computational cost of estimating the divergence is linear in the number of datapoints. For this reason there is no need for f to be a lower-dimensional mapping; guaranteeing that f is injective therefore relatively straightforward for the 5-VAF.
Flow-based generative models: Invertible flow-based functions (Rezende & Mohamed (0155) have been used to boost the representation power of generative models. Note our use of injective functions is quite distinct from the use of flow-based functions to boost generative model capacity. In our case, the injective function ʃ does not change the model it only changes the divergence. For this reason, the spread divergence doesn't 't require the log determinant of the Jacobian (which is required in Rezende & Mohamed| (2015): Behhmann et al. (2018)) meaning that more general invertible functions can be used boost the discriminatory power of spread divergence.
7 SUMMARY
We described how to define divergence even when two distributions do not have the same support. Previous approaches (Furmston & Barber 2009, Sonderby et al.) (2016) can be seen as special cases. We showed that defining divergences this way enables us train deterministic generative models using standard likelihood based approaches. In principle, we can learn the underlying true data generating process by the use of any valid spread divergence. In practice, however, the quality of the learned model can depend strongly on the choice of spread noise. We therefore investigated learning spread noise maximally discriminate two distributions. We found the resulting training approach stable and that it can significantly improve the image generation results. Whilst trate-of-the-art image generation is not the focus of this work, we obtained promising results. We also discussed the conditions under which spread MLE is consistent and asymptotically efficient, some of which are weaker than the equivalent MLE conditions. Perhaps the most appealing aspect of the spread noise is that is enables one to re-use standard machine learning approaches in statistics such as maximum likelihood to train models that would be otherwise unsuited to standard statistical training approaches.
"Total Variation distance between p(x) and (zx) is defined (up a constant scale) as TV(p(+)lla(z)) (p(z) ((x))dz, it belongs to f-divergence family (see Liese & Vajdal (006)).
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv: 2017.
D. Barber. Bayesian Reasoning and Machine Learning. Cambridge University Press, New York, NY, USA, 2012. ISBN 0521518144 1844,, 9780521518147.
J. Behrmann, D. Duvenaud, and J. Jacobsen. Invertible residual networks. 18I 2018.
arXiv preprint
O. Bermond and J. Cardoso. Approximate likelihood for noisy mixtures. In Proc. ICA '99, Pp. 325-330, 1999.
D. Berthelot, T. Schumm, and L. Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprini arXiv: 1703:/0770007 2017.
G. Casella and R. Berger. Statistical inference, volume 2. Duxbury Pacific Grove, CA, 2002.
H. Cramér. Mathematical methods of statistics, volume 9, Princeton university press, 1999.
S. Dragomir. Some general divergence measures for probability distributions. Acta Mathematica Hungarica, 109(4))331-455 Nov 2005. ISSN 1588-2632. doi: 10.10077/110444--055-001111-
W. Fedus, M. Rosca, B. Lakshminarayanan, A. Dai, S. Mohamed, and I. Goodfellow. Many paths to equilibrium: Gans do not need to decrease divergence every step. arXiv preprint 1rXi::1710.08446, 2017.
T. Ferguson. An inconsistent maximum likelihood estimate. Journal of the American Statistical Association, 77(380):831883) 1982.
T. Furmston and D. Barber. Solving deterministic policy (PO)MPDs using Expectation-axximssttioo and Antifreeze. First international workshop on learning and data mining for robotics (LEMIR), pp. 56-70, 2009. In conjunction with ECMI//KKD-220
M. Gelbrich. On a formula for the L2 Wasserstein metric between measures on euclidean and hilbert spaces. Mathematische Nachrichten, 147(1):185-22) 1990.
Gerchinovitz. P. Ménard, and Stoltz. Fano's inequality for random variables. arXiv, 2018. doi: 702.5:170..059885V..
A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, (3MM2)):733-773 2012.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of Wasser- stein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium, In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
A. Hyvärinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks, 10(3):626-634, May 1999. ISSN 1045-9227. doi: 109/72. 761722.
D. Im, S. Ahn, R. Memisevic, and Y. Bengio. Denoising criterion for variational auto-encoding framework. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
D. Kingma and Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
D. Kingma and M. Welling. Auto-Encoding Variational Bayes. arXiv:/312.611 [stat.MI), 2013.

---

Under review as a conference paper at ICLR 2020
N. Kodali, J. Abemeety, J. Hays, and Z. Kira. On convergence and stability of gans. arXiv preprint arXiv:170S.07215. 2017.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010.
E. Lehmann. Elements of large-sample theory. Springer Science & Business Media, 2004.
C. Li, W. Chang, Y. Cheng, Y. Yang, and B. Póczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances Neural Information Processing Systems, pp. 2203-2213, 2017.
F. Liese and I. Vajda. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10))4394 2006.
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systemx, pp. 700-709, 2018.
X. Mao, Q. Li, H. Xie, R. Lau, Z. Wang, and Paul Least squares generative adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2794-2802, 2017.
Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 1eX2.v:80000055: 2018.
S. Mohamed and B. Lakshminarayanan. Learning in implicit generative models. arXiv preprint, 2016. doi: arXiv: 111.v1610.00343..
B. Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
G. Peyré, M. Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 1(5-6)3355-60) 2019.
A. Ramdas, S. Reddi, B. Póczos, A. Singh, and L. Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
J. Rezende and S. Mohamed. Variational inference with normalizing flows. arXiv preprint 1505.00705 2015.
K. Roth, A. Lucchi, S. Nowozin, and T. Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances neural information processing systems, pp. 2018-2028, 2017.
N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 14(7):1233-1788, 2002.
C. Sønderby, J. Caballero, L. Theis, W. Shi, and F. Huszár. Amortised map inference for image super-resolution. arXiv preprint arXiv: 00000000044 2016.
B. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. Lanckriet. Hilbert space em- beddings and metrics on probability measures. Journal of Machine Learning Research, 11(Apr): 1517-1561, 2010.
B. Sriperumbudur, K. Fukumizu, and G. Lanckriet. Universality, Characteristie Kernels and RKHS Embedding of Measures. J. Mach. Learn. Res., 22:2389-2410, July 2011. ISSN 1532-4435.
B. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. Lanckriet. On the Empirical Estimation of Integral Probability Metrics. Electronic Journal of Statistics, 6:1550-1599, 2012.
M. Tipping and C. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B, 21/3:61 1-622, January 1999.
10

---

Under review as a conference paper at ICLR 2020
1. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558, 2017.
R. Turner and M. Sahani. A maximum-iikelihodd interpretation for slow feature analysis. Neural computation, 19(4):0022-138,, 2007.
A. Wald. Note on the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):595-66) 601, 1949.
O. Winther and K. Petersen. Bayesian independent component analysis: Variational methods and non-negative decompositions. Digital Signal Processing, 17(5):858 872, 2007. ISSN 1051-2004. Special Issue on Bayesian Source Separation.
M. Zhang, T. Bird, R. Habib, T. Xu, and D. Barber. Variational f-divergence minimization. arXiv preprint arXiv: 11899, 2018.

---

Under review as a conference paper at ICLR 2020
A PROOF OF THEOREM A
Theorem 1. Consider distributions p. q. and L' integrable function K and Fourier Transform F {K}
F(K),
Let F(K) # or }(K} at most a countable set. Then
F())
(28)
Proof. When F(K) + O,
F(q} is trivial. We first show
that the Fourier transform of an L' function is continuous on IR., When F(K} 0 on at most countable set, we then show that F(4} and F(p} cannot be different at set of countable points.
Since any distribution q is in I,", we can write
lim Fla)(w+ - Fla)(w)) = lim / ((x)) (e (e-2xrie(utt))) - 22view) dx -0
-
lim / |((x))|e 2Tize - 1/dz
2rize lim |a(x)) -0
2 - 1| dæ (Dominated Convergence Theorem)
So F(g) is (uniformly) continuous. The same argument applies to show that F(p) is uniformly continuous.
Since F(K} = O on at most countable set C, we assume there is point wo C where F ((g)(wo) + FlP) (wr). Without loss of generality, we assume (wo) FFp)
wo-th that not in C. F[K (wo +h) * 0 and it follows therefore that = 0. For points implics
+ h) - (wo + f)) = 0. By continuity of F(p) and F(g). we have (wo)
h) Flp h) + 0 when h + O, which leads to contradiction (0 cannot be zero).
B SPREAD NOISE MAKES DISTRIBUTIONS MORE SIMILAR
The data processing inequality for f-divergences (see for example Gerchinovitz et al. (2018) states that D, (P())ll(v)) S Dy ()(p())]lg(z)) For completeness, provide here elementary proof of this result. We consider the following joint distributions
q(y, x)) (()),
p(y = p(ul))p((p))pa))
(29)
whose marginals are the spreaded distributions
p(y) = / pvyl x ())(x),
7(v) = / p(y)x)g(()
(30)
The divergence between the two joint distributions
[ glv. x()
p(ul=)p(x) p(ul))q(=))
Dj (p(x)))((()))
(31)
The fddivergence between two marginal distributions is no larger than the f-divergence between the joint (see also Zhang et al. (2018). To see this, consider
q(u)) f a(ol))s (P(u)e)) dydu
Dy(p(u, e) lll u))
g(u,
g(w)) (ʃ (fatola) ((vl))q(u)
p(u,v)
du du du (y((w()l((())) (g(u)
/ q())s (p(U)
Hence,
D, (((()))))) < (y(( 1)444 æ)) = ()(p())(((())
(32)
Intuitively, spreading two distributions increases their overlap, reducing the divergence. When p and q do not have the same support, D ()(()))))(z)) can be infinite not well-defined.
12

---

Under review as a conference paper at ICLR 2020
C RELATION TO MMD
Spread divergence can be generally constructed from f-divergence, we show how to build a connection maximum mean discrepancy (Gretton al. (2012) by using the spread total variation distance:
For translation invariant kernel a(, the MMD is (we use to denote the MMD distance with kernel k))
Th (<(x)))la(x)) = -1f 11l (()))))) (((
Suppose both and it's square root of Fourier transform v ("represents the Fourier transform) are absolutely integratable, can also rewrite the MMD distance the as following (see Sriperumbudur et al. (2100) for a derivation):
7h (pll)) (2mr
4/ (2m)--/4110
+ gl
where 4 = (Vk)y (V represents the inverse Fourier transform).
We can further define 1 - where Z (<xxdz. Therefore, O is probability density function and the MMD can be written as
(p(pl))) = (2T) d/4 2 $ al1zz
x $ * t * 4l 1.2-
Total variation distance between p(x) and ((x) is defined as
(p ||p 4l12,-
In the spread total variation distance, we can define spread as the noise distribution k = 0 thus
TV(pllg) | + دل|ا
So MMD and spread Total Variation distance can be written different norms of common objective.
D SPREAD DIVERGENCE BETWEEN TWO DELTA DISTRIBUTIONS
Let (1()) 8(æ (o).P,()) 6(x p1), assume Gaussian spread noise p(y|z) = N (yllr,
and 0² 0.5,
KL(po(=)llp (x)) KL(Po(V)l
KL ( ((oolE)Do(e)) / p(ule)p((s))
KL(N (ului, II (vluz,
o² o² (MI -ui-14 =22)²
log 0²
20²
2
(11 112)2
E STATISTICAL PROPERTIES OF MAXIMUM LIKELIHOOD ESTIMATOR
E. EXISTENCE OF SPREAD MLE
In some situations there may not exist Maximum Likelihood Estimator (MLE) for p( (), but there is MLE for the spread model p(y(0) p(y) ())2)))) For example, suppose that X N(u, o²) (n, < o² < oo). So 0 o2) E R Rt. Assume only have one data point x. Then the log-likelihood function L(r; 0) x log 2nd (a 2 Maximising with respect to μ4, we have does not exist.
= x and the log- log-likelihood becomes unbounded as o² + 0. In this sense, the MLE for (H, o?)
13

---

Under review as a conference paper at ICLR 2020
On the other hand, noise with fixed variance 07. Since we only have one data point z, the spread data distribution becomes p(ylæ) = N ( of) ( and the model is p(y(0) = N (ulu, + We can sample N points from the spread model, so the spread log likelihood function is (neglecting constants) LyI
check whether the MLE for p(y exists. We assume Gaussian spread
UN;; 0) = N log(o² + of)) 200 En (vi 11²2. The MLE solution for μ is = + E V/ the MLE solution for o² is 0² (y - n¹²² o. which has bounded spread likelihood value. Note that the limit of large number of spread samples -> 0o the MIE x² = N (y: - n1² - of tends to O. Throughout, however, the (scaled by N) log likelihood remains bounded.
E.2 CONSISTENCY
Consistency of an estimator an important property that guarantees the validity of the resulting estimate at convergence the number of data points tends to infinity. In what follows, we refer to sufficient conditions for a consistent MLE estimator, before addressing the question whether using spread MLE is also consistent and under what conditions.
E.2.1 CONSISTENCY FOR MLE
Sufficient conditions for the MLE being consistent and converging the global maximum are given in Wald (1949). However, they are usually difficult to check even for some standard distributions. The sufficient conditions for MLE being consistent and converging to a local maxima are given in Cramér 1999) and are more straight forward to check:
C1. (Identifiable): p(=[00;) p(=)02)) 01 = 02.
C2. The parameter space θ is open interval (0, 0). O -00 <0<0<< xc.
C3. p(x(0) is continuous in 0 and differentiable with respect to 0 for all
C4. The set
(x pz(x) 0} is independent of A.
Let X2...... be i.i.d with density p(=(0)) (0 E Q) satisfying conditions CI-C4, then there exists
a sequence 0, = 0, (Xi, Xn) of local maxima of the likelihood function L(0o) = II p(z; which is consistent:
2, G for all 0 θ
The proof can be found in Lehmann (2004) or Cramér 1999).
E.2.2 CONSISTENCY OF SPREAD MILE
provide the necessary conditions for Spread MLE being consistent.
C1. (Identifiable); p(z is identifiable. From section it follows immediately that p(y)01) p(y)02) + p(z(01) p(r (2) + 01 02, where the final implication follows from the assumption that p(a (0) identifiable. Hence if p(x(0) ) is identifiable, so p(y(0).
C2. The parameter space θ an open interval (0, 0). -o S l condition is unchanged for p(yl0).
0 S oc. This
C3. On p(y]0), require the same condition p(z(0) as in MLE; ply is continuous in and differentiable with respect to 0 for all y.
C4. For spread noise p(ylæ) who has full support on Id (for example Gaussian noise), p(y(0) is greater than zero everywhere and hence the original condition C4 is automatically guaran- teed.
The conditions that guarantee consistency for spread MLE are weaker for the spread model p(y(d) than for the standard model p=z]0), since C4 is automatically satisfied. Ferguson (1982) gives example for which MLE exists but not consistent by violating condition C4, whereas spread MLE can be used obtain consistent estimator.
14

---

Under review as a conference paper at ICLR 2020
E.3 ASYMPTOTIC EFFICIENCY
A key desirable property of any estimator is that it is efficient. The Cramer-Raa bound places a lower bound on the variance of any unbiased estimator and an efficient estimator much reach this minimal value in the limit of large amount of data. Under certain conditions (see below) the Maximum Likelihood Estimator attains this minimal variance value meaning that there is no better estimator possible than maximum likelihood (in the limit of large amount of data). This is one of the reasons that the maximum likelihood a cherished criterion.
E.3.1 ASYMPTOTIC EFFICIENCY FOR MLE
Building upon conditions C1-C4, additional conditions on p(z]0) are required to show MLE is asymptotical efficient:
C5. For all x in its support, the density pe( three times differentiable with respect 0 and the third derivative is continuous.
C6. The derivatives of the integral I (e((r)d respect to 0 can be obtained by differentiating under the integral sign, that is: Vo f pa (x))d= @apa
C7. There exists positive number c(6b) and a function Ma, (z) such that
logpo(z)) & Mo, (r) for all x E A. - 8ol < c(0o)
003
where A is the support set of and Eg, [Mo (x)) oo.
Let XI,..,, Xn be i.i.d with density (x)) and satisfy conditions C1-C7, then any consistent sequence i (X1, X,)) of roots of the likelihood equation satisfies
Vn(ō - 06) 4 (O, F(Oo)-
where (06) is the inverse of Fisher information matrix (also called Cramér-Ras Lower Bound, which is lower bound on variance of any unbiased estimators ). The conditions and proof can be found in Lehmann 2004).
E.3.2 ASYMPTOTIC EFFICIENCY FOR MLE
As with MLE above, we require further conditions ply for ensuring spread MLE is asymptoti- cally efficient:
C5. On p(y(0), we require the same condition as applied to p(z(0) the MLE case; for all in its support, the density po(y) is three times differentiable with respect 0 and the third derivative is continuous.
C6. For spread noise p(y(z), which has full support on I2 (for example Gaussian noise), the support of is independent of f. Leibniz's rule allows us differentiate under the integral: (Oppoy)dyy so this condition is automatically satisfied.
Vo po(y)dy
C7. On p(y(0), require the same condition as applied p(z(0) in the MLE case; There exist positive number c(fb) and a function Ma, (y) such that
log (v() < Mo. (v) for all y A. 10 0ol < c(fo)
003
where A is the support set of and Es, [Mo, (v)l < cc.
Thus the conditions that guarantee asymptotically efficient for the spread model p(y(f) are weaker than for the standard model p(x) since C4 and C6 are automatically satisfied.
$ LLibniz's rule tells us: d (b))) p(z, 0) = (200) Öop(a,, d) + p(b(0), 0) ( p(a(0), (0)). if a()) and b(8) are independent of then S p(z, fyd Sht aop(z, 0)dr.
15

---

Under review as a conference paper at ICLR 2020
F PERTURBATION APPROXIMATION OF GAUSSIAN SPREAD
Herein, in the fixed noise setting, we derive a perturbation based approximation to the spread noise, which in princple can lead lower variance estimator. We can write function with perturbed input and integrated over noise as
Epee) [f(x + E)),
(33)
where p(E) = N (g10, E). Taylor expanding around ç = 0, we have
f(z) + f + [ +((9)
pp()) [f(x + E)] E pp))
(34)
2 f(z) + Ro(e) [SH
(35)
2
fx) + "Tr (Ep(e) [sE"] H)
(36)
2
f1) + Tr (EH)
(37)
2
Where H is the Hessian matrix H, = 82
Tw,ōu, When x's dimension is small, we can use equation 37
to explicitly calculate the trace. When the dimension of x is large, we can form a Monte Carlo
estimation of equation 35 To this we first sample & ~ p(E) and then calculate the Hessian-vector
product HE This can be efficiently calculated by AutoDiff backward mode (Schraudolph (002); Pearlmutter (1994), without storing the Hessian matrix in memory.
For example, in 8- VAE with fixed Gaussian noise, according to the bound equation 56 fignorng the constant):
f N (ylzz,0lx) log (v)) 2 1N(c.10,0"xx) [M + oc.)))
(38)
+ +EN(c,10,o2x)) [E((oo.)) plog(( p(æ ge(ua + rex) + Co(x)e)) + logp(z = pa(a + σe + Co (z)e)]
(39)
H(E((z)) + Ex(ejo.)) [logp(a go + Cp(x)))) +og +og( =m((=) ( + Tr (H)
f(x)
(40)
8 f(m) dw,day and C.(x) is the Cholesky decomposition of (x)
Where H is the Hessian matrix H,
G SPREAD DIVERGENCE FOR DETERMINISTIC DEEP GENERATIVE MODELS
Instead of minimising the likelihood, we train an implicit generative model by minimising the spread
divergence
min (L (o()))
(41)
For Gaussian noise with fixed diagonal noise p(u|=) = NNrk,, we can write
p(u) N
N (vlon,
(42)
and
(v) plul=)po(z)ddr = / N (slgo(z)) px)))) = po(ul=)p(=)d=
(43)
For the spread divergence with learned covariance Gaussian noise which is discussed in section we can write
4.1
N Σ Al (ul Ex)
Po Nool Ex),
p(y) = N
(44)
n=1
16

---

Under review as a conference paper at ICLR 2020
and spread divergence with learned injective function discussed in section 4.2)
N
(o(y)) = N(ylfo(())) ply) = N
Nyulfo(=)
(45)
According to our general theory,
min (z)) = 0
p(x) = (p(
(46)
Here
(L(P())lbo(v) N ==1
Σ
p(y) log po (y)dy + const.
(47)
Typically, the integral y will be intractable and resort to unbiased sampled estimate (though see below for Gaussian g). Neglecting constants, the KL divergence estimator is
log Po(
(48)
NS
where Vn is perturbed version of Zn. For example y ~ ~N ( o² for fixed Gaussian noise case and other cases are similar. In most cases of interest, with non-linear y. the distribution (v) is intractable. We therefore the variational lower bound
log po()) 2 (2(())) (- log go (2/b) log (po(y ())(()))) dz
(49)
Parameterising the variational distribution as Gaussian,
(0(2ly) = N (v)
(50)
then we can reparameterise and write
log fo (v) 2 H(E((v)) + EE o,1) [log (po = us + Colv)e)p(== = ps(v) + C+(v)e)) (51)
where (y)) the entropy of Gaussian with covariance Eg(y). For fixed covariance Gaussian spread noise in D dimensions, this is
log (v) 2 [(2)(w) (202) (v go (ms(r) Co(u)e)) + logg(z moly) Co(w)e)
+const.
(52)
where (y) is the Cholesky decomposition of (v).
We can integrate equation 52 over to give the bound
f NV (vlæ, o² xx) log (v) 2
[H(Es(v)) + Extejo.) [logp(= = ma(x) + (v)e
(53)
D12 EN(c)Oo)) [E((ll [w (o Cc(w)))) + const.
1
(54)
where
[wV go (molv) C
Ex(glu0011
0² 2EN 0.1x) (eago σex) + Co(y))) + ENic. 0x1x) [(x go(uo (x + oe.) + Co(x)e))
(55)
We notice that the second term is zero, so the final bound for the fixed Gaussian spread KI. divergence (ignoring the constant)
/ N (vlz, xxIxx log (v( EN( xx)2 (HEs (v)) Ex(eoon [log(2 (4(y) Co(v))]]]
(202)072 Extee Ex(ejO.) (xr Lolsoouu go(us (x + dew) + Cot))²]
(56)
17

---

Under review as a conference paper at ICLR 2020
By analogy, for spread KL divergence with learned variance, the bound is (ignoring the constant)
/ N (ulz, Exp log (o()) 2 EM(wl=,E,) H (v))(v)) + Beioi (xteoon [log((= Syez)
= (v( Co(U)c]]
-E((.,0o..)) Evtcion) [< Lle-gelpg go(ps (x
+ + Co(s)e)) Ez' (x- go(mo (a Syez) + ((())e))
(57)
Where Sy is the cholesky decomposition of Eg. For specific covariance structure introduced in section section(4.I), efficient methods for sampling, matrix inverting and log determinant calculation available, see appendix
For spread KL divergence with learned injective function, the bound is (ignoring the constant)
f N ( log (o) oNlul,,02x)) (H{((v)) Ex(eion (logp(= ma(v) Ca())e]]
(202) D/2 E 1l(e [ENCelo.)) [UF(()) - fulgo(us (f((r) + ocz) + Co(v)c)))
-
(58)
The overall procedure is therefore a straightforward modification of the standard VAE method Kingma Welling 2013) with additional learning the spread to maximize the divergence:
1. Choose noise distribution p(ulæ))
2. Choose tractable family for the variational distribution, for example ga(2ly)
N (z] (v)) and initialise Φ.
3. We then sample a Yn, for each datapoint (if we're using samples)
4. If learning the spread noise:
(a) Draw samples to estimate - log (V)) according to the corresponding bound.
(b) Do gradient ascent step in tb.
5. Draw samples to estimate log (y(.) according to the corresponding bound.
6. Do gradient ascent step in (0, Φ).
7. Go to 3 and repeat until convergence.
H MNIST EXPERIMENT
We first scaled the MNIST data lie in [0, 1]. We use Laplace spread noise = 0.3 and Gaussian spread noise a = 0.3 for the 5VAA.. Both encoder and decoder contains 3 feed-forward layers, each layer with 400 units and ReLu activation function. The latent dimension is = 64. The variational inference network (=ly) NV (=)mo(v).og!1 dz has similar structure for the mean network Mo (y). For fixed spread 8-VAE, learning was done using the Adam Kingma & Ba (014)) optimizer with learning rate -4 for 200 epochs. For 5- VAE with learned spread (learned covariance), we additioanly train the covariance for epochs using Adam optimizer with learning rate 5e after everytime we train the model for 10 epochs.
I CELEBA EXPERIMENT
We pre-processed CelebA images by first taking 140x140 centre crops and then resizing to 64x64.
Pixel values were then rescaled to lie in [0, 1]. For the learned spread we use Gaussian noise with learned injective function ResNet f.) neural net with kernel size 3 x and stride 1. We use spectral normalization Miyato et al. (2018) to satisfy the Lipschitz constraint:
I(.) + g.((), where (- is one layer convolutional
replace the weight matrix of the convolution kernel by WSN (w) := e x (/o(w) where a (w) is the spectral norm of w and c E (0, 1). This guarantees that fo is invertible see Behrmann al. (2018).
The encoder and decoder are 4-layer convolutional neural net with batch norm Loffe & Szegedy (015)). Both encoder and decoder used fully convolutional architectures with 5x5 convolutional
18

---

Under review as a conference paper at ICLR 2020
filters and used vertical and horizontal strides 2 except the last deconvolution layer used stride The injective function f is a 1 layer convolutional network with 3 kernl and stride 1. Conv stands for convolution with & filters, DeConv; for deconvolution with k filters, BN for the batch normalization loffe & Szegedy connected layer mapping to R
2015). Relu for the rectified linear units, and FC2 for the fully
R64x64x3
injective E R64x64x3
ConV128 + BN + Relu
+ Conv256 + BN Relu
7 Conv612 + BN 7 Relu
ConV1024 BN Relu FC
# E 1000
7 FC10x10x1022
DeConv512 + BN Relu
DeConv256 + BN + Relu
DeConv12s BN Relu DeConv; + sigmoid(.)
injective E R64x6443
We use batch size 100 and latent dimension 2dim 100 in all CelabA experiments. For the 8 J-VAE with fixed spread, we use the fixed Gaussian noise with 0 mean and (0.5)* covariance. We train the model for 500 epochs using Adam optimizer with learning rate le -1, The learning rate decay with scaling factor 0.9 every 100000 iterations.
For the VAE with fixed spread We first train a VAE with fixed f(x) = x and fixed Gaussian noise with 0 mean and (0.5)²/ diagonal covariance for 300 epochs, the learning rate decay with scaling factor 0.9 every 100000. Then we start iterative training by doing one step inner maximisation over the spread divergence's parameter using Adam optimizer with learning rate 1e and one step minimization over the model parameter's (0, 6) using Adam optimizer for additional 200 epochs.
We share the first 300 epochs between two models. When sample form models, first sample from a 100 dimensional standard Gaussian distribution 2 ~ N(0, I) and use the same latent code to get samples from both 5- VAE with fixed and learned spread, sample quality between two models.
easily compare the
J WOODBERRY
When evaluating the log probability of this Gaussian, we use the Woodberry identity
Σ-1 -Io2)-'-(Io))
- "10'="'1">"">>111
so we only have to invert a R x R matrix. A similar trick is applied to calculate the log determinant:
log det det((e))) log det + 1." (I=")))_) 2D log
The parameter L is trained using reparameterization trick. When sampling & ~ N N(, LL + Io?). we first sample E Rd from A (=10, I) and then sample noise ~~ N (0, Lo thus sample from N (N, LL + Io²) can be represented by La + Iae μ.
K DETERMINISTIC LINEAR LATENT MODEL.
Our aim here is to show how the classical deterministic PCA algorithm can be derived through a approach, rather than the classical probbbbbbiii least-squares derivation. This is remarkable since the likelihood itself is not defined for this model.
For isotropic Gaussian observation noise with variance 72 the Probabilistic PCA model Tipping & Bishop 1999) for X -ddimensional observations and Z-dimensional latent is
= Fz + 7e, ~~ N(0, ~~ (0,
(x = N (v FT + 1xx
(59)
19

---

Under review as a conference paper at ICLR 2020
When y 0, the generative mapping from to is deterministic and the model (0 has support only on a subset IR and the data likelihood general not defined for Z X.
In the following the model to iid data (II...., 2N } using maximum likelihood, the only information required from the dataset is the data covariance Σ. For 7 > O, the maximum likelihood solution for PPCA is - - I2)² R, where Az, Uz are the Z largest eigenvalues, eigenvectors of Σ; is an
consider general 7 later setting 7 to zero throughout the calculation. To fit
== Uz (Az-72 arbitrary orthogonal matrix. Using spread noise p(y|z) = N (ylæ, x²Ix), the spreaded distribution is Gaussian (o(y) N (vl0, 0, FF + (72 + x²) Thus, pe (v) is of the same form PPCA, albeit with an inflated covariance matrix. Adding Gaussian spread noise to the data also simply inflates the sample covariance to Σ' Σ + o²1x.
Since the eigenvalues of Σ' Σ + 0 I²Ix are simply A' = = o2Ixx with unchanged eigenvectors, the optimal deterministic (y 0) latent linear model has solution Uz (A22 o² R.. We have thus recovered the standard PCA solution; however, the derivation is non-standard since the likelihood the deterministic latent linear model = is not defined. Since classical deterministic PCA cannot normally be described terms of likelihood, the usual derivation of PCA is to define as the optimal least squares reconstruction solution based on a linear projection to lower-dimenssonal subspace, see for example Barber (2012). Nevertheless, using the spread divergence, learn sensible model and recover the true data generating process if the data were exactly generated according to the deterministic model.
20

---

Under review as a conference paper at ICLR 2020
0
1
8
6
0
T
5
3
3
2
2
6
5
Q
3
3 Z 6 / 3 8
5 9 0 9
3
6
/
3
0
8
5
7 4
2
4
3
6
L
9
3
0
3
7
7
4
8
6
4
2
6
6
6
?
8
1
7
2
6
9
0
0
9
4
5
2 5 f
6
1
7
8
3
3
1
8
8 1 0
J
0
8
6
4
0
3
9
7
6
9
(
E
3
0
3
(a) Laplace with fixed covariance
e
6
1
1
(
0
T
3
0
1
6
8
7
9
3
3
8 4 0 9
3
6
1
3
0
8
8
7
0 4 4 7 8 6 7 0
2
3
8
2
7
3
4 4 7 5 3
0
1
7
7
4
8
6
2
1
6
6
9
6
/
8
8
6
1
0
7
6
4 5
0
/
4
6
3
8
c
1
8
4
1
9
4
0
3
9
2
6
5
4
8
7
0
3
(b) Gaussian with fixed covariance
L
6
/ 7
6
0
7
5
3
2
1
8
3
9
4
3
3
6 7 8 9
3
6
1
3
0
8
B
7 9
2 4 Q
2
3
3
6
1
7
3
4
3
0
7
4
8
6
2
6
6
6
2
S
1
7
2 8
2
6
9
1
0
9
6
5
0
6
4
7
6
3
8
q
3
9 3 0
5 1
7
7
9
9
6
0
0
3
9
9
4
9
4
8
I
0
3
Gaussian with learned covariance
Figure 4: Samples from generative model (deterministic output) trained using 5-VAE with (a) Laplace noise with fixed covariance, (a) Gaussian noise with fixed covariance and (c) Gaussian noise with learned covariance.
21

---

Under review as a conference paper at ICLR 2020
(a) Fixed spread noise
T
E
2
A
(b) Learned spread noise
Figure 5: Samples from a generative model with deterministic output trained using 8- VAR with (a) fixed and (b) learned spread with injective mean transform.
22


Reviews:
[{"review_id": "SJlHMmjiKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduced a way to modify densities such that their support agrees and that the Kullback-Leibler divergence can be computed without diverging. Proof of concept of using the spread KL divergence to ICA and Deep Generative Models ($\\delta$-VAE) are reported based on the study of spread MLE.\n\nComments:\n\nIn Sec 1, mention that f should be strictly convex at 1. Also mention \nJensen-Shannon divergence, a KL symmetrization, which is always finite \nand used in GAN analysis.\n\nIn Sec 2, you can also choose to dilute the densities with a mixture: \n(1-\\epsilon)p+\\epsilon noise.\nExplain why spread is better than that? Does spreading introduce \nspurious modes?, does it change distribution sufficiency? \n(Fisher-Neymann thm)\n\nIn Formula 4, there is an error: missing denominator of \\sigma^2. See \nAppendix D too.\n\nIn footnote 4, page 8, missing a 1/2 factor in from of TV (that is upper \nbounded by 1 and not 2)\n\nKL is relative entropy= cross-entropy minus entropy. What about spread KL?\nIn general, what statistical properties are kept by using the spread? \n(or its convolution subcase?)\n\nIs spreading a trick that introduces a hyperparameter that can then be \noptimized for retaining discriminatory power, or is there\nsome deeper statistical theory to motivate it. I think spread MLE should \nbe further explored and detailed to other scenarii.\n\nSpreading can be done with convolution and in general by Eq.3:\n\nThen what is the theoretical interpretation of doing non-convolutional \nspreading?\n\n\nA drawback is that optimization on the spread noise hyperparameter is \nnecessary (Fig 3b is indeed much better than Fig 3a).\nIs there any first principles that can guide this optimization rather \nthan black-box optimization?\n\nOverall, it is a nice work but further statistical guiding principles \nor/and new ML applications of spread divergences/MLE will strengthen the \nwork.\nThe connection, if any, with Jensen-Shannon divergence shall be stated \nand explored.\n\nMinor comments:\n\nIn the abstract, state KL divergence instead of divergence because \nJensen-Shannon divergence exists always.\n\n\nTypos:\np. 6 boumd->bound\nBibliography : Cramir->Cramer, and various upper cases missing (eg. \nwasserstein ->Wasserstein)\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJe9q6g09r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new divergence, called spread divergence, to distinguish probability models. The approach is motivated from the concern that traditional divergence such as f-divergence or KL divergence may not always exist, in which the spread divergence may be a substitute. Some empirical supports are provided for the proposed method. Below I will summarize my concerns.\n\n1. The spread divergence is proven no larger than the traditional divergence, so the paper claims this as an advantage of using spread divergence. My question is that if KL or f-divergence of two probability models is infinity, which means they distinguish the models very well, whether a new method is necessary (though it may provide a finite value). \n\n2. As a new method, it would be useful to thoroughly compare it with the traditional ones. There is a lack theoretical comparison with the KL or f-divergence. Some numerical examples are provided but seem not enough. For instance, the applications focus on the situations where likelihood is not defined, i.e., data are deterministic w/o observation noise. It is interesting to see other examples where likelihood is defined and how traditional methods perform.\n\n3. Kernel based spread divergence has been a major focus of this paper. It is interesting to see which kernel maximizes the spread divergence. Section 3.2 considers Gaussian kernel. Is this an optimal option?\n\n4. Section 5 compares EM and spread EM based on one experiment and claims the latter has smaller error.   Does the same conclusion holds true in other examples?\n\nI believe the motivation of this paper is interesting. This would be a stronger paper if more theoretical and empirical analysis can be added.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1ex_oAT9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the paper must be Desk-rejected as the identity of the authors was revealed. \nThis thing aside, the paper is an interesting contribution. The concept of spread divergence can be valuable in many context. The presentation is thorough and the theoretical part is correct. On the other hand, the examples are quite diverse and include a standard model (ICA) as well as modern deep generative models. Thus, it represents a valuable contribution worth of publication, if we ignore the identity revelation aspect. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies spread divergence between distributions, which may exist in settings where the divergence between said distributions does not. The reviewers feel this work does not have sufficient technical novelty to merit acceptance at this time.
Decision: Reject