Spread Divergences

1 INTRODUCTION . A divergence D ( p||q ) ( see , for example Dragomir ( 2005 ) ) is a measure of the difference between two distributions p and q with the property D ( p||q ) ≥ 0 , and D ( p||q ) = 0 ⇔ p = q ( 1 ) Some of our results are specific to the f -divergence , defined as Df ( p||q ) = Eq ( x ) [ f ( p ( x ) q ( x ) ) ] ( 2 ) where f ( x ) is a convex function with f ( 1 ) = 0 . An important special case of an f -divergence is the well-known Kullback-Leibler divergence KL ( p||q ) = Ep ( x ) [ log p ( x ) q ( x ) ] which is widely used to train models using maximum likelihood . We are interested in situations in which the supports of the two distributions are different , supp ( p ) 6= supp ( q ) . In this case the divergence may not be defined . For example , for p ( x ) being an empirical data distribution on continuous dataset x1 , . . . , xN , p ( x ) = 1 N ∑N n=1 δ ( x− xn ) where δ ( · ) is the Dirac Delta function . For a model q ( x ) with support R , then KL ( q||p ) is not formally defined . This is a challenge since implicit generative models of the form q ( x ) = ∫ δ ( x− gθ ( z ) ) p ( z ) dz only have limited support ; in this case maximum likelihood to learn the model parameter θ is not available and alternative approaches are required – see Mohamed & Lakshminarayanan ( 2016 ) for a recent survey . 2 SPREAD DIVERGENCES . The aim is , from q ( x ) and p ( x ) to define new distributions q̃ ( y ) and p̃ ( y ) that have the same support1 . Using the notation ∫ x to denote integration ∫ ( · ) dx for continuous x , and ∑ x∈X for discrete x with domain X , we define a random variable y with the same domain as x and distributions p̃ ( y ) = ∫ x p ( y|x ) p ( x ) , q̃ ( y ) = ∫ x p ( y|x ) q ( x ) ( 3 ) where p ( y|x ) is a ‘ noise ’ process designed to ‘ spread ’ the mass of p and q such that p̃ ( y ) and q̃ ( y ) have the same support . For example , if we use a Gaussian p ( y|x ) = N ( y x , σ2 ) , then p̃ and q̃ both have support R. We therefore use noise with the property that , despite D ( p||q ) not existing , D ( p̃||q̃ ) does exist and we define the Spread Divergence D̃ ( p||q ) ≡ D ( p̃||q̃ ) ( 4 ) Note that this satisfies the divergence requirement D̃ ( p||q ) ≥ 0 . The second requirement , D̃ ( p||q ) = 0⇔ p = q , is guaranteed for certain ‘ noise ’ processes , as described in section ( 2.1 ) . 1For simplicity , we use univariate x , with the extension to the multivariate setting being straightforward . Spread divergences have many potential applications . For example , for a model q ( x|θ ) with parameter θ and empirical data distribution p ( x ) , maximum likelihood training corresponds to minimising KL ( p||q ) with respect to θ . However , for implicit models , the divergence KL ( p||q ) does not exist . However , if a spread divergence exists , provided that the data is distributed according the model p ( x ) = q ( x|θ0 ) for some unknown parameter θ0 , the spread divergence D ( p ( x ) ||q ( x|θ ) ) has a minimum at θ = θ0 . That is ( for identifiable models ) we can correctly learn the underlying data generating process , even when the original divergence is not defined . 2.1 NOISE REQUIREMENTS FOR A SPREAD DIVERGENCE . Our main interest is in using noise to define a new divergence in situations in which the original divergence D ( p||q ) is itself not defined . For discrete variables x ∈ { 1 , . . . , n } , y ∈ { 1 , . . . , n } , the noise Pij = p ( y = i|x = j ) must be a distribution ∑ i Pij = 1 , Pij ≥ 0 and∑ j Pijpj = ∑ j Pijqj ∀i ⇒ pj = qj ∀j ( 5 ) which is equivalent to the requirement that the matrix P is invertible , see appendix ( B ) . There is an additional requirement that the spread divergence exists . In the case of f -divergences , the spread divergence exists provided that p̃ and q̃ have the same support . This is guaranteed if∑ j Pijpj > 0 , ∑ j Pijqj > 0 ∀i ( 6 ) which is satisfied if Pij > 0 . In general , therefore , there is a space of noise distributions p ( y|x ) that define a valid spread divergence . The ‘ antifreeze ’ method of Furmston & Barber ( 2009 ) is a special form of spread noise to define a valid Kullback-Leibler divergence ( see also Barber ( 2012 ) ) . For continuous variables , in order that D̃ ( p||q ) = 0 ⇒ p = q , the noise p ( y|x ) , with dim ( Y ) = dim ( X ) must be a probability density and satisfy∫ p ( y|x ) p ( x ) dx = ∫ p ( y|x ) q ( x ) dx ∀y ∈ Y ⇒ p ( x ) = q ( x ) ∀x ∈ X ( 7 ) This is satisfied if there exists a transform p−1 such that∫ p−1 ( x′|y ) p ( y|x ) dy = δ ( x′ − x ) ( 8 ) where δ ( · ) is the Dirac delta function . As for the discrete case , the spread divergence exists provided that p̃ and q̃ have the same support , which is guaranteed if p ( y|x ) > 0 . A well known example of such an invertible integral transform is the Weierstrass Transform p ( y|x ) = N ( y x , σ2 ) , which has an explicit representation for p−1 . In general , however , we can demonstrate the existence of a spread divergence without the need for an explicit representation of p−1 . As we will see below , the noise requirements for defining a valid spread divergence such that D ( p̃||q̃ ) = 0⇔ p = q are analogous to the requirements on kernels such that the Maximum Mean Discrepancy MMD ( p , q ) = 0 ⇔ p = q , see Sriperumbudur et al . ( 2011 ) and Sriperumbudur et al . ( 2012 ) . 3 STATIONARY SPREAD DIVERGENCES . Consider stationary noise p ( y|x ) = K ( y − x ) where K ( x ) is a probability density function with K ( x ) > 0 , x ∈ R. In this case p̃ and q̃ are defined as a convolution p̃ ( y ) = ∫ K ( y−x ) p ( x ) dx = ( K ∗ p ) ( y ) , q̃ ( y ) = ∫ K ( y−x ) q ( x ) dx = ( K ∗ q ) ( y ) ( 9 ) Since K > 0 , p̃ and q̃ are guaranteed to have the same support R. A sufficient condition for the existence of the Fourier Transform F { f } of a function f ( x ) for real x is that f is absolutely integrable . All distributions p ( x ) are absolutely integrable , so that both F { p } and F { q } are guaranteed to exist . Assuming F { K } exists , we can use the convolution theorem to write F { p̃ } = F { K } F { p } , F { q̃ } = F { K } F { q } ( 10 ) Hence , we can write D ( p̃||q̃ ) = 0⇔ p̃ = q̃ ⇔ F { K } F { p } = F { K } F { q } ⇔ F { p } = F { q } ⇔ p = q ( 11 ) where we used the invertibility of the Fourier transform and assumed that F { K } 6= 0 , or equivalently2 , F { K } > 0 . Hence , provided that K ( x ) > 0 and F { K } > 0 then K ( x ) defines a valid spread divergence . Note that other transforms have a corresponding convolution theorem3 and the above derivation holds , with the requirement that the corresponding transform of K ( x ) is non-zero . As an example of such a noise process , consider Gaussian noise , K ( y − x ) = 1√ 2πσ2 e− 1 2σ2 ( y−x ) 2 ( 12 ) leading to a positive Fourier Transform : F { K } ( ω ) = 1√ 2πσ2 ∫ ∞ −∞ eiωxe− 1 2σ2 x2dx = e− σ2ω2 2 > 0 ( 13 ) Similarly , for Laplace noise K ( x ) = 12be − 1b |x| p ( y|x ) = K ( y − x ) = 1 2b e− 1 b |y−x| , F { K } ( ω ) = √ 2 π b−1 b−2 + ω2 > 0 ( 14 ) Since K > 0 and F { K } > 0 , this also defines a valid spread divergence over R . 3.1 INVERTIBLE MAPPINGS . Consider p ( y|x ) = K ( y − f ( x ) ) for strictly monotonic f . Then , using the change of variables p̃ ( y ) = ∫ K ( y − z ) pz ( z ) dz , pz ( z ) = px ( f−1 ( z ) ) ( |J ( x = f−1 ( z ) ) | ) −1 ( 15 ) where J is the Jacobian of f . For distributions p ( x ) with bounded domain , for example x ∈ [ 0 , 1 ] , we can use a logit function , f ( x ) = − log ( x−1 − 1 ) , which maps the interval [ 0 , 1 ] to R. Us- ing then , for example , Gaussian spread noise p ( y|x ) = N ( y f ( x ) , σ2 ) , both p̃ ( y ) and q̃ ( y ) have support R. If D ( p̃||q̃ ) is zero then p = q on the domain [ 0 , 1 ] . 3.2 MAXIMISING THE SPREAD . From the data processing inequality ( see appendix ( A ) ) , spread noise will always decrease the f - divergence Df ( p̃ ( y ) ||q̃ ( y ) ) ≤ Df ( p ( x ) ||q ( x ) ) . If we are to use a spread divergence to train a model , there is the danger that adding too much noise may make the spreaded empirical distribution and spreaded model distribution so similar that it becomes difficult to numerically distinguish them , impeding training . In general , therefore , it would be useful to add noise such that we define a valid spread divergence , but can maximally still discern the difference between the two distributions . To gain intuition , we define p and q to generate data in separated linear subspaces , p ( x ) = ∫ δ ( x− a−Az ) p ( z ) dz , q ( x ) = ∫ δ ( x− b−Bz ) p ( z ) dz , p ( z ) = N ( z 0 , IZ ) . Using Gaussian spread , p ( y|x ) = N ( y µ , Σ ) , what is the optimal µ , Σ that maximises the divergence ? Clearly , as Σ tends to zero , the divergence increases to infinity , meaning that we must at least constrain the entropy of p ( y|x ) to be finite . In this case the spreaded distributions are given by p̃ ( y ) = N ( y µ+ a , AAT + Σ ) , q̃ ( y ) = N ( y µ+ b , BBT + Σ ) ( 16 ) We define a simple Factor Analysis noise model with Σ = σ2I + uuT , where σ2 is fixed and uTu = 1 . The entropy of p ( y|x ) is then fixed and independent of u . Also , for simplicity , we assume A = B . It is straightforward to show that the spread divergence KL ( p̃||q̃ ) is maximised for u pointing orthogonal to the vector ( AAT + σ2I ) −1 ( b− a ) . Then u optimally points along the direction in which the support lies . The support of p ( y|x ) must be the whole space but to maximise the divergence the noise preferentially spreads along directions defined by p and q , see figure ( 1 ) . 2If F { K } can change sign , by continuity , there must exist a point at which F { K } = 0 . 3This includes the Laplace , Mellin and Hartley transforms .

Summary: This paper introduces spread divergences. Spread divergences are obtained by taking the divergence between smoothed/noisy versions of two distributions. A spread divergence between two distributions of non-overlapping support can be defined even when the corresponding divergence is not. The authors discuss conditions under which the data generating process can be identified by minimizing spread divergences and apply spread divergences to the examples of PCA, ICA, and noiseless VAE.