A Mean-Field Theory for Kernel Alignment with Random Features in Generative Adverserial Networks

1 INTRODUCTION . A fundamental and long-standing problem in unsupervised learning systems is to capture the underlying distribution of data . While deep generative models such as Boltzmann machines Salakhutdinov & Hinton ( 2009 ) and auto-encoding variational Bayes Kingma & Welling ( 2013 ) accomplish this task to some extent , they are inadequate for many intractable probabilistic computations that arise in maximum likelihood estimation . Moreover , in many machine learning tasks such as caption generation Xu et al . ( 2015 ) , the main objective is to obtain new samples rather than to accurately estimate the underlying data distribution . Generative adverserial network ( GAN ) Goodfellow et al . ( 2014 ) provides a framework to directly draw new samples without estimating data distribution . It consists of a deep feedforward network to generate new samples from a base distribution ( e.g . Gaussian distribution ) , and a discriminator network to accept or reject the generated samples . However , training GAN requires finding a Nash equilibrium of a non-convex minimax game with continuous , high-dimensional parameters . Consequently , it is highly unstable and prone to miss modes Salimans et al . ( 2016 ) ; Che et al . ( 2016 ) . To obtain more stable models , the generative moment matching networks ( GMMNs ) Li et al . ( 2015 ) are proposed , wherein instead of training a discriminator network , a non-parametric statistical hypothesis test is performed to accept or reject the generated samples via the computation of the kernel maximum mean discrepancy Gretton et al . ( 2007 ) . While leveraging a statistical test simplifies the loss function for training GMMN , in practice , the diversity of generated samples by GMMN is highly sensitive to the choice of the kernel . Thus , to improve the sampling performance , the kernel function also needs to be jointly optimized with the generator . Rather than i optimizing the kernel directly , the MMD GAN model Li et al . ( 2017 ) is proposed in which an embedding function is optimized in conjunction with a fixed user-defined kernel ( e.g . RBF Gaussian kernel ) . However , there are no theoretical guarantees that the user-defined kernel is the ‘ right ’ kernel for embedded features . Contributions . To address the kernel model selection problem in MMD GAN Li et al . ( 2017 ) , in this paper we put forth a novel framework to learn a good kernel function from training data . Our kernel learning approach is based on a distributional optimization problem to learn a good distribution for the random feature model of Rahimi and Recht Rahimi & Recht ( 2008 ; 2009 ) to approximate the kernel . Since optimization with respect to the distribution of random features is infinite dimensional , we consider a Monte Carlo approximation to obtain a more tractable finite dimensional optimization problem with respect to the samples of the distribution . We then use a particle stochastic gradient descent ( SGD ) to solve the approximated finite dimensional optimization problem . We provide a theoretical guarantee for the consistency of the finite sample-average approximations . Based on a mean-field analysis , we also show the consistency of the proposed particle SGD . In particular , we show that when the number of particles tends to infinity , the empirical distribution of the particles in SGD follows the path of the gradient descent flow of the distributional optimization problem on the Wasserstein manifold . 2 PRELIMINARIES OF MMD GANS . Assume we are given data { vi } ni=1 that are sampled from an unknown distribution PV with the support V . In many unsupervised tasks , we wish to attain new samples from the distribution PV without directly estimating it . Generative Adversarial Network ( GAN ) Goodfellow et al . ( 2014 ) provides such a framework . In vanilla GAN , a deep network G ( · ; W ) parameterized by W ∈ W is trained as a generator to transform the samples Z ∼ PZ , Z ∈ Z from a user-defined distribution PZ ( e.g . Gaussian ) into a new sample G ( Z ; W ) ∼ PW , such that the distributions PW and PV are close . In addition , a discriminator network D ( · ; δ ) parameterized by δ ∈ ∆ is also trained to reject or accept the generated samples as a realization of the data distribution . The training of the generator and discriminator networks is then accomplished via solving a minimax optimization problem as below min W∈W max δ∈∆ IEPV [ D ( X ; δ ) ] + IEPZ [ log ( 1−D ( G ( Z ; W ) ; δ ) ) ] . ( 1 ) In the high dimensional settings , the generator trained via the min-max program of equation 1 can potentially collapse to a single mode of distribution where it always emits the same point Che et al . ( 2016 ) . To overcome this shortcoming , other adversarial generative models are proposed in the literature , which propose to modify or replace the discriminator network by a statistical two-sample test based on the notion of the maximum mean discrepancy which is defined below : Definition 2.1 . ( MAXIMUM MEAN DISCREPANCY GRETTON ET AL . ( 2007 ) ) Let ( X , d ) be a metric space , F be a class of functions f : X → IR , and P , Q ∈ B ( X ) be two probability measures from the set of all Borel probability measures B ( X ) onX . The maximum mean discrepancy ( MMD ) between the distributions P and Q with respect to the function class F is defined below DF [ P , Q ] def = sup f∈F ∫ X f ( x ) ( P −Q ) ( dx ) . ( 2 ) Different choices of the function class F in equation 2 yield different adversarial models such as Wasserstein GANs ( WGAN ) Arjovsky et al . ( 2017 ) , f -GANs Nowozin et al . ( 2016 ) , and GMMN and MMD GAN Li et al . ( 2017 ; 2015 ) . In the latter two cases , the function class F def= { f : X → IR : ‖f‖HX ≤ 1 } , where HX is a reproducing kernel Hilbert space ( RKHS ) of functions with a kernel K : X × X → IR , denoted by ( HX , K ) . Then , the squared MMD loss in equation 2 as a ii measure of the distance between the distributions PV and PW has the following expression DK [ PV , PW ] def = sup f : X→IR : ‖f‖HX≤1 ∫ X f ( x ) ( PV − PW ) ( dx ) ( 3 ) = IEV , V ′∼PV [ K ( V ; V ′ ) ] + IEW , W ′∼PW [ K ( W , W ′ ) ] − 2IEV ∼PV , W∼PW [ K ( V ; W ) ] , ( 4 ) where X = V ∪ W . Instead of training the generator via solving the minimax optimization in equation 1 , the MMD GAN model of Li et al . ( 2015 ) proposes to optimize the discrepancy between two distributions via optimization of an embedding function ι : IRD 7→ IRp , p ≤ D , i.e. , min W∈W max ι∈Q MMDk◦ι [ PV , PW ] , ( 5 ) where k : IRp × IRp → IR is a user-defined fixed kernel . In Li et al . ( 2015 ) , the proposal for the kernel k : IRp × IRp → IR is a mixture of the Gaussians , k ◦ ι ( x , y ) = k ( ι ( x ) , ι ( y ) ) = m∑ i=1 ( ‖ι ( x ) − ι ( y ) ‖22 σ2i ) , ( 6 ) where the bandwidth parameters σ1 , · · · , σm > 0 are manually selected . Nevertheless , in practice there is no guarantee that the user-defined kernel k ( ι ( x ) , q ( y ) ) can capture the structure of the embedded features ι ( x ) . 3 PROPOSED APPROACH : KERNEL LEARNING WITH RANDOM FEATURES FOR . MMD GANS In this section , we first expound our kernel learning approach . Then , we describe a novel MMD GAN model based on the proposed kernel learning approach . 3.1 ROBUST DISTRIBUTIONAL OPTIMIZATION FOR KERNEL LEARNING . To address the kernel model selection issue in MMD GAN Li et al . ( 2017 ) , we consider a kernel optimization scheme with random features Rahimi & Recht ( 2008 ; 2009 ) . Let ϕ : IRd × IRD → [ −1 , 1 ] denotes the explicit feature maps and µ ∈ M ( IRD ) denotes a probability measure from the space of probability measuresM ( IRD ) on IRD . The kernel function is characterized via the explicit feature maps using the following integral equation Kµ ( x , y ) = IEµ [ ϕ ( x ; ξ ) ϕ ( y ; ξ ) ] = ∫ Ξ ϕ ( x ; ξ ) ϕ ( y ; ξ ) µ ( dξ ) . ( 7 ) Let MMDµ [ PV , PW ] def = MMDKµ [ PV , PW ] . Then , the kernel optimization problem in can be formulated as a distribution optimization for random features , i.e , min W∈W sup µ∈P MMDµ [ PV , PW ] . ( 8 ) Here , P is the set of probability distributions corresponding to a kernel class K. In the sequel , we consider P to be the distribution ball of radius R as below P def= IBpR ( µ0 ) def = { µ ∈M ( IRD ) : Wp ( µ , µ0 ) ≤ R } , ( 9 ) where µ0 is a user-defined base distribution , and To establish the proof , we consider the Wp ( · , · ) is the p-Wasserstein distance defined as below Wp ( µ1 , µ2 ) def = ( inf π∈Π ( µ1 , µ2 ) ∫ IRD×IRD ‖ξ1 − ξ2‖p2dπ ( ξ1 , ξ2 ) ) 1 p , ( 10 ) iii where the infimum is taken with respect to all couplings π of the measures µ , µ0 ∈ M ( IRD ) , and Π ( µ , µ0 ) is the set of all such couplings with the marginals µ and µ0 . The kernel MMD loss function in equation 8 is defined with respect to the unknown distributions of the data-set PV and the model PW . Therefore , we construct an unbiased estimator for the MMD loss function in equation 8 based on the training samples . To describe the estimator , sample the labels from a uniform distribution y1 , · · · , yn ∼i.i.d . Uniform { −1 , +1 } , where we assume that the number of positive and negative labels are balanced . In particular , consider the set of positive labels I = { i ∈ { 1 , 2 , · · · , n } : yi = +1 } , and negative labels J = { 1 , 2 , · · · , n } /I , where their cardinality is |I| = |J | = n2 . We consider the following assignment of labels : • Positive class labels : If yi = +1 , sample the corresponding feature map from data-distribution xi = vi ∼ PV . • Negative class labels : If yi = −1 , sample from the corresponding feature map from the generated distribution xi = G ( Zi , W ) ∼ PW , Zi ∼ PZ . By this construction , the joint distribution of features and labels PY , X has the marginals PX|Y=+1 = PV , and PX|Y=−1 = PW . Moreover , the following statistic , known as the kernel alignment in the literature ( see , e.g. , Sinha & Duchi ( 2016 ) ; Cortes et al . ( 2012 ) ) , is an unbiased estimator of the MMD loss in equation 8 , min W∈W sup µ∈P M̂MDµ [ PV , PW ] def = 8 n ( n− 1 ) ∑ 1≤i < j≤n yiyjKµ ( xi , xj ) . ( 11 ) See Appendix C.1 for the related proof . The kernel alignment in equation 11 can also be viewed through the lens of the risk minimization min W∈W inf µ∈P M̂MD α µ [ PV , PW ] def = 8 n ( n− 1 ) α ∑ 1≤i < j≤n ( αyiyj −Kµ ( xi , xj ) ) 2 ( 12a ) = 8 n ( n− 1 ) α ∑ 1≤i < j≤n ( αyiyj − IEµ [ ϕ ( xi ; ξ ) ϕ ( xj ; ξ ) ] ) 2 . ( 12b ) Here , α > 0 is a scaling factor that determines the separation between feature vectors , and K∗ def = αyyT is the ideal kernel that provides the maximal separation between the feature vectors over the training data-set , i.e. , K∗ ( xi , xj ) = α when features have identical labels yi = yj , and K∗ ( xi , xj ) = −α otherwise . Upon expansion of the risk function in equation 12 , it can be easily shown that it reduces to the kernel alignment in equation 11 when α → +∞ . Intuitively , the risk minimization in equation 12 gives a feature space in which pairwise distances are similar to those in the output space Y = { −1 , +1 } .

Summary: This paper proposes to learn a kernel for training MMD-GAN by optimizing over the probability distribution that defines the kernel by means of random features. This is unlike the usual setting of MMD-GAN where the kernel is parametrized by composing a fixed top-kernel with a discriminator network that is optimized during training. The main motivation for this approach is to avoid having to 'manually' fix some parameters of the top-level kernel like the bandwidth of the RBF kernel. They provide an algorithm to achieve such optimization in probability space along with some consistency results and perform experiments on MNIST and Cifar10 to demonstrate empirically the advantage of such an approach over those that fix the top-level kernel.