Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

Several works in implicit and explicit generative modeling empirically observed that feature-learning discriminators outperform fixed-kernel discriminators in terms of the sample quality of the models . We provide separation results between probability metrics with fixed-kernel and feature-learning discriminators using the function classes F2 and F1 respectively , which were developed to study overparametrized two-layer neural networks . In particular , we construct pairs of distributions over hyper-spheres that can not be discriminated by fixed kernel ( F2 ) integral probability metric ( IPM ) and Stein discrepancy ( SD ) in high dimensions , but that can be discriminated by their feature learning ( F1 ) counterparts . To further study the separation we provide links between the F1 and F2 IPMs with sliced Wasserstein distances . Our work suggests that fixed-kernel discriminators perform worse than their feature learning counterparts because their corresponding metrics are weaker . 1 Introduction . The field of generative modeling , whose aim is to generate artificial samples of some target distribution given some true samples of it , is broadly divided into two types of models ( Mohamed and Lakshminarayanan , 2017 ) : explicit generative models , which involve learning an estimate of the log-density of the target distribution which is then sampled ( e.g . energy-based models ) , and implicit generative models , where samples are generated directly by transforming some latent variable ( e.g . generative adversarial networks ( Goodfellow et al. , 2014 ) , normalizing flows ( Rezende and Mohamed , 2015 ) ) . Several works have observed experimentally that both in implicit and in explicit generative models , using ‘ adaptive ’ or ‘ feature-learning ’ function classes as discriminators yields better generative performance than ‘ lazy ’ or ‘ kernel ’ function classes . Within implicit models , Li et al . ( 2017 ) show that generative moment matching networks ( GMMN ) generate significantly better samples for the CIFAR-10 and MNIST datasets when using maximum mean discrepancy ( MMD ) with learned instead of fixed features . For a related method and in a similar spirit , Santos et al . ( 2019 ) show that for image generation , fixed-feature discriminators are only successful when we take an amount of features exponential in the intrinsic dimension of the dataset . Genevay et al . ( 2018 ) study implicit generative models with the Sinkhorn divergence as discriminator , and they also show that other than for simple datasets like MNIST , learning the Wasserstein cost is crucial for good performance . As to explicit models , Grathwohl et al . ( 2020 ) train energy-based models with a Stein discrepancy based on neural networks and show improved performance with respect to kernel classes . Chang et al . ( 2020 ) show that Stein variational gradient descent ( SVGD ) fails in high dimensions , and that learning the kernel helps . Given the abundant experimental evidence , the aim of this work is to provide some theoretical results that showcase the advantages of feature-learning over kernel discriminators . For 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the sake of simplicity , we compare the discriminative behavior of two function classes F1 and F2 , arising from infinite-width two-layer neural networks with different norms penalties on its weights ( Bach , 2017 ) . F1 displays an adaptive behavior , while F2 is an RKHS which consequently has a lazy behavior . Namely , our main contributions are : ( i ) We construct a sequence of pairs of distributions over hyperspheres Sd−1 of increasing dimensions , such that the F2 integral probability metric ( IPM ) between the pair decreases exponentially in the dimension , while the F1 IPM remains high . ( ii ) We construct a sequence of pairs of distributions over Sd−1 such that the F2 Stein discrepancy ( SD ) between the pair decreases exponentially in the dimension , while the F1 SD remains high . ( iii ) We prove polynomial upper and lower bounds between theF1 IPM and the max-sliced Wasserstein distance for distributions over Euclidean balls . For a class F̃2 related to F2 , we prove similar upper and lower bounds between the F̃2 IPM and the sliced Wasserstein distance for distributions over Euclidean balls . Our findings reinforce the idea that generative models with kernel discriminators have worse performance because their corresponding metrics are weaker and thus unable to distinguish between different distributions , especially in high dimensions . 2 Related work . A recent line of research has studied the question of how neural networks compare to kernel methods , with a focus on supervised learning problems . Bach ( 2017 ) shows the approximation benefits of the F1 space for adapting to low-dimensional structures compared to the ( kernel ) space F2 ; an analysis that we leverage . The function space F1 was also studied by Ongie et al . ( 2019 ) ; Savarese et al . ( 2019 ) ; Williams et al . ( 2019 ) , which focus on the ReLU activation function . More recently , several works showed that wide neural networks trained with gradient methods may behave like kernel methods in certain regimes ( see , e.g. , Jacot et al. , 2018 ) . Examples of works that compare ‘ active/feature-learning ’ and ‘ kernel/lazy ’ regimes for supervised learning include Chizat and Bach ( 2020 ) ; Ghorbani et al . ( 2019 ) ; Wei et al . ( 2020 ) ; Woodworth et al . ( 2020 ) , and Domingo-Enrich et al . ( 2021 ) for energy-based models . We are not aware of any works that study how feature-learning function classes and kernel classes differ as discriminators for IPMs or Stein discrepancies . It turns out that the F2 integral probability metric that we study is in fact MMD for certain kernels that often admit a closed form ( Roux and Bengio , 2007 ; Cho and Saul , 2009 ; Bach , 2017 ) . MMDs are probability metrics that were first introduced by Gretton et al . ( 2007 , 2012 ) for kernel twosample tests , and that have enjoyed ample success with the advent of deep-learning-based generative modeling as discriminating metrics : Li et al . ( 2015 ) and Dziugaite et al . ( 2015 ) introduced GMMN , which differ from GANs in that the discriminator network is replaced by a fixed-kernel MMD . Li et al . ( 2017 ) introduces an improvement on GMMN by using the MMD loss on learned features . From this viewpoint , our separation results in Sec . 5 can be interpreted as instances in which a given fixed-kernel MMD provably has less discriminative power than adaptive discriminators . Other related work includes the Stein discrepancy literature . Stein ’ s method ( Stein , 1972 ) dates to the 1970s . Gorham and Mackey ( 2015 ) introduced a computational approach to compute the Stein discrepancy in order to assess sample quality . Later , Chwialkowski et al . ( 2016 ) , Liu et al . ( 2016 ) and Gorham and Mackey ( 2017 ) introduced the more practical kernelized Stein discrepancy ( KSD ) for goodness-of-fit tests . Liu and Wang ( 2016 ) introduced SVGD , the first method to use the KSD to obtain samples from a distribution . Barp et al . ( 2019 ) employed KSD to train parametric generative models , and Grathwohl et al . ( 2020 ) trained models replacing KSD by a neural-network-based SD . Our work also touches on sliced and spiked Wasserstein distances . Sliced Wasserstein distances were introduced first by Kolouri et al . ( 2016 ) ; Kolouri et al . ( 2019 ) . Spiked Wasserstein distances , which are a generalization , were studied later by Paty and Cuturi ( 2019 ) , and they also appear in Niles-Weed and Rigollet ( 2019 ) as a good statistical estimator . Nadjahi et al . ( 2020 ) and Lin et al . ( 2021 ) have studied statistical properties of sliced and spiked Wasserstein distances , respectively . 3 Framework . 3.1 Notation . If V is a normed vector space , we use BV ( β ) to denote the closed ball of V of radius β , and BV : = BV ( 1 ) for the unit ball . If K denotes a subset of the Euclidean space , P ( K ) is the set of Borel probability measures , M ( K ) is the space of finite signed Radon measures andM+ ( K ) is the set of finite positive Radon measures . If γ is a signed Radon measure over K , then ‖γ‖TV is the total variation ( TV ) norm of γ . Throughout the paper , and unless otherwise specified , σ : R→ R denotes a generic non-linear activation function . We use ( · ) + : R→ R to denote the ReLu activation , defined as ( x ) + = max { x , 0 } . τ denotes the uniform probability measure over a space that depends on the context . We use Sd for the d-dimensional hypersphere and log for the natural logarithm . 3.2 Overparametrized two-layer neural network spaces . Feature-learning regime . We define F1 as the Banach space of functions f : K → R such that for some γ ∈ M ( Sd ) , for all x ∈ K we have f ( x ) = ∫ Sd σ ( 〈θ , x〉 ) dγ ( θ ) for some signed Radon measure γ ( Bach , 2017 ) . The norm of F1 is defined as ‖f‖F1 = inf { ‖γ‖TV |f ( · ) = ∫ Sd σ ( 〈θ , ·〉 ) dγ ( θ ) } . Kernel regime . We define F2 as the ( reproducing kernel ) Hilbert space of functions f : K → R such that for some absolutely continuous ρ ∈ M ( Sd ) with dρdτ ∈ L 2 ( Sd ) ( where τ is the uniform probability measure over Sd ) , we have that for all x ∈ K , f ( x ) = ∫ Sd σ ( 〈θ , x〉 ) dρ ( θ ) . The norm of F2 is defined as ‖f‖2F2 = inf { ∫ Sd h ( θ ) 2 dτ ( θ ) | f ( · ) = ∫ Sd σ ( 〈θ , ·〉 ) h ( θ ) dτ ( θ ) } . As an RKHS , the kernel of F2 is k ( x , y ) = ∫ Sd σ ( 〈x , θ〉 ) σ ( 〈y , θ〉 ) dτ ( θ ) . ( 1 ) Such kernels admit closed form expressions for different choices of activation functions , among which ReLu ( Roux and Bengio , 2007 ; Cho and Saul , 2009 ; Bach , 2017 ) . Remark that since ∫ |h ( θ ) |dτ ( θ ) ≤ ( ∫ h ( θ ) 2 dτ ( θ ) ) 1/2 by the Cauchy-Schwarz inequality , we have F2 ⊂ F1 . In particular , when σ is the ReLu unit , Bach ( 2017 ) shows that two-layer networks with a single neuron belong to F1 but not to F2 , and their L2 approximations in F2 have exponentially high norm in the dimension . Informally , one should understand F1 as the space of two-layer networks where both the input layer and output layer parameters are trained , in the limit of an infinite number of neurons . On the other hand , F2 is the space of infinite-width two-layer networks where only the output layer parameters are trained while the input layer parameters are sampled uniformly on the sphere and kept fixed .

Summary: This paper provides a theoretical justification for why feature learning (theoretical proxy for that is the F_1 space) may be crucial as opposed to learning with fixed features (theoretical proxy for that is the RKHS with NTK or F_2 space) when using statistical pseudometrics that are computed using the sup over function spaces (e.g. IPM pseudometrics and Stein discrepancy) to train a generative model to match a given training distribution. In particular, they construct sequences of pairs of distributions for which they can prove that as d increases, F_1 pseudometrics yield much higher “distance” between the distributions than F_2 pseudometrics. In particular, the mathematical exercise suggests that distributions that differ in high frequency components benefit more from feature learning.