Separation Results between Fixed-Kernel and Feature-Learning Probability Metrics

Several works in implicit and explicit generative modeling empirically observed that feature-learning discriminators outperform fixed-kernel discriminators in terms of the sample quality of the models . We provide separation results between probability metrics with fixed-kernel and feature-learning discriminators using the function classes F2 and F1 respectively , which were developed to study overparametrized two-layer neural networks . In particular , we construct pairs of distributions over hyper-spheres that can not be discriminated by fixed kernel ( F2 ) integral probability metric ( IPM ) and Stein discrepancy ( SD ) in high dimensions , but that can be discriminated by their feature learning ( F1 ) counterparts . To further study the separation we provide links between the F1 and F2 IPMs with sliced Wasserstein distances . Our work suggests that fixed-kernel discriminators perform worse than their feature learning counterparts because their corresponding metrics are weaker . 1 Introduction . The field of generative modeling , whose aim is to generate artificial samples of some target distribution given some true samples of it , is broadly divided into two types of models ( Mohamed and Lakshminarayanan , 2017 ) : explicit generative models , which involve learning an estimate of the log-density of the target distribution which is then sampled ( e.g . energy-based models ) , and implicit generative models , where samples are generated directly by transforming some latent variable ( e.g . generative adversarial networks ( Goodfellow et al. , 2014 ) , normalizing flows ( Rezende and Mohamed , 2015 ) ) . Several works have observed experimentally that both in implicit and in explicit generative models , using ‘ adaptive ’ or ‘ feature-learning ’ function classes as discriminators yields better generative performance than ‘ lazy ’ or ‘ kernel ’ function classes . Within implicit models , Li et al . ( 2017 ) show that generative moment matching networks ( GMMN ) generate significantly better samples for the CIFAR-10 and MNIST datasets when using maximum mean discrepancy ( MMD ) with learned instead of fixed features . For a related method and in a similar spirit , Santos et al . ( 2019 ) show that for image generation , fixed-feature discriminators are only successful when we take an amount of features exponential in the intrinsic dimension of the dataset . Genevay et al . ( 2018 ) study implicit generative models with the Sinkhorn divergence as discriminator , and they also show that other than for simple datasets like MNIST , learning the Wasserstein cost is crucial for good performance . As to explicit models , Grathwohl et al . ( 2020 ) train energy-based models with a Stein discrepancy based on neural networks and show improved performance with respect to kernel classes . Chang et al . ( 2020 ) show that Stein variational gradient descent ( SVGD ) fails in high dimensions , and that learning the kernel helps . Given the abundant experimental evidence , the aim of this work is to provide some theoretical results that showcase the advantages of feature-learning over kernel discriminators . For 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the sake of simplicity , we compare the discriminative behavior of two function classes F1 and F2 , arising from infinite-width two-layer neural networks with different norms penalties on its weights ( Bach , 2017 ) . F1 displays an adaptive behavior , while F2 is an RKHS which consequently has a lazy behavior . Namely , our main contributions are : ( i ) We construct a sequence of pairs of distributions over hyperspheres Sd−1 of increasing dimensions , such that the F2 integral probability metric ( IPM ) between the pair decreases exponentially in the dimension , while the F1 IPM remains high . ( ii ) We construct a sequence of pairs of distributions over Sd−1 such that the F2 Stein discrepancy ( SD ) between the pair decreases exponentially in the dimension , while the F1 SD remains high . ( iii ) We prove polynomial upper and lower bounds between theF1 IPM and the max-sliced Wasserstein distance for distributions over Euclidean balls . For a class F̃2 related to F2 , we prove similar upper and lower bounds between the F̃2 IPM and the sliced Wasserstein distance for distributions over Euclidean balls . Our findings reinforce the idea that generative models with kernel discriminators have worse performance because their corresponding metrics are weaker and thus unable to distinguish between different distributions , especially in high dimensions . 2 Related work . A recent line of research has studied the question of how neural networks compare to kernel methods , with a focus on supervised learning problems . Bach ( 2017 ) shows the approximation benefits of the F1 space for adapting to low-dimensional structures compared to the ( kernel ) space F2 ; an analysis that we leverage . The function space F1 was also studied by Ongie et al . ( 2019 ) ; Savarese et al . ( 2019 ) ; Williams et al . ( 2019 ) , which focus on the ReLU activation function . More recently , several works showed that wide neural networks trained with gradient methods may behave like kernel methods in certain regimes ( see , e.g. , Jacot et al. , 2018 ) . Examples of works that compare ‘ active/feature-learning ’ and ‘ kernel/lazy ’ regimes for supervised learning include Chizat and Bach ( 2020 ) ; Ghorbani et al . ( 2019 ) ; Wei et al . ( 2020 ) ; Woodworth et al . ( 2020 ) , and Domingo-Enrich et al . ( 2021 ) for energy-based models . We are not aware of any works that study how feature-learning function classes and kernel classes differ as discriminators for IPMs or Stein discrepancies . It turns out that the F2 integral probability metric that we study is in fact MMD for certain kernels that often admit a closed form ( Roux and Bengio , 2007 ; Cho and Saul , 2009 ; Bach , 2017 ) . MMDs are probability metrics that were first introduced by Gretton et al . ( 2007 , 2012 ) for kernel twosample tests , and that have enjoyed ample success with the advent of deep-learning-based generative modeling as discriminating metrics : Li et al . ( 2015 ) and Dziugaite et al . ( 2015 ) introduced GMMN , which differ from GANs in that the discriminator network is replaced by a fixed-kernel MMD . Li et al . ( 2017 ) introduces an improvement on GMMN by using the MMD loss on learned features . From this viewpoint , our separation results in Sec . 5 can be interpreted as instances in which a given fixed-kernel MMD provably has less discriminative power than adaptive discriminators . Other related work includes the Stein discrepancy literature . Stein ’ s method ( Stein , 1972 ) dates to the 1970s . Gorham and Mackey ( 2015 ) introduced a computational approach to compute the Stein discrepancy in order to assess sample quality . Later , Chwialkowski et al . ( 2016 ) , Liu et al . ( 2016 ) and Gorham and Mackey ( 2017 ) introduced the more practical kernelized Stein discrepancy ( KSD ) for goodness-of-fit tests . Liu and Wang ( 2016 ) introduced SVGD , the first method to use the KSD to obtain samples from a distribution . Barp et al . ( 2019 ) employed KSD to train parametric generative models , and Grathwohl et al . ( 2020 ) trained models replacing KSD by a neural-network-based SD . Our work also touches on sliced and spiked Wasserstein distances . Sliced Wasserstein distances were introduced first by Kolouri et al . ( 2016 ) ; Kolouri et al . ( 2019 ) . Spiked Wasserstein distances , which are a generalization , were studied later by Paty and Cuturi ( 2019 ) , and they also appear in Niles-Weed and Rigollet ( 2019 ) as a good statistical estimator . Nadjahi et al . ( 2020 ) and Lin et al . ( 2021 ) have studied statistical properties of sliced and spiked Wasserstein distances , respectively . 3 Framework . 3.1 Notation . If V is a normed vector space , we use BV ( β ) to denote the closed ball of V of radius β , and BV : = BV ( 1 ) for the unit ball . If K denotes a subset of the Euclidean space , P ( K ) is the set of Borel probability measures , M ( K ) is the space of finite signed Radon measures andM+ ( K ) is the set of finite positive Radon measures . If γ is a signed Radon measure over K , then ‖γ‖TV is the total variation ( TV ) norm of γ . Throughout the paper , and unless otherwise specified , σ : R→ R denotes a generic non-linear activation function . We use ( · ) + : R→ R to denote the ReLu activation , defined as ( x ) + = max { x , 0 } . τ denotes the uniform probability measure over a space that depends on the context . We use Sd for the d-dimensional hypersphere and log for the natural logarithm . 3.2 Overparametrized two-layer neural network spaces . Feature-learning regime . We define F1 as the Banach space of functions f : K → R such that for some γ ∈ M ( Sd ) , for all x ∈ K we have f ( x ) = ∫ Sd σ ( 〈θ , x〉 ) dγ ( θ ) for some signed Radon measure γ ( Bach , 2017 ) . The norm of F1 is defined as ‖f‖F1 = inf { ‖γ‖TV |f ( · ) = ∫ Sd σ ( 〈θ , ·〉 ) dγ ( θ ) } . Kernel regime . We define F2 as the ( reproducing kernel ) Hilbert space of functions f : K → R such that for some absolutely continuous ρ ∈ M ( Sd ) with dρdτ ∈ L 2 ( Sd ) ( where τ is the uniform probability measure over Sd ) , we have that for all x ∈ K , f ( x ) = ∫ Sd σ ( 〈θ , x〉 ) dρ ( θ ) . The norm of F2 is defined as ‖f‖2F2 = inf { ∫ Sd h ( θ ) 2 dτ ( θ ) | f ( · ) = ∫ Sd σ ( 〈θ , ·〉 ) h ( θ ) dτ ( θ ) } . As an RKHS , the kernel of F2 is k ( x , y ) = ∫ Sd σ ( 〈x , θ〉 ) σ ( 〈y , θ〉 ) dτ ( θ ) . ( 1 ) Such kernels admit closed form expressions for different choices of activation functions , among which ReLu ( Roux and Bengio , 2007 ; Cho and Saul , 2009 ; Bach , 2017 ) . Remark that since ∫ |h ( θ ) |dτ ( θ ) ≤ ( ∫ h ( θ ) 2 dτ ( θ ) ) 1/2 by the Cauchy-Schwarz inequality , we have F2 ⊂ F1 . In particular , when σ is the ReLu unit , Bach ( 2017 ) shows that two-layer networks with a single neuron belong to F1 but not to F2 , and their L2 approximations in F2 have exponentially high norm in the dimension . Informally , one should understand F1 as the space of two-layer networks where both the input layer and output layer parameters are trained , in the limit of an infinite number of neurons . On the other hand , F2 is the space of infinite-width two-layer networks where only the output layer parameters are trained while the input layer parameters are sampled uniformly on the sphere and kept fixed .

Summary: This paper argues that discriminators based on learned features have significantly more discrimination power than discriminators based on fixed features. It does so by defining two function classes that can be interpreted as infinite width one hidden layer neural networks, one corresponding to the learned, or adaptive regime (F_1), and another corresponding to the fixed, or kernel regime (F_2). Various interesting separation and characterization results are shown between F_1 and F_2, using practically relevant metrics.