Title: Provable benefits of score matching
Venue: NeurIPS 2023
Accepted: yes
Keywords: theory

score matching

exponential families

sample complexity

computational hardness
Abstract: Score matching is an alternative to maximum likelihood (ML) for estimating a probability distribution parametrized up to a constant of proportionality. By fitting the ''score'' of the distribution, it sidesteps the need to compute this constant of proportionality (which is often intractable).
While score matching and variants thereof are popular in practice, precise theoretical understanding of the benefits and tradeoffs with maximum likelihood---both computational and statistical---are not well understood. In this work, we give the first example of a natural exponential family of distributions such that the score matching loss is computationally efficient to optimize, and has a comparable statistical efficiency to ML, while the ML loss is intractable to optimize using a gradient-based method. The family consists of exponentials of polynomials of fixed degree, and our result can be viewed as a continuous analogue of recent developments in the discrete setting. Precisely, we show: (1) Designing a zeroth-order or first-order oracle for optimizing the maximum likelihood loss is NP-hard. (2) Maximum likelihood has a statistical efficiency polynomial in the ambient dimension and the radius of the parameters of the family. (3) 
Minimizing the score matching loss is both computationally and statistically efficient, with complexity polynomial in the ambient dimension.

Provable benefits of score matching

Abstract

Score matching is an alternative to maximum likelihood (ML) for estimating a probability distribution parametrized up to a constant of proportionality. By fitting the 'score' of the distribution, it sidesteps the need to compute this constant of proportionality (which is often intractable). While score matching and variants thereof are popular in practice, precise theoretical understanding of the benefits and tradeoffs with maximum likelihood-both computational and statistical-are not well understood. In this work, we give the first example of a natural exponential family of distributions such that the score matching loss is computationally efficient to optimize, and has a comparable statistical efficiency to ML, while the ML loss is intractable to optimize using a gradient-based method. The family consists of exponentials of polynomials of fixed degree, and our result can be viewed as a continuous analogue of recent developments in the discrete setting. Precisely, we show: (1) Designing a zeroth-order or first-order oracle for optimizing the maximum likelihood loss is NP-hard. (2) Maximum likelihood has a statistical efficiency polynomial in the ambient dimension and the radius of the parameters of the family. (3) Minimizing the score matching loss is both computationally and statistically efficient, with complexity polynomial in the ambient dimension.

1 Introduction

Energy-based models are a flexible class of probabilistic models with wide-ranging applications. They are parameterized by a class of energies E θ ( x ) which in turn determines the distribution

up to a constant of proportionality Z θ that is called the partition function. One of the major challenges of working with energy-based models is designing efficient algorithms for fitting them to data. Statistical theory tells us that the maximum likelihood estimator (MLE)-i.e., the parameters θ which maximize the likelihood-enjoys good statistical properties including consistency and asymptotic efficiency.

However, there is a major computational impediment to computing the MLE: Both evaluating the log-likelihood and computing its gradient with respect to θ (i.e., implementing zeroth and first order oracles, respectively) seem to require computing the partition function, which is often computationally intractable. More precisely, the gradient of the negative log-likelihood depends on ∇ θ log Z θ = E p θ [ ∇ θ E θ ( x )] . A popular approach is to estimate this quantity by using a Markov chain to approximately sample from p θ . However in high-dimensional settings, Markov chains often require many, sometimes even exponentially many, steps to mix.

Score matching (Hyvärinen, 2005) is a popular alternative that sidesteps needing to compute the partition function of sample from p θ . The idea is to fit the score of the distribution, in the sense that

we want θ such that ∇ x log p ( x ) matches ∇ x log p θ ( x ) for a typical sample from p . This approach turns out to have many nice properties. It is consistent in the sense that minimizing the objective function yields provably good estimates for the unknown parameters. Moreover, while the definition depends on the unknown ∇ x log p ( x ) , by applying integration by parts, it is possible to transform the objective into an equivalent one that can be estimated from samples.

The main question is to bound its statistical performance, especially relative to that of the maximum likelihood estimator. Recent work by Koehler et al. (2022) showed that the cost can be quite steep. They gave explicit examples of distributions that have bad isoperimetric properties (i.e., large Poincaré constant) and showed how such properties can cause poor statistical performance.

Despite wide usage, there is little rigorous understanding of when score matching helps . This amounts to finding a general setting where maximizing the likelihood with standard first-order optimization is provably hard, and yet score matching is both computationally and statistically efficient, with only a polynomial loss in sample complexity relative to the MLE. In this work, we show the first such guarantees, and we do so for a natural class of exponential families defined by polynomials. As we discuss in Section 1.1, our results parallel recent developments in learning graphical models-where it is known that pseudolikelihood methods allow efficient learning of distributions that are hard to sample from-and can be viewed as a continuous analogue of such results.

In general, an exponential family on R n has the form p θ ( x ) ∝ h ( x ) exp( ⟨ θ, T ( x ) ⟩ ) where h ( x ) is the base measure , θ is the parameter vector , and T ( x ) is the vector of sufficient statistics . Exponential families are one of the most classic parametric families of distributions, dating back to works by Darmois (1935), Koopman (1936) and Pitman (1936). They have a number of natural properties, including: (1) The parameters θ are uniquely determined by the expectation of the sufficient statistics E p θ [ T ] ; (2) The distribution p θ is the maximum entropy distribution, subject to having given values for E p θ [ T ] ; (3) They have conjugate priors (Brown, 1986), which allow characterizations of the family for the posterior of the parameters given data.

For any (odd positive integer) constant d and norm bound B ≥ 1 , we study a natural exponential family P n,d,B on R n where

The sufficient statistics T ( x ) ∈ R M -1 consist of all monomials in x 1 , . . . , x n of degree at least 1 and at most d ( where M = ( n + d d ) ) .

The base measure is defined as h ( x ) = exp( -∑ n i =1 x d +1 i ) . 1

The parameters θ lie in an l ∞ -ball: θ ∈ Θ B = { θ ∈ R M -1 : ∥ θ ∥ ∞ ≤ B } .

Towards stating our main results, we formally define the maximum likelihood and score matching objectives, denoting by ˆ E the empirical average over the training samples drawn from some p ∈ P n,d,B :

where K p is a constant depending only on p and (1) follows by integration by parts (Hyvärinen, 2005). In the special case of exponential families, (1) is a quadratic, and in fact the optimum can be written in closed form:

where ( JT ) x : ( M -1) × n is the Jacobian of T at the point x , ∆ f = ∑ i ∂ 2 i f is the Laplacian, applied coordinate wise to the vector-valued function f .

With this setting in place, we show the following intractability result.

Theorem 1.1 (Informal, computational lower bound) . Unless RP = NP , there is no poly( n, N ) -time algorithm that evaluates L MLE ( θ ) and ∇ L MLE ( θ ) given θ ∈ Θ B and arbitrary samples x 1 , . . . , x N ∈ R n , for d = 7 , B = poly( n ) . Thus, optimizing the MLE loss using a zeroth-order or first-order method is computationally intractable.

The main idea of the proof is to construct a polynomial F C ( x ) which has roots exactly at the satisfying assignments of a given 3SAT formula C . We then argue that exp( -γF C ( x )) , for sufficiently large γ > 0 , concentrates near the satisfying assignments. Finally, we show sampling from this distribution or approximating log Z θ or ∇ θ log Z θ (where θ ∈ R M -1 is the parameter vector corresponding to the polynomial -γF C ( x ) ) would enable efficiently finding a satisfying assignment.

Our next result shows that MLE, though computationally intractable to compute via implementing zeroth or first order oracles, has (asymptotic) sample complexity poly( n, B ) (for constant d ).

Theorem 1.2 (Informal, efficiency of MLE) . The MLE estimator ˆ θ MLE = arg max θ L MLE ( θ ) has asymptotic sample complexity polynomial in n . That is, for all sufficiently large N it holds with probability at least 0 . 99 (over N samples drawn from p θ ∗ ) that:

The main proof technique for this is an anticoncentration bound of low-degree polynomials, for distributions in our exponential family.

Lastly, we prove that score matching also has polynomial (asymptotic) statistical complexity.

Theorem 1.3 (Informal, efficiency of SM) . The score matching estimator ˆ θ SM = arg max θ L SM ( θ ) also has asymptotic sample complexity at most polynomial in n . That is, for all sufficiently large N it holds with probability at least 0 . 99 (over N samples drawn from p θ ∗ ) that:

The main ingredient in this result is a bound on the restricted Poincaré constant -namely, the Poincaré constant, when restricted to functions that are linear in the sufficient statistics T . We bound this quantity for the exponential family we consider in terms of the condition number of the Fisher matrix of the distribution, which we believe is a result of independent interest. With this tool in hand, we can use the framework of Koehler et al. (2022), which relates the asymptotic sample complexity of score matching to the asymptotic sample complexity of maximum likelihood, in terms of the restricted Poincaré constant of the distribution.

1.1 Discussion and related work

Score matching: Score matching was proposed by Hyvärinen (2005), who also gave conditions under which it is consistent and asymptotically normal. Asymptotic normality is also proven for various kernelized variants of score matching in Barp et al. (2019). Koehler et al. (2022) prove that the statistical sample complexity of score matching is not much worse than the sample complexity of maximum likelihood when the distribution satisfies a (restricted) Poincaré inequality. While we leverage machinery from Koehler et al. (2022), their work only bounds the sample complexity of score matching by a quantity polynomial in the ambient dimension for a specific distribution in a specific bimodal exponential family. By contrast, we can handle an entire class of exponential families with low-degree sufficient statistics.

Poincaré vs Restricted Poincaré: We note that while Poincaré inequalities are directly related to isoperimetry and mixing of Markov chains, sample efficiency of score matching only depends on the Poincaré inequality holding for a restricted class of functions, namely, functions linear in the sufficient statistics. Hence, hardness of sampling only implies sample complexity lower bounds in cases where the family is expressive enough-indeed, the key to exponential lower bounds for score matching in Koehler et al. (2022) is augmenting the sufficient statistics with a function defined by a bad cut. This gap means that we can hope to have good sample complexity for score matching even in cases where sampling is hard-which we take advantage of in this work.

Learning exponential families: Despite the fact that exponential families are both classical and ubiquitous, both in statistics and machine learning, there is relatively little understanding about the computational-statistical tradeoffs to learn them from data, that is, what sample complexity can be achieved with a computationally efficient algorithm. Ren et al. (2021) consider a version of the

'interaction screening' estimator, a close relative of pseudolikelihood, but do not prove anything about the statistical complexity of this estimator. Shah et al. (2021) consider a related estimator, and analyze it under various low-rank and sparsity assumptions of reshapings of the sufficient statistics into a tensor. Unfortunately, these assumptions are somewhat involved, and it's unclear if they are needed for designing computationally and statistically efficient algorithms.

Discrete exponential families (Ising models): Ising models have the form p J ( x ) ∝ exp ( ∑ i ∼ j J ij x i x j + ∑ i J i x i ) where ∼ denotes adjacency in some (unknown) graph, and J ij , J i denote the corresponding pairwise and singleton potentials. Bresler (2015) gave an efficient algorithm for learning any Ising model over a graph with constant degree (and l ∞ -bounds on the coefficients); see also the more recent work (Dagan et al., 2021). In contrast, it is a classic result (Arora and Barak, 2009) that approximating the partition function of members in this family is NP-hard.

Similarly, the exponential family we consider is such that it contains members for which sampling and approximating their partition function is intractable (the main ingredient in the proof of Theorem 1.1). Nevertheless, by Theorem 3, we can learn the parameters for members in this family computationally efficiently, and with sample complexity comparable to the optimal one (achieved by maximum likelihood). This also parallels other developments in Ising models (Bresler et al., 2014; Montanari, 2015), where it is known that restricting the type of learning algorithm (e.g., requiring it to work with sufficient statistics only) can make a tractable problem become intractable.

The parallels can be drawn even on an algorithmic level: a follow up work to Bresler (2015) by Vuffray et al. (2016) showed that similar results can be shown in the Ising model setting by using the 'screening estimator', a close relative of the classical pseudolikelihood estimator (Besag, 1977) which tries to learn a distribution by matching the conditional probability of singletons, and thereby avoids having to evaluate a partition function. Since conditional probabilities of singletons capture changes in a single coordinate, they can be viewed as a kind of 'discrete gradient'-a further analogy to score matching in the continuous setting. 2

2 Preliminaries

We consider the following exponential family. Fix positive integers n, d, B ∈ N where d is odd. Let h ( x ) = exp( -∑ n i =1 x d +1 i ) , and let T ( x ) ∈ R M -1 be the vector of monomials in x 1 , . . . , x n of degree at least 1 and at most d (so that M = ( n + d d ) ). Define Θ ⊆ R M -1 by Θ = { θ ∈ R M -1 : ∥ θ ∥ ∞ ≤ B } . For any θ ∈ Θ define p θ : R n → [0 , ∞ ) by

where Z θ = ∫ R n h ( x ) exp( ⟨ θ, T ( x ) ⟩ ) dx is the normalizing constant. Then we consider the family P n,d,B := ( p θ ) θ ∈ Θ B . Throughout, we will assume that B ≥ 1 .

Polynomial notation: Let R [ x 1 , . . . , x n ] ≤ d denote the space of polynomials in x 1 , . . . , x n of degree at most d . We can write any such polynomial f as f ( x ) = ∑ | d |≤ d a d x d where d denotes a degree function d : [ n ] → N , and | d | = ∑ n i =1 d ( i ) , and we write x d to denote ∏ n i =1 x d ( i ) i . Note that every d with 1 ≤ | d | ≤ d corresponds to an index of T , i.e. T ( x ) d = x d .

Let ∥·∥ mon denote the ℓ 2 norm of a polynomial in the monomial basis; that is, ∥ ∑ d a d x d ∥ mon = (∑ d a 2 d ) 1 / 2 . For any function f : R n → R , let ∥ f ∥ 2 L 2 ([ -1 , 1] n ) = E x ∼ Unif([ -1 , 1] n ) f ( x ) 2 .

Statistical efficiency of MLE: For any θ ∈ R M -1 , the Fisher information matrix of p θ with respect to the sufficient statistics T ( x ) is defined as

It is well-known that for any exponential family with no affine dependencies among the sufficient statistics (see e.g., Theorem 4.6 in Van der Vaart (2000)), it holds that for any θ ∗ ∈ R M -1 , given N

independent samples x (1) , . . . , x ( N ) ∼ p θ ∗ , the estimator ˆ θ MLE = ˆ θ MLE ( x (1) , . . . , x ( N ) ) satisfies

Statistical efficiency of score matching: Our analysis of the statistical efficiency of score matching is based on a result due to Koehler et al. (2022). We state a requisite definition followed by the result. Definition 2.1 (Restricted Poincaré for exponential families) . The restricted Poincaré constant of p ∈ P n,d,B is the smallest C P > 0 such that for all w ∈ R M -1 , it holds that

Theorem 2.2 (Koehler et al. (2022)) . Under certain regularity conditions (see Lemma B.4), for any p θ ∗ with restricted Poincaré constant C P and with λ min ( I ( θ ∗ )) > 0 , given N independent samples x (1) , . . . , x ( N ) ∼ p θ ∗ , the estimator ˆ θ SM = ˆ θ SM ( x (1) , . . . , x ( N ) ) satisfies where Γ satisfies

where ( JT )( x ) i = ∇ x T i ( x ) and ∆ T ( x ) = Tr ∇ 2 x T ( x ) .

3 Hardness of Implementing Optimization Oracles for P n, 7 , poly( n )

In this section we prove NP -hardness of implementing approximate zeroth-order and first-order optimization oracles for maximum likelihood in the exponential family P n, 7 ,Cn 2 log( n ) (for a sufficiently large constant C ) as defined in Section 2; we also show that approximate sampling from this family is NP -hard. See Theorems 3.4, 3.5, and A.5 respectively. All of the hardness results proceed by reduction from 3SAT and use the same construction.

The idea is that for any formula C on n variables, we can construct a non-negative polynomial F C of degree at most 6 in variables x 1 , . . . , x n , which has roots exactly at the points of the hypercube H := {-1 , 1 } n ⊆ R n that correspond to satisfying assignments (under the bijection that x i = 1 corresponds to True and x i = -1 corresponds to False). Intuitively, the distribution with density proportional to exp( -γF C ( x )) will, for sufficiently large γ > 0 , concentrate on the satisfying assignments. It is then straightforward to see that sampling from this distribution or efficiently computing either log Z θ or ∇ θ log Z θ (where θ ∈ R M -1 is the parameter vector corresponding to the polynomial -γF C ( x ) ) would enable efficiently finding a satisfying assignment.

The remainder of this section makes the above intuition precise; important details include (1) incorporating the base measure h ( x ) = exp( -∑ n i =1 x 8 i ) into the density function, and (2) showing that a polynomially-large temperature γ suffices.

Definition 3.1 (Clause/formula polynomials) . Given a 3 -clause formula of the form C = ˜ x i ∨ ˜ x j ∨ ˜ x k where ˜ x i = x i or ˜ x i = ¬ x i , we construct a polynomial H C ∈ R [ x 1 , . . . , x n ] ≤ 6 defined by

For example, if C = x 1 ∨ x 2 ∨ ¬ x 3 , then H C = ( x 1 -1) 2 ( x 2 -1) 2 ( x 3 +1) 2 . Further, given a 3 -SAT formula C = C 1 ∧ · · · ∧ C m on m clauses 3 , we define the polynomial

It can be seen that any x ∈ H corresponds to a satisfying assignment for C if and only if H C ( x ) = 0 . Note that there are possibly points outside H which satisfy H C ( x ) = 0 . To avoid these solutions, we introduce another polynomial:

where

Definition 3.2 (Hypercube polynomial) . We define G : R n → R by G ( x ) = ∑ n i =1 (1 -x 2 i ) 2 .

Note that G ( x ) ≥ 0 for all x , and the roots of G ( x ) are precisely the vertices of H . Therefore for any α, β > 0 , the roots (in R n ) of the polynomial F C ( x ) = αH C ( x ) + βG ( x ) are precisely the vertices of H that correspond to satisfying assignments for C .

Definition 3.3. Let C be a 3CNF formula with n variables and m clauses. Let α, β > 0 . Then we define a distribution P C ,α,β with density function

where Z C ,α,β = ∫ R n h ( x ) exp( -αH C ( x ) -βG ( x )) dx.

This distribution lies in the exponential family P n,d,B , for d = 7 and B = Ω( β + mα ) (Lemma A.2). Thus, if θ ( C , α, β ) is the parameter vector that induces P C ,α,β , then it suffices to show that (a) approximating log Z θ ( C ,α,β ) , (b) approximating ∇ θ log Z θ ( C ,α,β ) , and (c) sampling from P C ,α,β are NP -hard (under randomized reductions). We sketch the proofs below; details are in Appendix A.

Hardness of approximating log Z C ,α,β : In order to prove (a), we bound the mass of P C ,α,β in each orthant of R n . In particular, we show that for α = Ω( n ) and β = Ω( m log m ) , any orthant corresponding to a satisfying assignment has exponentially larger contribution to Z C ,α,β than any orthant corresponding to an unsatisfying assignment. A consequence is that the partition function Z C ,α,β is exponentially larger when the formula C is satisfiable than when it isn't (Lemma A.6).

But then approximating Z C ,α,β allows distinguishing a satisfiable formula from an unsatisfiable formula, which is NP -hard. This implies the following theorem (proof in Section A.2):

Theorem 3.4. Fix n ∈ N and let B ≥ Cn 2 for a sufficiently large constant C . Unless RP = NP , there is no poly( n ) -time algorithm which takes as input an arbitrary θ ∈ Θ B and outputs an approximation of log Z θ with additive error less than n log 1 . 16 .

Hardness of approximating ∇ θ log Z θ ( C ,α,β ) : Note that ∇ θ log Z θ = E x ∼ p θ [ T ( x )] , so in particular approximating the gradient yields an approximation to the mean E x ∼ p θ [ x ] . Since P C ,α,β is concentrated in orthants corresponding to satisfying assignments of C , we would intuitively expect that if C has exactly one satisfying assignment v ∗ , then sign( E p θ [ x ]) corresponds to this assignment. Formally, we show that if α = Θ( n ) and β = Ω( mn log m ) , then E x ∼ p C ,α,β [ v ∗ i x i ] ≥ 1 / 20 for all i ∈ [ n ] (Lemma A.7).

Since solving a formula with a unique satisfying assignment is still NP -hard, we get the following theorem (proof in Section A.3):

Theorem 3.5. Fix n ∈ N and let B ≥ Cn 2 log( n ) for a sufficiently large constant C . Unless RP = NP , there is no poly( n ) -time algorithm which takes as input an arbitrary θ ∈ Θ B and outputs an approximation of ∇ θ log Z θ with additive error (in an l ∞ sense) less than 1 / 20 .

With the above two theorems in hand, we are ready to present the formal version of Theorem 1.1; the proof is immediate from the definition of L MLE ( θ ) (see Section A.5).

Corollary 3.6. Fix n, N ∈ N and let B ≥ Cn 2 log n for a sufficiently large constant C . Unless RP = NP , there is no poly( n, N ) -time algorithm which takes as input an arbitrary θ ∈ Θ B , and an arbitrary sample x 1 , . . . , x N ∈ R n , and outputs an approximation of L MLE ( θ ) up to additive error of n log 1 . 16 , or ∇ θ L MLE ( θ ) up to an additive error of 1 / 20 .

Hardness of approximate sampling: We show that for α = Ω( n ) and β = Ω( m log m ) , the likelihood that x ∼ P C ,α,β lies in an orthant corresponding to a satisfying assignment for C is at least 1 / 2 (Lemma A.4). Hardness of approximate sampling follows immediately (Theorem A.5). Hence, although we show that score matching can efficiently estimate θ ∗ from samples produced by nature, knowing θ ∗ isn't enough to efficiently generate samples from the distribution.

4 Statistical Efficiency of Maximum Likelihood

In this section we prove Theorem 1.2 by showing that for any θ ∈ Θ B , we can lower bound the smallest eigenvalue of the Fisher information matrix I ( θ ) . Concretely, we show:

Theorem 4.1. For any θ ∈ Θ B , it holds that

As a corollary, given N samples from p θ , it holds as N →∞ that √ N ( ˆ θ MLE -θ ) → N (0 , Γ MLE ) where ∥ Γ MLE ∥ op ≤ ( nB ) O ( d 3 ) . Moreover, for sufficiently large N , with probability at least 0 . 99 it

Once we have the bound on λ min ( I ( θ )) , the first corollary follows from standard bounds for MLE (Section 2), and the second corollary follows from Markov's inequality (see e.g., Remark 4 in Koehler et al. (2022)). Lower-bounding λ min ( I ( θ )) itself requires lower-bounding the variance of any polynomial (with respect to p θ ) in terms of its coefficients. The proof consists of three parts. First, we show that the norm of a polynomial in the monomial basis is upper-bounded in terms of its L 2 norm on [ -1 , 1] n :

The key idea behind this proof is to work with the basis of (tensorized) Legendre polynomials, which is orthonormal with respect to the L 2 norm. Once we write the polynomial with respect to this basis, the L 2 norm equals the Euclidean norm of the coefficients. Given this observation, all that remains is to bound the coefficients after the change-of-basis. The complete proof is deferred to Appendix C.

Next, we show that if a polynomial f : R n → R has small variance with respect to p , then there is some box on which f has small variance with respect to the uniform distribution. This provides a way of comparing the variance of f with its L 2 norm (after an appropriate rescaling).

Lemma 4.3. Fix any θ ∈ Θ B and define p := p θ . Define R := 2 d +8 nBM . Then for any f ∈ R [ x 1 , . . . , x n ] ≤ d , there is some z ∈ R n with ∥ z ∥ ∞ ≤ R and some ϵ ≥ 1 / (2( d +1) MR d ( n + B )) such that

where ˜ U is the uniform distribution on { x ∈ R n : ∥ x -z ∥ ∞ ≤ ϵ } .

In order to prove this result, we pick a random box of radius ϵ (within a large bounding box of radius R ). In expectation, the variance on this box (with respect to p ) is not much less than Var p ( f ) . Moreover, for sufficiently small ϵ , the density function of p on this box has bounded fluctuations, allowing comparison of Var p ( f ) and Var ˜ U ( f ) . This argument is formalized in Appendix C.

Together, Lemma 4.2 and 4.3 allow us to lower bound the variance Var p ( f ) in terms of ∥ f ∥ mon .

Lemma 4.4. Fix any θ ∈ Θ B and define p := p θ . Define R := 2 d +8 nBM . Then for any f ∈ R [ x 1 , . . . , x n ] ≤ d with f (0) = 0 , it holds that

See Appendix C for the proof.We are now ready to finish the proof of Theorem 4.1.

Proof of Theorem 4.1 . Fix θ ∈ Θ B . Pick any w ∈ R M and define f ( x ) = ⟨ w,T ( x ) ⟩ . By definition of I ( θ ) , we have Var p θ ( f ) = w ⊤ I ( θ ) w . Moreover, ∥ f ∥ 2 mon = ∥ w ∥ 2 2 . Thus, Lemma 4.4 gives us that w ⊤ I ( θ ) w ≥ ( nB ) -O ( d 3 ) ∥ w ∥ 2 2 , using that R = 2 d +8 nBM and M = ( n + d d ) . The bound λ min ( I ( θ )) ≥ ( nB ) -O ( d 3 ) follows.

5 Statistical Efficiency of Score Matching

In this section we prove Theorem 1.3. The main technical ingredient is a bound on the restricted Poincaré constants of distributions in P n,d,B . For any fixed θ ∈ Θ B , we showthat C P can be bounded in terms of the condition number of the Fisher information matrix I ( θ ) . We describe the building blocks of the proof below.

Fix θ, w ∈ R M -1 and define f ( x ) := ⟨ w,T ( x ) ⟩ . First, we need to upper bound Var p θ ( f ) . This is where (the first half of) the condition number appears. Using the crucial fact that the restricted Poincaré constant only considers functions f that are linear in the sufficient statistics, and the definition of I ( θ ) , we get the following bound on Var p θ ( f ) in terms of the coefficient vector w . The proof is deferred to Section D.

Lemma 5.1. Fix θ, w ∈ R M -1 and define f ( x ) := ⟨ w,T ( x ) ⟩ . Then

Next, we lower bound E x ∼ p θ ∥∇ x f ( x ) ∥ 2 2 . To do so, we could pick an orthonormal basis and bound E ⟨ u, ∇ x f ( x ) ⟩ 2 over all directions u in the basis; however, it is unclear how to choose this basis. Instead, we pick u ∼ N (0 , I n ) randomly, and use the following identity:

For any fixed u , the function g ( x ) = ⟨ u, ∇ x f ( x ) ⟩ is also a polynomial. If this polynomial had no constant coefficient, we could immediately lower bound E ⟨ u, ∇ x f ( x ) ⟩ 2 in terms of the remaining coefficients, as above. Of course, it may have a nonzero constant coefficient, but with some case-work over the value of the constant, we can still prove the following bound:

Lemma 5.2. Fix θ, ˜ w ∈ R M -1 and c ∈ R , and define g ( x ) := ⟨ ˜ w,T ( x ) ⟩ + c . Then

Proof. We have

where the inequality is by Lemma 5.1. We now distinguish two cases.

Case I. Suppose that | c + ˜ w ⊤ E x ∼ p θ [ T ( x )] | ≥ c/ 2 . Then

Case II. Otherwise, we have | c + ˜ w ⊤ E x ∼ p θ [ T ( x )] | < c/ 2 . By the triangle inequality, it follows that | ˜ w ⊤ E x ∼ p θ [ T ( x )] | ≥ c/ 2 , so ∥ ˜ w ∥ 2 ≥ c/ (2 ∥ E x ∼ p θ [ T ( x )] ∥ 2 ) . Therefore

from which we get that

as claimed.

With Lemma 5.1 and Lemma 5.2 in hand (taking g ( x ) = ⟨ u, ∇ x f ( x ) ⟩ in the latter), all that remains is to relate the squared monomial norm of ⟨ u, ∇ x f ( x ) ⟩ (in expectation over u ) to the squared monomial norm of f . This crucially uses the choice u ∼ N (0 , I n ) . We put together the pieces in the following lemma. The detailed proof is provided in Section D.

Finally, putting together Lemma 5.3, Theorem 4.1 (that lower bounds λ min ( I ( θ )) ), and Lemma B.2 (that upper bounds λ max ( I ( θ )) - a straightforward consequence of the distributions in P n,d,B having bounded moments), we can prove the following formal version of Theorem 1.3:

Theorem 5.4. Fix n, d, B, N ∈ N . Pick any θ ∗ ∈ Θ B and let x (1) , . . . , x ( N ) ∼ p θ ∗ be independent samples. Then as N →∞ , the score matching estimator ˆ θ SM = ˆ θ SM ( x (1) , . . . , x ( N ) ) satisfies

where ∥ Γ ∥ op ≤ ( nB ) O ( d 3 ) . As a corollary, for all sufficiently large N it holds with probability at least 0 . 99 that ∥ ∥ ∥ ˆ θ SM -θ ∗ ∥ ∥ ∥ 2 ≤ ( nB ) O ( d 3 ) /N .

Proof. We apply Theorem 2.2. By Lemma B.4 and the fact that λ min ( I ( θ ∗ )) > 0 (Theorem 4.1), the necessary regularity conditions are satisfied so that the score matching estimator is consistent and asymptotically normal, with asymptotic covariance Γ satisfying

where C P is the restricted Poincaré constant for p θ ∗ with respect to linear functions in T ( x ) (see Definition 2.1). By Lemma 5.3, we have

using parts (a) and (b) of Lemma B.2; Theorem 4.1; and the fact that M = ( n + d d ) . Substituting into (4) and bounding the remaining terms using Lemma B.3 and a second application of Theorem 4.1, we conclude that ∥ Γ ∥ op ≤ ( nB ) O ( d 3 ) as claimed. The high-probability bound now follows from Markov's inequality; see Remark 4 in Koehler et al. (2022) for details.

6 Conclusion

We have provided a concrete example of an exponential family-namely, exponentials of bounded degree polynomials-where score matching is significantly more computationally efficient than maximum likelihood estimation (through optimization with a zero- or first-order oracle), while still achieving the same sample efficiency up to polynomial factors. While score matching was designed to be more computationally efficient for exponential families, the determination of statistical complexity is more challenging, and we give the first separation between these two methods for a general class of functions.

As we have restricted our attention to the asymptotic behavior of both of the methods, an interesting future direction is to see how the finite sample complexities differ. One could also give a more fine-grained comparison between the polynomial dependencies of score matching and MLE, which we have not attempted to optimize. Finally, it would be interesting to relate our results with similar results and algorithms for learning Ising and higher-order spin glass models in the discrete setting, and give a more unified treatment of psueudo-likelihood or score/ratio matching algorithms in these different settings.

References

Arora, S. and Barak, B. (2009). Computational complexity: a modern approach . Cambridge University Press.

Barp, A., Briol, F.-X., Duncan, A., Girolami, M., and Mackey, L. (2019). Minimum stein discrepancy estimators. Advances in Neural Information Processing Systems , 32.

Besag, J. (1977). Efficiency of pseudolikelihood estimation for simple gaussian fields. Biometrika , pages 616-618.

Bresler, G. (2015). Efficiently learning ising models on arbitrary graphs. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 771-782.

Bresler, G., Gamarnik, D., and Shah, D. (2014). Structure learning of antiferromagnetic ising models. Advances in Neural Information Processing Systems , 27.

Brown, L. D. (1986). Fundamentals of statistical exponential families: with applications in statistical decision theory. Ims.

Cook, S. A. (1971). The complexity of theorem-proving procedures. In Proceedings of the third annual ACM symposium on Theory of computing , pages 151-158.

Dagan, Y., Daskalakis, C., Dikkala, N., and Kandiros, A. V. (2021). Learning ising models from one or multiple samples. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 161-168.

Darmois, G. (1935). Sur les lois de probabilitéa estimation exhaustive. CR Acad. Sci. Paris , 260(1265):85.

Forbes, P. G. and Lauritzen, S. (2015). Linear estimating equations for exponential families with application to gaussian linear concentration models. Linear Algebra and its Applications , 473:261283.

Hyvärinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4).

Hyvärinen, A. (2007). Some extensions of score matching. Computational statistics & data analysis , 51(5):2499-2512.

Koehler, F., Heckett, A., and Risteski, A. (2022). Statistical efficiency of score matching: The view from isoperimetry. arXiv preprint arXiv:2210.00726 .

Koepf, W. (1998). Hypergeometric summation. Vieweg, Braunschweig/Wiesbaden , 5(6).

Koopman, B. O. (1936). On distributions admitting a sufficient statistic. Transactions of the American Mathematical society , 39(3):399-409.

Montanari, A. (2015). Computational implications of reducing data to sufficient statistics.

Pitman, E. J. G. (1936). Sufficient statistics and intrinsic accuracy. In Mathematical Proceedings of the cambridge Philosophical society , volume 32, pages 567-579. Cambridge University Press.

Ren, C. X., Misra, S., Vuffray, M., and Lokhov, A. Y. (2021). Learning continuous exponential families beyond gaussian. arXiv preprint arXiv:2102.09198 .

Shah, A., Shah, D., and Wornell, G. (2021). A computationally efficient method for learning exponential family distributions. Advances in neural information processing systems , 34:1584115854.

Valiant, L. G. and Vazirani, V. V . (1985). Np is as easy as detecting unique solutions. In Proceedings of the seventeenth annual ACM symposium on Theory of computing , pages 458-463.

Van der Vaart, A. W. (2000). Asymptotic statistics , volume 3. Cambridge university press.

Vuffray, M., Misra, S., Lokhov, A., and Chertkov, M. (2016). Interaction screening: Efficient and sample-optimal learning of ising models. Advances in neural information processing systems , 29.

A Omitted Proofs from Section 3

Theorem A.1 (Valiant and Vazirani (1985); Cook (1971)) . Suppose that there is a randomized poly( n ) -time algorithm for the following problem: given a 3CNF formula C with n variables and at most 5 n clauses, under the promise that C has at most one satisfying assignment, determine whether C is satisfiable. Then, NP = RP .

Lemma A.2. In the setting of Definition 3.3, set d := 7 and B := 64 mα +2 β . Then p C ,α,β ∈ P n,d,B .

Proof. Since αH C ( x ) + βG ( x ) is a polynomial in x 1 , . . . , x n of degree at most 7 , there is some θ = θ ( C , α, β ) ∈ R M -1 such that ⟨ θ, T ( x ) ⟩ + αH C ( x )+ βG ( x ) is a constant independent of x . Then h ( x ) exp( -αH C ( x ) -βG ( x )) is proportional to h ( x ) exp( ⟨ θ, T ( x ) ⟩ ) , so p C ,α,β = p θ . Moreover, for any clause C j , every monomial of H C j has coefficient at most 64 in absolute value, so every monomial of H C has coefficient at most 64 m . Similarly, every monomial of G has coefficient at most 2 in absolute value. Thus, ∥ θ ∥ ∞ ≤ 64 mα +2 β =: B , so p C ,α,β ∈ P n,d,B .

Given a point v ∈ H , let O ( v ) := { x ∈ R n : x i v i ≥ 0; ∀ i ∈ [ n ] } denote the octant containing v , and let B r ( v ) := { x ∈ R n : ∥ x -v ∥ ∞ ≤ r } denote the ball of radius r with respect to ℓ ∞ norm.

Lemma A.3. Let p := p C ,α,β and Z := Z C ,α,β for some 3CNF C with m clauses and n variables, and some parameters α, β > 0 . Let r ∈ (0 , 1) . If β ≥ 40 r -2 log(4 n/r ) , then for any v ∈ H that is a satisfying assignment for C ,

For any w ∈ H that is not a satisfying assignment for C ,

Proof. We begin by lower bounding the probability over B r ( v ) . Pick any clause C ℓ included in C . We claim that H C ℓ ( v ′ ) ≤ 81 r 2 for all v ′ ∈ B r ( v ) . Indeed, say that C ℓ = ˜ x i ∨ ˜ x j ∨ ˜ x k . Since v satisfies C ℓ , at least one of { f i ( v i ) , f j ( v j ) , f k ( v k ) } must be zero. Without loss of generality, say that f i ( v i ) = 0 ; also observe that | f j ( v j ) | , | f k ( v k ) | ≤ 2 . It follows that for any v ′ ∈ B r ( v ) , | f i ( v ′ i ) | ≤ r and | f j ( v ′ j ) | , | f j ( v ′ k ) | ≤ 2 + r ≤ 3 (since r ≤ 1 ). Therefore, we have

Summing over all m possible clauses, we have H C ( v ′ ) ≤ 81 mr 2 for all v ′ ∈ B r ( v ) . Hence,

where the second inequality (5) is by Lemma A.8. Next, we upper bound the probability over O ( w ) . Let C ℓ be any clause in C that is not satisfied by w . Say that C ℓ = ˜ x i ∨ ˜ x j ∨ ˜ x k . Then | f i ( w i ) | = | f j ( w j ) | = | f k ( w k ) | = 2 . Furthermore, for any w ′ ∈ O d ( w ) , we have

| f i ( w ′ i ) | , | f j ( w ′ j ) | , | f k ( w ′ k ) | ≥ 1 , and hence H C ℓ ( w ′ ) ≥ 1 . Since H C ′ ( x ) ≥ 0 for all x, C ′ , we conclude that H C ( w ′ ) ≥ H C ℓ ( w ′ ) ≥ 1 for all w ′ ∈ O ( w ) . In particular, this gives us

as claimed.

A.1 Hardness of approximate sampling

Lemma A.4. Let C be a satisfiable instance of 3 -SAT with m clauses and n variables. Let α, β > 0 satisfy α ≥ 2( n +1) and β ≥ 6480 m log(13 n √ m ) . Set p := p C ,α,β and Z := Z C ,α,β . If V ⊆ H is the set of satisfiable assignments for C , then

Proof. Let v ∈ H be any assignment that satisfies C , and let w ∈ H be any assignment that does not satisfy C . By Lemma A.3 with r = 1 / √ 162 m , we have

Since we chose α sufficiently large that e -1+ α/ 2 ≥ 2 n , we get that

Hence,

The lemma statement follows.

Theorem A.5. Let B ≥ Cn 2 for a sufficiently large constant C . Unless RP = NP , there is no algorithm which takes as input an arbitrary θ ∈ Θ B and outputs a sample from a distribution Q with TV( P θ , Q ) ≤ 1 / 3 in poly( n ) time.

Proof. Suppose that such an algorithm exists. For each n ∈ N define α = 2( n + 1) and β = 32400 n log(13 n √ 5 n ) . Given a 3CNF formula C with n variables and at most 5 n clauses, we can compute θ = θ ( C , α, β ) . By Lemma A.2 we have θ ∈ Θ B so long as B ≥ Cn 2 for a sufficiently large constant C . Thus, by assumption we can generate a a sample from a distribution Q with TV( P C ,α,β , Q ) ≤ 1 / 3 . But by Lemma A.4, we have Pr x ∼ P C ,α,β [sign( x ) satisfies C ] ≥ 1 / 2 . Thus, Pr x ∼ Q [sign( x ) satisfies C ] ≥ 1 / 6 . It follows that we can find a satisfying assignment with O (1) invocations of the sampling algorithm in expectation. By Theorem A.1 we get NP = RP .

A.2 Hardness of approximating zeroth-order oracle

Lemma A.6. Fix n, m ∈ N and let α ≥ 2( n +1) and β ≥ 6480 m log(13 n √ m ) . There is a constant A = A ( n, m, α, β ) so that the following hold for every 3CNF formula C with n variables and m clauses:

If C is unsatisfiable, then Z C ,α,β ≤ A

If C is satisfiable, then Z C ,α,β ≥ ( e/ 2) n A .

Proof. If C is unsatisfiable, then by the second part of Lemma A.3, we have

On the other hand, if C is satisfiable, then by the first part of Lemma A.3 with r = 1 / √ 162 m ,

Since α ≥ 2( n +1) , we get

Proof of Theorem 3.4. First, observe that the following problem is NP -hard (under randomized reductions): given two 3CNF formulas C , C ′ each with n variables and at most 10 n clauses, where it is promised that exactly one of the formulas is satisfiable, determine which of the formulas is satisfiable. Indeed, this follows from Theorem A.1: given a 3CNF formula C with n variables, at most 5 n clauses, and at most one satisfying assignment, consider adjoining either the clause x i or the clause ¬ x i to C . If C has a satisfying assignment v ∗ , then exactly one of the resulting formulas is satisfiable, and determining which one is satisfiable identifies v ∗ i . Repeating this procedure for all i ∈ [ n ] yields an assignment v , which satisfies C if and only if C is satisfiable.

For each n ∈ N define α = 2( n +1) and β = 64800 n log(13 n √ 10 n ) . Let B > 0 be chosen later. Suppose that there is a poly( n ) -time algorithm which, given θ ∈ Θ B , computes an approximation of log Z θ with additive error less than n log 1 . 16 . Then given two formulas C and C ′ with n variables and at most 10 n clauses each, we can compute θ = θ ( C , α, β ) and θ ′ = θ ( C ′ , α, β ) . By Lemma A.2, we have θ, θ ′ ∈ Θ B so long as B ≥ Cn 2 for a sufficiently large constant C . Hence by assumption we can compute approximations ˜ Z θ and ˜ Z θ ′ of Z θ and Z θ ′ respectively, with multiplicative error less than 1 . 16 n . However, by Lemma A.6 and the assumption that exactly one of C and C ′ is satisfiable, we know that ˜ Z θ > ˜ Z θ ′ if and only if C is satisfiable. Thus, NP = RP .

A.3 Hardness of approximating first-order oracle

Lemma A.7. Let C be a 3CNF formula with m clauses and n variables, and exactly one satisfying assignment v ∗ ∈ H . Let α = 4 n and β ≥ 25920 mn log(102 n √ mn ) , and define p := p C ,α,β and Z := Z C ,α,β . Then E x ∼ p [ v ∗ i x i ] ≥ 1 / 20 for all i ∈ [ n ] .

Proof. Without loss of generality take i = 1 and v ∗ i = 1 . Set r = 1 / ( √ 648 mn ) , α = 4 n , and β ≥ 40 r -2 log(4 n/r ) . We want to show that E x ∼ p [ x 1 ] ≥ 1 / 20 . We can write

by Lemma A.3. On the other hand, for any v ∈ H \ { v ∗ } ,

where the second inequality is by Lemma A.9 with k = 1 . Combining (7) and (8) with (6), we have

where the second inequality is by choice of α and r ; the third inequality is by nonnegativity of H ( x ) ; and the fourth inequality is by Lemma A.4 and uniqueness of the satisfying assignment v ∗ .

Proof of Theorem 3.5. Suppose that such an algorithm exists. Set α = 4 n and β = 129600 n 2 log(102 n 2 √ 5) . Given a 3CNF formula C with n variables, at most 5 n clauses, and exactly one satisfying assignment v ∗ ∈ H , we can compute θ = θ ( C , α, β ) . Let E ∈ R n be the algorithm's estimate of ∇ θ log Z θ = E x ∼ p C ,α,β T ( x ) . Then ∥ ∥ E -E x ∼ p C ,α,β T ( x ) ∥ ∥ ∞ < 1 / 20 . But by Lemma A.7, for each i ∈ [ n ] , the i -th entry of E x ∼ p C ,α,β T ( x ) , which corresponds to the monomial x i , has sign v ∗ i and magnitude at least 1 / 20 . Thus, sign ( E i ) = v ∗ i . So we can compute v ∗ in polynomial time. By Theorem A.1, it follows that NP = RP .

A.4 Integral bounds

Lemma A.8. Fix β > 150 and γ ∈ [0 , 1] . Define f : R → R by f ( x ) = γx 8 + β (1 -x 2 ) 2 . Pick any r ∈ (6 /β, 0 . 04) . Then

In particular, for any m ∈ N , if β ≥ 40 r -2 log(4 m/r ) , then

Proof. Set a = 1 / √ 2 . For any x ∈ [ a, ∞ ) we have f ′′ ( x ) = 56 γx 6 -4 β + 12 βx 2 ≥ β > 0 for β > 150 . Thus, f has at most one critical point in [ a, ∞ ) ; call this point t 0 . Since f ′ ( x ) = 8 γx 7 -4 βx (1 -x 2 ) , we have f ′ (1) = 8 γ ≥ 0 and f ′ (1 -3 /β ) ≤ 8 -4 β (1 -3 /β )(3 /β )(2 -3 /β ) < 0 . Thus, t 0 ∈ (1 -3 /β, 1] . Set r ′ = r -3 /β ≥ r/ 2 . Then

For every t ∈ R define I ( t ) = ∫ t + r ′ t exp( -f ( x )) dx . Since f is β -strongly convex on [ a, ∞ ) , we have for any t ≥ t 0 that

where the final inequality is because f ′ ( t ) ≥ 0 for t ∈ [ t 0 , ∞ ) . Thus, for any t ≥ t 0 ,

By induction, for any k ∈ N it holds that I ( t 0 + kr ′ ) ≤ exp( -βkr ′ 2 / 2) I ( t 0 ) , so

Similarly, for any t ∈ [ a + r ′ , t 0 ] , we have

using β -strong convexity on [ a, ∞ ) and the bound f ′ ( t ) ≤ 0 on [ a, t 0 ] . Thus, for any t ∈ [ a, t 0 -r ′ ] ,

so by induction, I ( t 0 -kr ′ ) ≤ exp( -β ( k -1) r ′ 2 / 2) I ( t 0 -r ′ ) for any 1 ≤ k ≤ K := ⌊ ( t 0 -a ) /r ′ ⌋ . It follows that

Finally, note that t 0 -( K -1) r ′ ≤ a +2 r ′ ≤ 0 . 8 . For any x ∈ [0 , 0 . 8] , we have f ′ ( x ) ≤ 8 x 7 -0 . 72 βx = x (8 x 6 -1 . 44 β ) ≤ 0 , since β > 150 . That is, f is non-increasing on [0 , t 0 -( K -1) r ′ ] . It follows that

Since ( K -1) r ′ ≥ t 0 -0 . 8 ≥ 1 -3 β -0 . 8 ≥ 0 . 1 , we conclude that

Combining (9), (10), and (11), we get

Substituting in r ≥ r/ 2 gives the claimed result.

Lemma A.9. Fix β ≥ 160 log(8) . Then for any 1 ≤ k ≤ 8 ,

Proof. Define a distribution q ( x ) ∝ exp( -x 8 -β (1 -x 2 ) 2 ) for x ∈ [0 , ∞ ) . We want to show that E q [ x k ] ≤ 2 k . Indeed,

where the first inequality is by an application of Lemma A.8 with r = 1 / 2 and m = 1 . Now by Jensen's inequality we get

and consequently, an application of Hölder inequality gives us E q [ x k ] ≤ 2 k , for any 1 ≤ k ≤ 8 .

A.5 Proof of Corollary 3.6

Proof of Corollary 3.6 . Recall that log p θ ( x ) = log h ( x ) + ⟨ θ, T ( x ) ⟩ -log Z θ . Therefore L MLE ( θ ) = ˆ E log h ( x ) + ⟨ θ, ˆ E T ( x ) ⟩ -log Z θ and ∇ θ L MLE ( θ ) = ˆ E T ( x ) - ∇ θ log Z θ . Note that we can compute ˆ E log h ( x ) and ˆ E T ( x ) exactly. It follows that if we can approximate L MLE ( θ ) up to an additive error of n log 1 . 16 , then we can compute log Z θ up to an additive error of n log 1 . 16 . Similarly, if we can compute ∇ θ L MLE ( θ ) up to an additive error of 1 / 20 , then we can compute ∇ θ log Z θ up to an additive error of 1 / 20 . This contradicts Theorems 3.4 and 3.5 respectively, completing the proof.

B Moment bounds

Lemma B.1 (Moment bound) . For any θ ∈ Θ B , i ∈ [ n ] , and ℓ ∈ N it holds that

Proof. Without loss of generality assume i = 1 . Let L 0 := 32 max( ℓ, BM 2 d +1 ) . Then

Now for any L ≥ L 0 ,

We can lower bound Z θ as

Thus,

where the last inequality uses the facts that L ≥ L 0 , B ≥ 1 , and M ≥ n to get

We conclude that

which completes the proof.

Lemma B.2 (Largest eigenvalue bound) . For any θ ∈ Θ B , it holds that

We also have the following consequences:

Proof. Fix any u, v ∈ [ M ] . Then T ( x ) u T ( x ) v = ∏ n i =1 x γ i i for some nonnegative integers γ 1 , . . . , γ n where d ′ := ∑ n i =1 γ i ≤ 2 d . Therefore

by Holder's inequality and Lemma B.1 (with ℓ = 2 d ). The claimed spectral bound follows. To prove (a), observe that

To prove (b), observe that I ( θ ) ⪯ E x ∼ p θ T ( x ) T ( x ) ⊤ . To prove (c), observe that for any i ∈ [ n ] ,

A union bound over i ∈ [ n ] completes the proof.

Lemma B.3 (Smoothness bounds) . For every θ ∈ Θ B , it holds that

and

Proof. Fix any j ∈ [ M ] ; then there is a degree function d with 1 ≤ | d | ≤ d so that T j ( x ) = x d = ∏ n i =1 x d ( i ) i . Therefore

for some w ∈ R M with ∥ w ∥ 2 2 = ∑ k ∈ [ n ]: d ( k ) ≥ 2 d ( k ) 2 ( d ( k ) -1) 2 ≤ d 4 . By Corollary B.2, we conclude that

Summing over j ∈ [ M ] gives the first claimed bound. For the second bound, observe that

.

For any j ∈ [ M ] and i ∈ [ n ] , there is some degree function d with | d | ≤ d and ∂ ∂x i T j ( x ) = | d | · x d -{ i } . Thus, by Holder's inequality and Lemma B.1 (with ℓ = 4 d ), we get

which proves the second bound.

The following regularity conditions are sufficient for consistency and asymptotic normality of score matching, assuming that the restricted Poincaré constant is finite and λ min ( I ( θ ∗ )) > 0 (see Proposition 2 in Forbes and Lauritzen (2015) together with Lemma 1 in Koehler et al. (2022)). We show that these conditions hold for our chosen exponential family.

Lemma B.4 (Regularity conditions) . For any θ ∈ R M , the quantities E x ∼ p θ ∥∇ h ( x ) ∥ 4 2 , E x ∼ p θ ∥ ∆ T ( x ) ∥ 2 2 , and E x ∼ p θ ∥ ( JT )( x ) ∥ 4 op are all finite. Moreover, p θ ( x ) → 0 and ∥∇ x p θ ( x ) ∥ 2 → 0 as ∥ x ∥ 2 →∞ .

Proof. Both of the quantities ∥∇ h ( x ) ∥ 4 2 and ∥ ∆ T ( x ) ∥ 2 2 can be written as a polynomial in x . Finiteness of the expectation under p θ follows from Holder's inequality and Lemma B.1 (with parameter B set to ∥ θ ∥ ∞ ). Finiteness of E x ∼ p θ ∥ ( JT )( x ) ∥ 4 op is shown in Lemma B.3 (again, with B := ∥ θ ∥ ∞ ). The decay condition p θ ( x ) → 0 holds because log p θ ( x ) + log Z θ = -∑ n i =1 x d +1 i + ⟨ θ, T ( x ) ⟩ . For x ∈ R n with L ≤ ∥ x ∥ ∞ ≤ 2 L , the RHS is at most -L d +1 + M ∥ θ ∥ ∞ (2 L ) d , which goes to -∞ as L →∞ . A similar bound shows that ∥∇ x p θ ( x ) ∥ 2 → 0 .

C Omitted Proofs from Section 4

Proof of Lemma 4.2 . We use the fact that the Legendre polynomials

for integers 0 ≤ k ≤ d , form an orthogonal basis for the vector space R [ x ] ≤ d with respect to L 2 [ -1 , 1] (see e.g. Koepf (1998)). We consider the normalized versions ˆ L k = √ 2 k +1 2 L k , so that ∥ ∥ ∥ ˆ L k ∥ ∥ ∥ L 2 [ -1 , 1] = 1 . By tensorization, the set of products of Legendre polynomials

as d ranges over degree functions with | d | ≤ d , form an orthonormal basis for R [ x 1 , . . . , x n ] ≤ d with respect to L 2 ([ -1 , 1] n ) .

Using the formula for L k , we obtain that the sum of absolute values of coefficients of L k (in the monomial basis) is at most 1 2 k ∑ k j =0 ( k j ) 2 2 k ≤ 2 2 k . By the bound ∥·∥ 2 ≤ ∥·∥ 1 and the definition of ˆ L k ,

and hence for any degree function d with | d | ≤ d ,

Consider any polynomial f ∈ R [ x 1 , . . . , x n ] ≤ d , and write f = ∑ | d |≤ d a d ˆ L d . By orthonormality, it holds that ∑ | d |≤ d a 2 d = ∥ f ∥ 2 L 2 ([ -1 , 1] n ) . Thus, by the triangle inequality and Cauchy-Schwarz,

as claimed.

Proof of Lemma 4.3 . Let f ∈ R [ x 1 , . . . , x n ] ≤ d be a polynomial of degree at most d in x 1 , . . . , x n . Define g ( x ) = f ( x ) -E x ∼ p f ( x ) . Set ϵ = 1 / (2( d +1) MR d ( n + B )) and let ( W i ) i ∈ I be ℓ ∞ -balls of radius ϵ partitioning { x ∈ R n : ∥ x ∥ ∞ ≤ R } . Define random variable X ∼ p |{∥ X ∥ ∞ ≤ R } and let ι ∈ I be the random index so that X ∈ W ι . Then

where the inequality uses guarantee (c) of Lemma B.2 that Pr x ∼ p [ ∥ x ∥ ∞ > R ] ≤ 1 / 2 .

Thus, there exists some ι ∗ ∈ I such that E X [ g ( X ) 2 | X ∈ W ι ∗ ] ≤ 2 Var p ( f ) . Let q : R n → R + be the density function of X | X ∈ W ι ∗ . Since q ( x ) ∝ p ( x ) [ x ∈ W ι ∗ ] , for any u, v ∈ W ι ∗ we have that

Applying Lemma C.1, we get that

by choice of ϵ . It follows that if ˜ U is the uniform distribution on W ι ∗ , then q ( x ) ≥ e -1 ˜ U ( x ) for all x ∈ R n . Thus,

as desired.

Lemma C.1. Fix R > 0 . For any degree function d : [ n ] → N with | d | ≤ d , and for any u, v ∈ R n with ∥ u ∥ ∞ , ∥ v ∥ ∞ ≤ R , it holds that

Proof. Define m ( x ) = x d = ∏ n x d ( i )

Proof of Lemma 4.4 . By Lemma 4.3, there is some z ∈ R n with ∥ z ∥ ∞ ≤ R and some ϵ ≥ 1 / (2( d +1) MR d ( n + B )) so that if ˜ U is the uniform distribution on { x ∈ R n : ∥ x -z ∥ ∞ ≤ ϵ } , then

2 e . Define g : R n → R by g ( x ) = f ( ϵx + z ) -E ˜ U f . Then by Lemma 4.2, .

Write f ( x ) = ∑ 1 ≤| d |≤ d α d x d and g ( x ) = ∑ 1 ≤| d |≤ d β d x d . We know that f ( x ) = g ( ϵ -1 ( x -z )) + E ˜ U f . Thus, for any nonzero degree function d , we have

Thus | α d | ≤ ϵ -d R d ∥ β ∥ 1 ≤ ϵ -d R d √ M ∥ g ∥ mon , and so summing over monomials gives

Substituting in the choice of ϵ from Lemma 4.3 completes the proof.

D Omitted Proofs from Section 5

Proof of Lemma 5.1 . We have

and since

the lemma statement follows.

Proof of Lemma 5.3 . Since f ( x ) = ∑ 1 ≤| d |≤ d w d x d , we have for any u ∈ R n that

where c ( u ) := ∑ i =1 u i w { i } and ˜ w ( u ) d := ∑ i =1 u i (1 + d ( i )) w d + { i } . But now

where the last inequality is by Lemma 5.2. Finally,

where the second equality is because E [ u i u j ] = ✶ [ i = j ] for all i, j ∈ [ n ] , and the last inequality is because every term w 2 d in ∥ w ∥ 2 2 appears in at least one of the terms of the previous summation (and has coefficient at least one). Putting everything together gives

where the last inequality is by Lemma 5.1.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors describe an exponential family where the sufficient statistic $T(x)$ contains all non-constant monomials of degree $\\leq d$, the background density is $h(x) = \\exp(-\\sum_{i=1}^n x_i^{d+1})$, and the parameter $\\theta$ is constrained to have infinity norm bounded by $B$. Using a reduction from $3\\mathsf{SAT}$, they show that under this exponential family, finding the MLE is NP hard, and thus unless $\\mathsf{NP} = \\mathsf{RP}$, that it would take time exponential in the random vector dimension $n$ to compute the MLE. \n\nThey also show that the MLE has asymptotic sample efficiency $(nB)^{O(d^3)}$. They show that score matching also has asymptotic sample efficiency $(nB)^{O(d^3)}$. Unlike the MLE, score matching can be solved in time polynomial in the dimension of $\\theta$, which if $d$ is considered constant, is polynomial in $n$. This is because the objective corresponding to score matching is convex in $\\theta$.\n\nThis is a concrete example of a situation where score matching has a provable advantage over the MLE (same asymptotic sample efficiency, but one needs exponential time unless $\\mathsf{NP} = \\mathsf{RP}$ and the other is polynomial time).\n\nSTRENGTHS:\n- clear contribution\n- relevant related work is discussed\n- this is a significant result (rigorous justification for potential benefit of score matching over MLE)\n\nWEAKNESSES:\nNo glaring weaknesses, but the paper was a bit difficult to follow; I found it hard to connect the math-heavy theorem/lemma statements together. For example, I did not understand how Lemmas 4.2 and 4.3 built toward Lemma 4.4 on my initial read. \n\nTypos spotted:\n- Equation under line 172, $\\|\\theta\\|_2^2$ should be $\\|\\theta^*\\|_2^2$?\n- Line 482, $f''(x) = 56\\gamma x^6 - 2\\beta + 6\\beta x^2$ should be $f''(x) = 56\\gamma x^6 - 4\\beta + 12\\beta x^2$ I think, but this doesn't affect anything\n- Lemma A.6, the $(2/e)^n$ should be replaced with $(e/2)^n$\n- Math at the bottom of page 16, second line, you want to minimize the exponent so instead of $-BM(BM)^{-d}$ it should be $-BM(BM)^{-1}$ right? I don't think this affects anything though since you bound it by $-1$ regardless\n- Math under line 519, I'm not sure that $(n+\\ell)\\log(2L) + 2 + n\\log(BM) \\leq \\frac14 L^{d+1}$, unless some lower bound on $d$ is assumed? Like I'm not sure if that holds for $d=1$?\n- Lemma B.3, bound on Laplacian in the Lemma statement doesn't seem to match the derivation (exponent of 2d vs 4d)\n- Line 564 - this appears to be claiming that $\\sum_{j=0}^k \\binom{k}{j}^2 = 2^k$, which is false? I think you could bound it by $2^{2k}$ though by Cauchy-Schwarz, so it probably doesn't matter, just changes some constants down the line.\n- Line 573 - $B_\\iota \\to W_\\iota$\n- Line 583 - $\\alpha_i \\to d_i$\n\nQUESTIONS:\n- Why is a nonzero constant not allowed for the polynomial defined by the sufficient statistic of the exponential family? It wasn't clear to me, also why that was a requirement for Lemma 4.4 to hold. Does that cause a problem?\n\n- This does not appear necessary to have in this work to me, but how close are we to obtaining finite sample bounds for score matching, analogous to the ones (I assume there are?) for MLE?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-20"}

comments: SUMMARY:
This paper provides an example of fitting exponential family models for which score matching 
and MLE are both statistically efficient, but MLE is computationally hard to optimize.

STRENGTHS:
The strength is in the construction of an example to showcase the benefit of score matching 
over MLE.

WEAKNESSES:
Several aspects  could be improved. 

1. First, for the computational lower bound, the points to evaluation loss and gradient is worst case. 
When one has samples from such distribution, is solving MLE still computationally hard? 

2. Although the aim of this paper is to provide examples to advocate for score matching, but for the 
distribution family proposed in the paper, are there simple (or simpler) estimators that are both computationally and 
statistically efficient? 

3. In the paper, the analysis of the statistical performance is quite crude, which makes it hard to see what's 
the statistical cost one needs to pay when computation is the restriction. Is there proper computation-statistic 
tradeoff for the model considered in the paper?

QUESTIONS:
1. Why constant is removed from the construction of the sufficient statistics?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a mathematical setting where Score Matching (SM) method has more statistical benefits than Maximum Likelihood (ML) technique, when estimating a parameterized probability distribution $p_\\theta \\in P(\\mathbb{R}^n)$ known up to a normalizing constant $Z_\\theta$. In particular, they describe an explicit exponential family of distributions $F$ for which SM loss $L_{SM}$ is efficient to compute, with same statistical efficiency as ML loss $L_{ML}$ (Theorem 2 and 3), while $L_{ML}$ is shown to be intractable in polynomial time depending on the parameters of this family (Theorem 1). Given an odd integer $d$ and $B>0$, any distribution $p_\\theta \\in F$ is notably defined by (i) its vector of sufficient statistics, which consists of all monomials in $x^1,..., x_n$ of at least degree $1$ and at most degree $d$, and (ii) its parameter $\\theta$, which lies in the $\\ell_\\infty$-ball with radius $B$. This work is the first one to put in perspective the statistical efficiency of SM and ML for a large family of continuous probability distributions. \n\n\nThree main theoretical results are stated here. In Theorem 1, the authors prove that, for any $p_\\theta \\in F$ (with $d=7$) and any set of $N$ independent samples from $p_\\theta$, it is NP-hard (in $n$ and $N$) to provide an accurate approximation of $L_{ML}(\\theta)$ and $\\nabla L_{ML}(\\theta)$. This result comes from the difficulty to approximate $Z_\\theta$, which is necessary to compute $L_{ML}(\\theta)$, while it does not appear in $L_{SM}(\\theta)$. In Theorem 2, they derive an upper bound of the $\\ell_2$-error between $\\theta$ and its ML estimator obtained via $N$ samples, in the limit where $N\\to \\infty$, for any $p_\\theta \\in F$. Their proof relies on the asymptotic result given by [1] and consists of lower bounding the smallest eigenvalue of the Fisher information matrix of $p_\\theta$. In Theorem 3, they derive the same upper bound in the case of the SM estimator for any $p_\\theta \\in F$, by invoking the asymptotic result from [2] and bounding the Poincaré constant of $p_\\theta$. Combined together, Theorems 2 and 3 show that ML and SM techniques roundly have the same statistical efficiency.\n\n[1] Asymptotic statistics, Van der Vaart, 2000.\n\n[2] Statistical Efficiency of Score Matching: The View from Isoperimetry, Koehler et al., 2022.\n\nSTRENGTHS:\n- This work provides a fair comparison between statistical efficiency for SM and ML technique, which is of primary interest for the community of machine learning.\n- Although Theorems 2 and 3 rely on important theoretical results, their results are not straightforward to obtain, and their proofs are original and clear to understand.\n\nWEAKNESSES:\nAlthough the theoretical results presented here are interesting in their own, they should be combined with numerical experiments, which should illustrate the trade-off between these two methods in practice, depending on the parameters of the exponential family.\n\nQUESTIONS:\nWhy do the authors do not keep the whole dependence in $d$ in the bound for Theorems 2 and 3 ? Does it change something between SM and ML ?\n\nLIMITATIONS:\nIn this work, the asymptotic theory is elaborated on the fact that the MLE and SM estimators can be computed exactly from a collection of samples. However, to compare these two techniques for practical purpose, one should include approximation error on the estimator.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper attempts to elucidate the theoretical reasoning for the benefits seen in score matching. The author proposes a family of exponential distributions that can efficiently compute the score matching loss while having a comparable statistical efficiency to that of maximum likelihood.

STRENGTHS:
- The paper is well written/organized and easy to follow. The author does a good job introducing related theoretical information.
- Equations are extensively described with a thorough description.

WEAKNESSES:
- There are no experimental results. It would be nice if there were some machine learning models that were optimized with score matching and ML and compared with each other.

QUESTIONS:
- What is the benefit of using this score matching method than taking the denoising score matching approach?
- Is this approach more computational efficient than denoising score matching? How well those this score matching approach scale for high dimensional data?

LIMITATIONS:
The author has adequately addressed limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06