SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chirag Pabbaraju
cpabbara@cs.stanford.edu
hlee283@jhu.edu
Dhruv Rohatgi
drohatgi@mit.edu
Ankur Moitra
moitra@mit.edu
asevekar@andrew.cmu.edu
Score matching is an alternative to maximum likelihood (ML) for estimating a probability distribution parametrized up to a constant of proportionality. By fitting the "score" of the distribution, it sidesteps the need to compute this constant of proportionality (which is often intractable). While score matching and variants thereof are popular in practice, precise theoretical understanding of the benefits and tradeoffs with maximum likelihood—both computational and statistical—are not well understood. In this work, we give the first example of a natural exponential family of distributions such that the score matching loss is computationally efficient to optimize, and has a comparable statistical efficiency to ML, while the ML loss is intractable to optimize using a gradient-based method. The family consists of exponentials of polynomials of fixed degree, and our result can be viewed as a continuous analogue of recent developments in the discrete setting. Precisely, we show: (1) Designing a zeroth-order or first-order oracle for optimizing the maximum likelihood loss is NP-hard. (2) Maximum likelihood has a statistical efficiency polynomial in the ambient dimension and the radius of the parameters of the family. (3) Minimizing the score matching loss is both computationally and statistically efficient, with complexity polynomial in the ambient dimension.
Energy-based models are a flexible class of probabilistic models with wide-ranging applications. They are parameterized by a class of energies  $E_{\theta}(x)$  which in turn determines the distribution
$$
p _ {\theta} (x) = \frac {\exp (- E _ {\theta} (x))}{Z _ {\theta}}
$$
up to a constant of proportionality  $Z_{\theta}$  that is called the partition function. One of the major challenges of working with energy-based models is designing efficient algorithms for fitting them to data. Statistical theory tells us that the maximum likelihood estimator (MLE)—i.e., the parameters  $\theta$  which maximize the likelihood—enjoys good statistical properties including consistency and asymptotic efficiency.
However, there is a major computational impediment to computing the MLE: Both evaluating the log-likelihood and computing its gradient with respect to  $\theta$  (i.e., implementing zeroth and first order oracles, respectively) seem to require computing the partition function, which is often computationally intractable. More precisely, the gradient of the negative log-likelihood depends on  $\nabla_{\theta}\log Z_{\theta} = \mathbb{E}_{p_{\theta}}[\nabla_{\theta}E_{\theta}(x)]$ . A popular approach is to estimate this quantity by using a Markov chain to approximately sample from  $p_{\theta}$ . However in high-dimensional settings, Markov chains often require many, sometimes even exponentially many, steps to mix.
Score matching (Hyvärinen, 2005) is a popular alternative that sidesteps needing to compute the partition function of sample from  $p_{\theta}$ . The idea is to fit the score of the distribution, in the sense that
we want  $\theta$  such that  $\nabla_x\log p(x)$  matches  $\nabla_{x}\log p_{\theta}(x)$  for a typical sample from  $p$ . This approach turns out to have many nice properties. It is consistent in the sense that minimizing the objective function yields provably good estimates for the unknown parameters. Moreover, while the definition depends on the unknown  $\nabla_{x}\log p(x)$ , by applying integration by parts, it is possible to transform the objective into an equivalent one that can be estimated from samples.
The main question is to bound its statistical performance, especially relative to that of the maximum likelihood estimator. Recent work by Koehler et al. (2022) showed that the cost can be quite steep. They gave explicit examples of distributions that have bad isoperimetric properties (i.e., large Poincaré constant) and showed how such properties can cause poor statistical performance.
Despite wide usage, there is little rigorous understanding of when score matching helps. This amounts to finding a general setting where maximizing the likelihood with standard first-order optimization is provably hard, and yet score matching is both computationally and statistically efficient, with only a polynomial loss in sample complexity relative to the MLE. In this work, we show the first such guarantees, and we do so for a natural class of exponential families defined by polynomials. As we discuss in Section 1.1, our results parallel recent developments in learning graphical models—where it is known that pseudolikelihood methods allow efficient learning of distributions that are hard to sample from—and can be viewed as a continuous analogue of such results.
In general, an exponential family on  $\mathbb{R}^n$  has the form  $p_{\theta}(x) \propto h(x)\exp(\langle\theta,T(x)\rangle)$  where  $h(x)$  is the base measure,  $\theta$  is the parameter vector, and  $T(x)$  is the vector of sufficient statistics. Exponential families are one of the most classic parametric families of distributions, dating back to works by Darmois (1935), Koopman (1936) and Pitman (1936). They have a number of natural properties, including: (1) The parameters  $\theta$  are uniquely determined by the expectation of the sufficient statistics  $\mathbb{E}_{p_\theta}[T]$ ; (2) The distribution  $p_{\theta}$  is the maximum entropy distribution, subject to having given values for  $\mathbb{E}_{p_\theta}[T]$ ; (3) They have conjugate priors (Brown, 1986), which allow characterizations of the family for the posterior of the parameters given data.
For any (odd positive integer) constant  $d$  and norm bound  $B \geq 1$ , we study a natural exponential family  $\mathcal{P}_{n,d,B}$  on  $\mathbb{R}^n$  where
1. The sufficient statistics  $T(x) \in \mathbb{R}^{M - 1}$  consist of all monomials in  $x_{1},\ldots ,x_{n}$  of degree at least 1 and at most  $d$  (where  $M = \binom{n+d}{d}$ ).
2. The base measure is defined as  $h(x) = \exp \left(-\sum_{i=1}^{n} x_i^{d+1}\right)$ .
3. The parameters  $\theta$  lie in an  $l_{\infty}$ -ball:  $\theta \in \Theta_B = \{\theta \in \mathbb{R}^{M - 1} : \| \theta \|_{\infty} \leq B\}$ .
Towards stating our main results, we formally define the maximum likelihood and score matching objectives, denoting by  $\hat{\mathbb{E}}$  the empirical average over the training samples drawn from some  $p\in \mathcal{P}_{n,d,B}$ :
$$
L _ {\mathbf {M L E}} (\theta) = \hat {\mathbb {E}} _ {x \sim p} [ \log p _ {\theta} (x) ]
$$
$$
\begin{array}{l} L _ {\mathbf {S M}} (\theta) = \frac {1}{2} \hat {\mathbb {E}} _ {x \sim p} [ \| \nabla \log p (x) - \nabla \log p _ {\theta} (X) \| ^ {2} ] + K _ {p} \\ = \hat {\mathbb {E}} _ {x \sim p} \left[ \operatorname {T r} \nabla^ {2} \log p _ {\theta} (x) + \frac {1}{2} \| \nabla \log p _ {\theta} (x) \| ^ {2} \right] \tag {1} \\ \end{array}
$$
where  $K_{p}$  is a constant depending only on  $p$  and (1) follows by integration by parts (Hyvärinen, 2005). In the special case of exponential families, (1) is a quadratic, and in fact the optimum can be written in closed form:
$$
\underset {\theta} {\arg \min } L _ {\mathbf {S M}} (\theta) = - \hat {\mathbb {E}} _ {x \sim p} [ (J T) _ {x} (J T) _ {x} ^ {T} ] ^ {- 1} \hat {\mathbb {E}} _ {x \sim p} \Delta T (x) \tag {2}
$$
where  $(JT)_x:(M - 1)\times n$  is the Jacobian of  $T$  at the point  $x$ ,  $\Delta f = \sum_{i}\partial_{i}^{2}f$  is the Laplacian, applied coordinate wise to the vector-valued function  $f$ .
With this setting in place, we show the following intractability result.
Theorem 1.1 (Informal, computational lower bound). Unless  $RP = NP$ , there is no poly(n,N)-time algorithm that evaluates  $L_{\mathrm{MLE}}(\theta)$  and  $\nabla L_{\mathrm{MLE}}(\theta)$  given  $\theta \in \Theta_B$  and arbitrary samples  $x_1,\ldots ,x_N\in \mathbb{R}^n$ , for  $d = 7$ ,  $B = \mathrm{poly}(n)$ . Thus, optimizing the MLE loss using a zeroth-order or first-order method is computationally intractable.
<sup>1</sup>We note that the choice of base measure is for convenience in ensuring tail bounds necessary in our proof.
The main idea of the proof is to construct a polynomial  $F_{\mathcal{C}}(x)$  which has roots exactly at the satisfying assignments of a given 3-SAT formula  $\mathcal{C}$ . We then argue that  $\exp(-\gamma F_{\mathcal{C}}(x))$ , for sufficiently large  $\gamma > 0$ , concentrates near the satisfying assignments. Finally, we show sampling from this distribution or approximating  $\log Z_{\theta}$  or  $\nabla_{\theta} \log Z_{\theta}$  (where  $\theta \in \mathbb{R}^{M-1}$  is the parameter vector corresponding to the polynomial  $-\gamma F_{\mathcal{C}}(x)$ ) would enable efficiently finding a satisfying assignment.
Our next result shows that MLE, though computationally intractable to compute via implementing zeroth or first order oracles, has (asymptotic) sample complexity  $\mathrm{poly}(n,B)$  (for constant  $d$ ).
Theorem 1.2 (Informal, efficiency of MLE). The MLE estimator  $\hat{\theta}_{\mathrm{MLE}} = \arg \max_{\theta} L_{\mathrm{MLE}}(\theta)$  has asymptotic sample complexity polynomial in  $n$ . That is, for all sufficiently large  $N$  it holds with probability at least 0.99 (over  $N$  samples drawn from  $p_{\theta^{*}}$ ) that:
$$
\left\| \hat {\theta} _ {\mathbf {M L E}} - \theta^ {*} \right\| ^ {2} \leq O \left(\frac {(n B) ^ {\operatorname {p o l y} (d)}}{N}\right).
$$
The main proof technique for this is an anticoncentration bound of low-degree polynomials, for distributions in our exponential family.
Lastly, we prove that score matching also has polynomial (asymptotic) statistical complexity.
Theorem 1.3 (Informal, efficiency of SM). The score matching estimator  $\hat{\theta}_{\mathrm{SM}} = \arg \max_{\theta} L_{\mathrm{SM}}(\theta)$  also has asymptotic sample complexity at most polynomial in  $n$ . That is, for all sufficiently large  $N$  it holds with probability at least 0.99 (over  $N$  samples drawn from  $p_{\theta^{*}}$ ) that:
$$
\left\| \hat {\theta} _ {\mathbf {S M}} - \theta^ {*} \right\| ^ {2} \leq O \left(\frac {(n B) ^ {\mathrm {p o l y} (d)}}{N}\right). \tag {3}
$$
The main ingredient in this result is a bound on the restricted Poincaré constant—namely, the Poincaré constant, when restricted to functions that are linear in the sufficient statistics  $T$ . We bound this quantity for the exponential family we consider in terms of the condition number of the Fisher matrix of the distribution, which we believe is a result of independent interest. With this tool in hand, we can use the framework of Koehler et al. (2022), which relates the asymptotic sample complexity of score matching to the asymptotic sample complexity of maximum likelihood, in terms of the restricted Poincaré constant of the distribution.
1.1 Discussion and related work
Score matching: Score matching was proposed by Hyvarinen (2005), who also gave conditions under which it is consistent and asymptotically normal. Asymptotic normality is also proven for various kernelized variants of score matching in Barp et al. (2019). Koehler et al. (2022) prove that the statistical sample complexity of score matching is not much worse than the sample complexity of maximum likelihood when the distribution satisfies a (restricted) Poincaré inequality. While we leverage machinery from Koehler et al. (2022), their work only bounds the sample complexity of score matching by a quantity polynomial in the ambient dimension for a specific distribution in a specific bimodal exponential family. By contrast, we can handle an entire class of exponential families with low-degree sufficient statistics.
Poincaré vs Restricted Poincaré: We note that while Poincaré inequalities are directly related to isoperimetry and mixing of Markov chains, sample efficiency of score matching only depends on the Poincaré inequality holding for a restricted class of functions, namely, functions linear in the sufficient statistics. Hence, hardness of sampling only implies sample complexity lower bounds in cases where the family is expressive enough—indeed, the key to exponential lower bounds for score matching in Koehler et al. (2022) is augmenting the sufficient statistics with a function defined by a bad cut. This gap means that we can hope to have good sample complexity for score matching even in cases where sampling is hard—which we take advantage of in this work.
Learning exponential families: Despite the fact that exponential families are both classical and ubiquitous, both in statistics and machine learning, there is relatively little understanding about the computational-statistical tradeoffs to learn them from data, that is, what sample complexity can be achieved with a computationally efficient algorithm. Ren et al. (2021) consider a version of the
"interaction screening" estimator, a close relative of pseudolikelihood, but do not prove anything about the statistical complexity of this estimator. Shah et al. (2021) consider a related estimator, and analyze it under various low-rank and sparsity assumptions of reshapings of the sufficient statistics into a tensor. Unfortunately, these assumptions are somewhat involved, and it's unclear if they are needed for designing computationally and statistically efficient algorithms.
Discrete exponential families (Ising models): Ising models have the form  $p_J(x) \propto \exp \left( \sum_{i \sim j} J_{ij} x_i x_j + \sum_i J_i x_i \right)$  where  $\sim$  denotes adjacency in some (unknown) graph, and  $J_{ij}, J_i$  denote the corresponding pairwise and singleton potentials. Bresler (2015) gave an efficient algorithm for learning any Ising model over a graph with constant degree (and  $l_{\infty}$ -bounds on the coefficients); see also the more recent work (Dagan et al., 2021). In contrast, it is a classic result (Arora and Barak, 2009) that approximating the partition function of members in this family is NP-hard.
Similarly, the exponential family we consider is such that it contains members for which sampling and approximating their partition function is intractable (the main ingredient in the proof of Theorem 1.1). Nevertheless, by Theorem 3, we can learn the parameters for members in this family computationally efficiently, and with sample complexity comparable to the optimal one (achieved by maximum likelihood). This also parallels other developments in Ising models (Bresler et al., 2014; Montanari, 2015), where it is known that restricting the type of learning algorithm (e.g., requiring it to work with sufficient statistics only) can make a tractable problem become intractable.
The parallels can be drawn even on an algorithmic level: a follow up work to Bresler (2015) by Vuffray et al. (2016) showed that similar results can be shown in the Ising model setting by using the "screening estimator", a close relative of the classical pseudolikelihood estimator (Besag, 1977) which tries to learn a distribution by matching the conditional probability of singletons, and thereby avoids having to evaluate a partition function. Since conditional probabilities of singletons capture changes in a single coordinate, they can be viewed as a kind of "discrete gradient"—a further analogy to score matching in the continuous setting.[2]
We consider the following exponential family. Fix positive integers  $n, d, B \in \mathbb{N}$  where  $d$  is odd. Let  $h(x) = \exp(-\sum_{i=1}^{n} x_i^{d+1})$ , and let  $T(x) \in \mathbb{R}^{M-1}$  be the vector of monomials in  $x_1, \ldots, x_n$  of degree at least 1 and at most  $d$  (so that  $M = \binom{n+d}{d}$ ). Define  $\Theta \subseteq \mathbb{R}^{M-1}$  by  $\Theta = \{\theta \in \mathbb{R}^{M-1} : \| \theta \|_{\infty} \leq B\}$ . For any  $\theta \in \Theta$  define  $p_\theta: \mathbb{R}^n \to [0, \infty)$  by
$$
p _ {\theta} (x) := \frac {h (x) \exp (\langle \theta , T (x) \rangle)}{Z _ {\theta}}
$$
where  $Z_{\theta} = \int_{\mathbb{R}^n}h(x)\exp (\langle \theta ,T(x)\rangle)dx$  is the normalizing constant. Then we consider the family  $\mathcal{P}_{n,d,B}\coloneqq (p_{\theta})_{\theta \in \Theta_B}$ . Throughout, we will assume that  $B\geq 1$ .
Polynomial notation: Let  $\mathbb{R}[x_1, \ldots, x_n]_{\leq d}$  denote the space of polynomials in  $x_1, \ldots, x_n$  of degree at most  $d$ . We can write any such polynomial  $f$  as  $f(x) = \sum_{|\mathbf{d}| \leq d} a_{\mathbf{d}} x_{\mathbf{d}}$  where  $\mathbf{d}$  denotes a degree function  $\mathbf{d}: [n] \to \mathbb{N}$ , and  $|\mathbf{d}| = \sum_{i=1}^{n} \mathbf{d}(i)$ , and we write  $x_{\mathbf{d}}$  to denote  $\prod_{i=1}^{n} x_i^{\mathbf{d}(i)}$ . Note that every  $\mathbf{d}$  with  $1 \leq |\mathbf{d}| \leq d$  corresponds to an index of  $T$ , i.e.  $T(x)_{\mathbf{d}} = x_{\mathbf{d}}$ .
Let  $\| \cdot \|_{\mathrm{mon}}$  denote the  $\ell^2$  norm of a polynomial in the monomial basis; that is,  $\| \sum_{\mathbf{d}}a_{\mathbf{d}}x_{\mathbf{d}}\|_{\mathrm{mon}} = \left(\sum_{\mathbf{d}}a_{\mathbf{d}}^{2}\right)^{1 / 2}$ . For any function  $f:\mathbb{R}^n\to \mathbb{R}$ , let  $\| f\|_{L^2 ([-1,1]^n)}^2 = \mathbb{E}_{x\sim \mathrm{Unif}([-1,1]^n)}f(x)^2$ .
Statistical efficiency of MLE: For any  $\theta \in \mathbb{R}^{M - 1}$ , the Fisher information matrix of  $p_{\theta}$  with respect to the sufficient statistics  $T(x)$  is defined as
$$
\mathcal {I} (\theta) := \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) T (x) ^ {\top} ] - \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] ^ {\top}.
$$
It is well-known that for any exponential family with no affine dependencies among the sufficient statistics (see e.g., Theorem 4.6 in Van der Vaart (2000)), it holds that for any  $\theta^{*}\in \mathbb{R}^{M - 1}$ , given  $N$
2In fact, ratio matching, proposed in Hyvarinen (2007) as a discrete analogue of score matching, relies on exactly this intuition.
independent samples  $x^{(1)},\ldots ,x^{(N)}\sim p_{\theta^*}$  , the estimator  $\hat{\theta}_{\mathbf{MLE}} = \hat{\theta}_{\mathbf{MLE}}(x^{(1)},\dots,x^{(N)})$  satisfies
$$
\sqrt {N} \left(\hat {\theta} _ {\mathbf {M L E}} - \theta^ {*}\right)\rightarrow \mathcal {N} (0, \mathcal {I} (\theta^ {*}) ^ {- 1}).
$$
Statistical efficiency of score matching: Our analysis of the statistical efficiency of score matching is based on a result due to Koehler et al. (2022). We state a requisite definition followed by the result.
Definition 2.1 (Restricted Poincaré for exponential families). The restricted Poincaré constant of  $p \in \mathcal{P}_{n,d,B}$  is the smallest  $C_P > 0$  such that for all  $w \in \mathbb{R}^{M - 1}$ , it holds that
$$
\operatorname {V a r} _ {p} (\langle w, T (x) \rangle) \leq C _ {P} \mathbb {E} _ {x \sim p} \| \nabla_ {x} \langle w, T (x) \rangle \| _ {2} ^ {2}.
$$
Theorem 2.2 (Koehler et al. (2022)). Under certain regularity conditions (see Lemma B.4), for any  $p_{\theta^*}$  with restricted Poincaré constant  $C_P$  and with  $\lambda_{\min}(\mathcal{I}(\theta^*)) > 0$ , given  $N$  independent samples  $x^{(1)}, \ldots, x^{(N)} \sim p_{\theta^*}$ , the estimator  $\hat{\theta}_{\mathrm{SM}} = \hat{\theta}_{\mathrm{SM}}(x^{(1)}, \ldots, x^{(N)})$  satisfies
$$
\sqrt {N} (\hat {\theta} _ {\mathbf {S M}} - \theta^ {*}) \rightarrow \mathcal {N} (0, \Gamma)
$$
where  $\Gamma$  satisfies
$$
\| \Gamma \| _ {\mathcal {O P}} \leq \frac {2 C _ {P} ^ {2} \left(\| \theta^ {*} \| _ {2} ^ {2} \mathbb {E} _ {x \sim p _ {\theta^ {*}}} \left\| (J T) (x) \right\| _ {\mathcal {O P}} ^ {4} + \mathbb {E} _ {x \sim p _ {\theta^ {*}}} \| \Delta T (x) \| _ {2} ^ {2}\right)}{\lambda_ {\min } (\mathcal {I} (\theta^ {*})) ^ {2}}
$$
where  $(JT)(x)_i = \nabla_x T_i(x)$  and  $\Delta T(x) = \operatorname{Tr} \nabla_x^2 T(x)$ .
3 Hardness of Implementing Optimization Oracles for  $\mathcal{P}_{n,7,\mathrm{poly}(n)}$
In this section we prove NP-hardness of implementing approximate zeroth-order and first-order optimization oracles for maximum likelihood in the exponential family  $\mathcal{P}_{n,7,Cn^2\log (n)}$  (for a sufficiently large constant  $C$ ) as defined in Section 2; we also show that approximate sampling from this family is NP-hard. See Theorems 3.4, 3.5, and A.5 respectively. All of the hardness results proceed by reduction from 3-SAT and use the same construction.
The idea is that for any formula  $\mathcal{C}$  on  $n$  variables, we can construct a non-negative polynomial  $F_{\mathcal{C}}$  of degree at most 6 in variables  $x_{1},\ldots ,x_{n}$ , which has roots exactly at the points of the hypercube  $\mathcal{H} := \{-1,1\}^n\subseteq \mathbb{R}^n$  that correspond to satisfying assignments (under the bijection that  $x_{i} = 1$  corresponds to True and  $x_{i} = -1$  corresponds to False). Intuitively, the distribution with density proportional to  $\exp (-\gamma F_{\mathcal{C}}(x))$  will, for sufficiently large  $\gamma >0$ , concentrate on the satisfying assignments. It is then straightforward to see that sampling from this distribution or efficiently computing either  $\log Z_{\theta}$  or  $\nabla_{\theta}\log Z_{\theta}$  (where  $\theta \in \mathbb{R}^{M - 1}$  is the parameter vector corresponding to the polynomial  $-\gamma F_{\mathcal{C}}(x)$ ) would enable efficiently finding a satisfying assignment.
The remainder of this section makes the above intuition precise; important details include (1) incorporating the base measure  $h(x) = \exp(-\sum_{i=1}^{n} x_i^8)$  into the density function, and (2) showing that a polynomially-large temperature  $\gamma$  suffices.
Definition 3.1 (Clause/formula polynomials). Given a 3-clause formula of the form  $C = \tilde{x}_i \vee \tilde{x}_j \vee \tilde{x}_k$  where  $\tilde{x}_i = x_i$  or  $\tilde{x}_i = \neg x_i$ , we construct a polynomial  $H_C \in \mathbb{R}[x_1, \ldots, x_n]_{\leq 6}$  defined by
$$
H _ {C} (x) = f _ {i} \left(x _ {i}\right) ^ {2} f _ {j} \left(x _ {j}\right) ^ {2} f _ {k} \left(x _ {k}\right) ^ {2}
$$
$$
f _ {i} (t) = \left\{ \begin{array}{l l} (t + 1) & \text {i f x _ {i} i s n e g a t e d i n C} \\ (t - 1) & \text {o t h e r w i s e} \end{array} \right..
$$
For example, if  $C = x_{1} \vee x_{2} \vee \neg x_{3}$ , then  $H_{C} = (x_{1} - 1)^{2}(x_{2} - 1)^{2}(x_{3} + 1)^{2}$ . Further, given a 3-SAT formula  $\mathcal{C} = C_1 \wedge \dots \wedge C_m$  on  $m$  clauses<sup>3</sup>, we define the polynomial
$$
H _ {\mathcal {C}} (x) = H _ {C _ {1}} (x) + \dots + H _ {C _ {m}} (x).
$$
It can be seen that any  $x \in \mathcal{H}$  corresponds to a satisfying assignment for  $\mathcal{C}$  if and only if  $H_{\mathcal{C}}(x) = 0$ . Note that there are possibly points outside  $\mathcal{H}$  which satisfy  $H_{\mathcal{C}}(x) = 0$ . To avoid these solutions, we introduce another polynomial:
<sup>3</sup>It suffices to work with  $m = O(n)$ , see Theorem A.1.
Definition 3.2 (Hypercube polynomial). We define  $G: \mathbb{R}^n \to \mathbb{R}$  by  $G(x) = \sum_{i=1}^{n} (1 - x_i^2)^2$ .
Note that  $G(x) \geq 0$  for all  $x$ , and the roots of  $G(x)$  are precisely the vertices of  $\mathcal{H}$ . Therefore for any  $\alpha, \beta > 0$ , the roots (in  $\mathbb{R}^n$ ) of the polynomial  $F_{\mathcal{C}}(x) = \alpha H_{\mathcal{C}}(x) + \beta G(x)$  are precisely the vertices of  $\mathcal{H}$  that correspond to satisfying assignments for  $\mathcal{C}$ .
Definition 3.3. Let  $\mathcal{C}$  be a 3-CNF formula with  $n$  variables and  $m$  clauses. Let  $\alpha, \beta > 0$ . Then we define a distribution  $P_{\mathcal{C},\alpha,\beta}$  with density function
$$
p _ {\mathcal {C}, \alpha , \beta} (x) := \frac {h (x) \exp (- \alpha H _ {\mathcal {C}} (x) - \beta G (x))}{Z _ {\mathcal {C} , \alpha , \beta}}
$$
where  $Z_{\mathcal{C},\alpha ,\beta} = \int_{\mathbb{R}^n}h(x)\exp (-\alpha H_{\mathcal{C}}(x) - \beta G(x))dx$