SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This distribution lies in the exponential family  $\mathcal{P}_{n,d,B}$ , for  $d = 7$  and  $B = \Omega (\beta +m\alpha)$  (Lemma A.2). Thus, if  $\theta (\mathcal{C},\alpha ,\beta)$  is the parameter vector that induces  $P_{\mathcal{C},\alpha ,\beta}$ , then it suffices to show that (a) approximating  $\log Z_{\theta (\mathcal{C},\alpha ,\beta)}$ , (b) approximating  $\nabla_{\theta}\log Z_{\theta (\mathcal{C},\alpha ,\beta)}$ , and (c) sampling from  $P_{\mathcal{C},\alpha ,\beta}$  are NP-hard (under randomized reductions). We sketch the proofs below; details are in Appendix A.
Hardness of approximating  $\log Z_{\mathcal{C},\alpha,\beta}$ : In order to prove (a), we bound the mass of  $P_{\mathcal{C},\alpha,\beta}$  in each orthant of  $\mathbb{R}^n$ . In particular, we show that for  $\alpha = \Omega(n)$  and  $\beta = \Omega(m\log m)$ , any orthant corresponding to a satisfying assignment has exponentially larger contribution to  $Z_{\mathcal{C},\alpha,\beta}$  than any orthant corresponding to an unsatisfying assignment. A consequence is that the partition function  $Z_{\mathcal{C},\alpha,\beta}$  is exponentially larger when the formula  $\mathcal{C}$  is satisfiable than when it isn't (Lemma A.6).
But then approximating  $Z_{\mathcal{C},\alpha,\beta}$  allows distinguishing a satisfiable formula from an unsatisfiable formula, which is NP-hard. This implies the following theorem (proof in Section A.2):
Theorem 3.4. Fix  $n \in \mathbb{N}$  and let  $B \geq Cn^2$  for a sufficiently large constant  $C$ . Unless  $RP = NP$ , there is no poly(n)-time algorithm which takes as input an arbitrary  $\theta \in \Theta_B$  and outputs an approximation of  $\log Z_{\theta}$  with additive error less than  $n \log 1.16$ .
Hardness of approximating  $\nabla_{\theta}\log Z_{\theta (\mathcal{C},\alpha ,\beta)}$  : Note that  $\nabla_{\theta}\log Z_{\theta} = \mathbb{E}_{x\sim p_{\theta}}[T(x)]$  , so in particular approximating the gradient yields an approximation to the mean  $\mathbb{E}_{x\sim p_{\theta}}[x]$  . Since  $P_{\mathcal{C},\alpha ,\beta}$  is concentrated in orthants corresponding to satisfying assignments of  $\mathcal{C}$  , we would intuitively expect that if  $\mathcal{C}$  has exactly one satisfying assignment  $v^{*}$  , then sign  $(\mathbb{E}_{p_{\theta}}[x])$  corresponds to this assignment. Formally, we show that if  $\alpha = \Theta (n)$  and  $\beta = \Omega (mn\log m)$  , then  $\mathbb{E}_{x\sim p_{\mathcal{C},\alpha ,\beta}}[v_i^* x_i]\geq 1 / 20$  for all  $i\in [n]$  (Lemma A.7).
Since solving a formula with a unique satisfying assignment is still NP-hard, we get the following theorem (proof in Section A.3):
Theorem 3.5. Fix  $n \in \mathbb{N}$  and let  $B \geq Cn^2\log (n)$  for a sufficiently large constant  $C$ . Unless  $RP = NP$ , there is no poly(n)-time algorithm which takes as input an arbitrary  $\theta \in \Theta_B$  and outputs an approximation of  $\nabla_{\theta}\log Z_{\theta}$  with additive error (in an  $l_{\infty}$  sense) less than 1/20.
With the above two theorems in hand, we are ready to present the formal version of Theorem 1.1; the proof is immediate from the definition of  $L_{\mathrm{MLE}}(\theta)$  (see Section A.5).
Corollary 3.6. Fix  $n, N \in \mathbb{N}$  and let  $B \geq Cn^2 \log n$  for a sufficiently large constant  $C$ . Unless  $RP = NP$ , there is no  $\mathrm{poly}(n, N)$ -time algorithm which takes as input an arbitrary  $\theta \in \Theta_B$ , and an arbitrary sample  $x_1, \ldots, x_N \in \mathbb{R}^n$ , and outputs an approximation of  $L_{\mathrm{MLE}}(\theta)$  up to additive error of  $n \log 1.16$ , or  $\nabla_\theta L_{\mathrm{MLE}}(\theta)$  up to an additive error of  $1/20$ .
Hardness of approximate sampling: We show that for  $\alpha = \Omega (n)$  and  $\beta = \Omega (m\log m)$ , the likelihood that  $x\sim P_{\mathcal{C},\alpha ,\beta}$  lies in an orthant corresponding to a satisfying assignment for  $\mathcal{C}$  is at least  $1 / 2$  (Lemma A.4). Hardness of approximate sampling follows immediately (Theorem A.5). Hence, although we show that score matching can efficiently estimate  $\theta^{*}$  from samples produced by nature, knowing  $\theta^{*}$  isn't enough to efficiently generate samples from the distribution.
4 Statistical Efficiency of Maximum Likelihood
In this section we prove Theorem 1.2 by showing that for any  $\theta \in \Theta_B$ , we can lower bound the smallest eigenvalue of the Fisher information matrix  $\mathcal{I}(\theta)$ . Concretely, we show:
Theorem 4.1. For any  $\theta \in \Theta_B$ , it holds that
$$
\lambda_ {\min } (\mathcal {I} (\theta)) \geq (n B) ^ {- O (d ^ {3})}.
$$
As a corollary, given  $N$  samples from  $p_{\theta}$ , it holds as  $N \to \infty$  that  $\sqrt{N} (\hat{\theta}_{\mathrm{MLE}} - \theta) \to N(0, \Gamma_{\mathrm{MLE}})$  where  $\| \Gamma_{\mathrm{MLE}} \|_{op} \leq (nB)^{O(d^3)}$ . Moreover, for sufficiently large  $N$ , with probability at least 0.99 it holds that  $\left\| \hat{\theta}_{\mathrm{MLE}} - \theta \right\|_2^2 \leq (nB)^{O(d^3)} / N$ .
Once we have the bound on  $\lambda_{\mathrm{min}}(\mathcal{I}(\theta))$ , the first corollary follows from standard bounds for MLE (Section 2), and the second corollary follows from Markov's inequality (see e.g., Remark 4 in Koehler et al. (2022)). Lower-bounding  $\lambda_{\mathrm{min}}(\mathcal{I}(\theta))$  itself requires lower-bounding the variance of any polynomial (with respect to  $p_\theta$ ) in terms of its coefficients. The proof consists of three parts. First, we show that the norm of a polynomial in the monomial basis is upper-bounded in terms of its  $L^2$  norm on  $[-1,1]^n$ :
Lemma 4.2. For  $f \in \mathbb{R}[x_1, \ldots, x_n]_{\leq d}$ , we have  $\| f \|_{mon}^2 \leq \binom{n+d}{d} (16e)^d \| f \|_{L^2([-1,1]^n)}^2$ .
The key idea behind this proof is to work with the basis of (tensorized) Legendre polynomials, which is orthonormal with respect to the  $L^2$  norm. Once we write the polynomial with respect to this basis, the  $L^2$  norm equals the Euclidean norm of the coefficients. Given this observation, all that remains is to bound the coefficients after the change-of-basis. The complete proof is deferred to Appendix C.
Next, we show that if a polynomial  $f: \mathbb{R}^n \to \mathbb{R}$  has small variance with respect to  $p$ , then there is some box on which  $f$  has small variance with respect to the uniform distribution. This provides a way of comparing the variance of  $f$  with its  $L^2$  norm (after an appropriate rescaling).
Lemma 4.3. Fix any  $\theta \in \Theta_B$  and define  $p\coloneqq p_{\theta}$ . Define  $R\coloneqq 2^{d + 8}nBM$ . Then for any  $f\in \mathbb{R}[x_1,\ldots ,x_n]_{\leq d}$ , there is some  $z\in \mathbb{R}^n$  with  $\| z\|_{\infty}\leq R$  and some  $\epsilon \geq 1 / (2(d + 1)MR^{d}(n + B))$  such that
$$
\operatorname {V a r} _ {p} (f) \geq \frac {1}{2 e} \operatorname {V a r} _ {\tilde {\mathcal {U}}} (f),
$$
where  $\widetilde{\mathcal{U}}$  is the uniform distribution on  $\{x\in \mathbb{R}^n:\| x - z\|_{\infty}\leq \epsilon \}$
In order to prove this result, we pick a random box of radius  $\epsilon$  (within a large bounding box of radius  $R$ ). In expectation, the variance on this box (with respect to  $p$ ) is not much less than  $\operatorname{Var}_p(f)$ . Moreover, for sufficiently small  $\epsilon$ , the density function of  $p$  on this box has bounded fluctuations, allowing comparison of  $\operatorname{Var}_p(f)$  and  $\operatorname{Var}_{\tilde{\mathcal{U}}}(\bar{f})$ . This argument is formalized in Appendix C.
Together, Lemma 4.2 and 4.3 allow us to lower bound the variance  $\mathrm{Var}_p(f)$  in terms of  $\| f\|_{\mathrm{mon}}$ .
Lemma 4.4. Fix any  $\theta \in \Theta_B$  and define  $p := p_\theta$ . Define  $R := 2^{d+8}nBM$ . Then for any  $f \in \mathbb{R}[x_1, \ldots, x_n]_{\leq d}$  with  $f(0) = 0$ , it holds that
$$
\operatorname {V a r} _ {p} (f) \geq \frac {1}{2 ^ {2 d} (d + 1) ^ {2 d} (1 6 e) ^ {d + 1} M ^ {2 d + 3} R ^ {2 d ^ {2} + 2 d} (n + B) ^ {2 d}} \| f \| _ {m o n} ^ {2}.
$$
See Appendix C for the proof. We are now ready to finish the proof of Theorem 4.1.
Proof of Theorem 4.1. Fix  $\theta \in \Theta_B$ . Pick any  $w \in \mathbb{R}^M$  and define  $f(x) = \langle w, T(x) \rangle$ . By definition of  $\mathcal{I}(\theta)$ , we have  $\operatorname{Var}_{p_\theta}(f) = w^\top \mathcal{I}(\theta)w$ . Moreover,  $\| f \|_{\mathrm{mon}}^2 = \| w \|_2^2$ . Thus, Lemma 4.4 gives us that  $w^\top \mathcal{I}(\theta)w \geq (nB)^{-O(d^3)} \| w \|_2^2$ , using that  $R = 2^{d + 8} nBM$  and  $M = \binom{n+d}{d}$ . The bound  $\lambda_{\min}(\mathcal{I}(\theta)) \geq (nB)^{-O(d^3)}$  follows.
5 Statistical Efficiency of Score Matching
In this section we prove Theorem 1.3. The main technical ingredient is a bound on the restricted Poincaré constants of distributions in  $\mathcal{P}_{n,d,B}$ . For any fixed  $\theta \in \Theta_B$ , we show that  $C_P$  can be bounded in terms of the condition number of the Fisher information matrix  $\mathcal{I}(\theta)$ . We describe the building blocks of the proof below.
Fix  $\theta, w \in \mathbb{R}^{M-1}$  and define  $f(x) \coloneqq \langle w, T(x) \rangle$ . First, we need to upper bound  $\operatorname{Var}_{p_{\theta}}(f)$ . This is where (the first half of) the condition number appears. Using the crucial fact that the restricted Poincaré constant only considers functions  $f$  that are linear in the sufficient statistics, and the definition of  $\mathcal{I}(\theta)$ , we get the following bound on  $\operatorname{Var}_{p_{\theta}}(f)$  in terms of the coefficient vector  $w$ . The proof is deferred to Section D.
Lemma 5.1. Fix  $\theta, w \in \mathbb{R}^{M-1}$  and define  $f(x) \coloneqq \langle w, T(x) \rangle$ . Then
$$
\| w \| _ {2} ^ {2} \lambda_ {\min} (\mathcal {I} (\theta)) \leq \operatorname {V a r} _ {p _ {\theta}} (f) \leq \| w \| _ {2} ^ {2} \lambda_ {\max} (\mathcal {I} (\theta)).
$$
Next, we lower bound  $\mathbb{E}_{x\sim p_{\theta}}\| \nabla_x f(x)\| _2^2$  . To do so, we could pick an orthonormal basis and bound  $\mathbb{E}\langle u,\nabla_xf(x)\rangle^2$  over all directions  $u$  in the basis; however, it is unclear how to choose this basis. Instead, we pick  $u\sim \mathcal{N}(0,I_n)$  randomly, and use the following identity:
$$
\mathbb {E} _ {x \sim p _ {\theta}} [ \| \nabla_ {x} f (x) \| _ {2} ^ {2} ] = \mathbb {E} _ {x \sim p _ {\theta}} \mathbb {E} _ {u \sim N (0, I _ {n})} \langle u, \nabla_ {x} f (x) \rangle^ {2}
$$
For any fixed  $u$ , the function  $g(x) = \langle u, \nabla_x f(x) \rangle$  is also a polynomial. If this polynomial had no constant coefficient, we could immediately lower bound  $\mathbb{E} \langle u, \nabla_x f(x) \rangle^2$  in terms of the remaining coefficients, as above. Of course, it may have a nonzero constant coefficient, but with some case-work over the value of the constant, we can still prove the following bound:
Lemma 5.2. Fix  $\theta, \tilde{w} \in \mathbb{R}^{M-1}$  and  $c \in \mathbb{R}$ , and define  $g(x) := \langle \tilde{w}, T(x) \rangle + c$ . Then
$$
\mathbb {E} _ {x \sim p _ {\theta}} [ g (x) ^ {2} ] \geq \frac {c ^ {2} + \| \tilde {w} \| _ {2} ^ {2}}{4 + 4 \| \mathbb {E} [ T (x) ] \| _ {2} ^ {2}} \min  (1, \lambda_ {\operatorname* {m i n}} (\mathcal {I} (\theta))).
$$
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} [ g (x) ^ {2} ] = \mathrm {V a r} _ {p _ {\theta}} (g) + \mathbb {E} _ {x \sim p _ {\theta}} [ g (x) ] ^ {2} \\ = \operatorname {V a r} _ {p _ {\theta}} (g - c) + (c + \tilde {w} ^ {\top} \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ]) ^ {2} \\ \geq \| \tilde {w} \| _ {2} ^ {2} \lambda_ {\min} (\mathcal {I} (\theta)) + (c + \tilde {w} ^ {\top} \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ]) ^ {2} \\ \end{array}
$$
where the inequality is by Lemma 5.1. We now distinguish two cases.
Case I. Suppose that  $|c + \tilde{w}^{\top}\mathbb{E}_{x\sim p_{\theta}}[T(x)]|\geq c / 2$  . Then
$$
\mathbb {E} _ {x \sim p _ {\theta}} [ g (x) ^ {2} ] \geq \| \tilde {w} \| _ {2} ^ {2} \lambda_ {\min } (\mathcal {I} (\theta)) + \frac {c ^ {2}}{4} \geq \frac {c ^ {2} + \| \tilde {w} \| _ {2} ^ {2}}{4} \min (1, \lambda_ {\min } (\mathcal {I} (\theta))).
$$
Case II. Otherwise, we have  $|c + \tilde{w}^{\top}\mathbb{E}_{x\sim p_{\theta}}[T(x)]| < c / 2$ . By the triangle inequality, it follows that  $|\tilde{w}^{\top}\mathbb{E}_{x\sim p_{\theta}}[T(x)]|\geq c / 2$ , so  $\| \tilde{w}\| _2\geq c / (2\| \mathbb{E}_{x\sim p_\theta}[T(x)]\| _2)$ . Therefore
$$
c ^ {2} + \| \tilde {w} \| _ {2} ^ {2} \leq \left(1 + 4 \left\| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \right\| _ {2} ^ {2}\right) \| \tilde {w} \| _ {2} ^ {2},
$$
from which we get that
$$
\mathbb {E} _ {x \sim p _ {\theta}} [ g (x) ^ {2} ] \geq \| \tilde {w} \| _ {2} ^ {2} \lambda_ {\min } (\mathcal {I} (\theta)) \geq \frac {c ^ {2} + \| \tilde {w} \| _ {2} ^ {2}}{1 + 4 \| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \| _ {2} ^ {2}} \lambda_ {\min } (\mathcal {I} (\theta))
$$
With Lemma 5.1 and Lemma 5.2 in hand (taking  $g(x) = \langle u, \nabla_x f(x) \rangle$  in the latter), all that remains is to relate the squared monomial norm of  $\langle u, \nabla_x f(x) \rangle$  (in expectation over  $u$ ) to the squared monomial norm of  $f$ . This crucially uses the choice  $u \sim N(0, I_n)$ . We put together the pieces in the following lemma. The detailed proof is provided in Section D.
Lemma 5.3. Fix  $\theta, w \in \mathbb{R}^{M-1}$ . Define  $f(x) \coloneqq \langle w, T(x) \rangle$ . Then
$$
\operatorname {V a r} _ {p _ {\theta}} (f) \leq \left(4 + 4 \| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \| _ {2} ^ {2}\right) \frac {\lambda_ {m a x} (\mathcal {I} (\theta))}{\min  \left(1 , \lambda_ {m i n} (\mathcal {I} (\theta))\right)} \mathbb {E} _ {x \sim p _ {\theta}} [ \| \nabla_ {x} f (x) \| _ {2} ^ {2} ].
$$
Finally, putting together Lemma 5.3, Theorem 4.1 (that lower bounds  $\lambda_{\min}(\mathcal{I}(\theta))$ ), and Lemma B.2 (that upper bounds  $\lambda_{\max}(\mathcal{I}(\theta))$  - a straightforward consequence of the distributions in  $\mathcal{P}_{n,d,B}$  having bounded moments), we can prove the following formal version of Theorem 1.3:
Theorem 5.4. Fix  $n, d, B, N \in \mathbb{N}$ . Pick any  $\theta^{*} \in \Theta_{B}$  and let  $x^{(1)}, \ldots, x^{(N)} \sim p_{\theta^{*}}$  be independent samples. Then as  $N \to \infty$ , the score matching estimator  $\hat{\theta}_{\mathbf{SM}} = \hat{\theta}_{\mathbf{SM}}(x^{(1)}, \ldots, x^{(N)})$  satisfies
$$
\sqrt {N} (\hat {\theta} _ {\mathbf {S M}} - \theta^ {*}) \rightarrow N (0, \Gamma)
$$
where  $\| \Gamma \|_{op} \leq (nB)^{O(d^3)}$ . As a corollary, for all sufficiently large  $N$  it holds with probability at least 0.99 that  $\left\| \hat{\theta}_{\mathbf{SM}} - \theta^{*}\right\|_{2}^{2} \leq (nB)^{O(d^{3})} / N$ .
Proof. We apply Theorem 2.2. By Lemma B.4 and the fact that  $\lambda_{\min}(I(\theta^{*})) > 0$  (Theorem 4.1), the necessary regularity conditions are satisfied so that the score matching estimator is consistent and asymptotically normal, with asymptotic covariance  $\Gamma$  satisfying
$$
\left\| \Gamma \right\| _ {\mathfrak {o p}} \leq \frac {2 C _ {P} ^ {2} \left(\| \theta \| _ {2} ^ {2} \mathbb {E} _ {x \sim p _ {\theta^ {*}}} \| (J T) (x) \| _ {\mathfrak {o p}} ^ {4} + \mathbb {E} _ {x \sim p _ {\theta^ {*}}} \| \Delta T (x) \| _ {2} ^ {2}\right)}{\lambda_ {\min } \left(\mathcal {I} \left(\theta^ {*}\right)\right) ^ {2}} \tag {4}
$$
where  $C_P$  is the restricted Poincaré constant for  $p_{\theta^*}$  with respect to linear functions in  $T(x)$  (see Definition 2.1). By Lemma 5.3, we have
$$
\begin{array}{l} C _ {P} \leq (4 + 4 \| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \| _ {2} ^ {2}) \frac {\lambda_ {\max } (\mathcal {I} \left(\theta^ {*}\right))}{\min  \left(1 , \lambda_ {\min } (\mathcal {I} \left(\theta^ {*}\right)\right)} \\ \leq \left(4 + 4 B ^ {2 d} M ^ {2 d + 2} 2 ^ {2 d (d + 6) + 1}\right) \frac {B ^ {2 d} M ^ {2 d + 1} 2 ^ {2 d (d + 6) + 1}}{(n B) ^ {- O (d ^ {3})}} \leq (n B) ^ {O (d ^ {3})} \\ \end{array}
$$
using parts (a) and (b) of Lemma B.2; Theorem 4.1; and the fact that  $M = \binom{n+d}{d}$ . Substituting into (4) and bounding the remaining terms using Lemma B.3 and a second application of Theorem 4.1, we conclude that  $\| \Gamma \|_{\mathrm{op}} \leq (nB)^{O(d^3)}$  as claimed. The high-probability bound now follows from Markov's inequality; see Remark 4 in Koehler et al. (2022) for details.
We have provided a concrete example of an exponential family—namely, exponentials of bounded degree polynomials—where score matching is significantly more computationally efficient than maximum likelihood estimation (through optimization with a zero- or first-order oracle), while still achieving the same sample efficiency up to polynomial factors. While score matching was designed to be more computationally efficient for exponential families, the determination of statistical complexity is more challenging, and we give the first separation between these two methods for a general class of functions.
As we have restricted our attention to the asymptotic behavior of both of the methods, an interesting future direction is to see how the finite sample complexities differ. One could also give a more fine-grained comparison between the polynomial dependencies of score matching and MLE, which we have not attempted to optimize. Finally, it would be interesting to relate our results with similar results and algorithms for learning Ising and higher-order spin glass models in the discrete setting, and give a more unified treatment of pseudo-likelihood or score/ratio matching algorithms in these different settings.
CP is supported by Moses Charikar and Greg Valiant's Simons Investigator Awards. DR is supported in part by a U.S. DoD NDSEG Fellowship. AS is supported in part by NSF grants DMS-2054503 and 2238125. AM is supported by a grant from the ONR, NSF Award 1918656 and a David and Lucile Packard Fellowship. AR is supported in part by NSF awards IIS-2211907, CCF-2238523, and an Amazon Research Award.
Theorem A.1 (Valiant and Vazirani (1985); Cook (1971)). Suppose that there is a randomized poly(n)-time algorithm for the following problem: given a 3-CNF formula  $\mathcal{C}$  with  $n$  variables and at most  $5n$  clauses, under the promise that  $\mathcal{C}$  has at most one satisfying assignment, determine whether  $\mathcal{C}$  is satisfiable. Then,  $NP = RP$ .
Lemma A.2. In the setting of Definition 3.3, set  $d \coloneqq 7$  and  $B \coloneqq 64m\alpha + 2\beta$ . Then  $p_{\mathcal{C},\alpha,\beta} \in \mathcal{P}_{n,d,B}$ .
Proof. Since  $\alpha H_{\mathcal{C}}(x) + \beta G(x)$  is a polynomial in  $x_1, \ldots, x_n$  of degree at most 7, there is some  $\theta = \theta(\mathcal{C}, \alpha, \beta) \in \mathbb{R}^{M-1}$  such that  $\langle \theta, T(x) \rangle + \alpha H_{\mathcal{C}}(x) + \beta G(x)$  is a constant independent of  $x$ . Then  $h(x) \exp(-\alpha H_{\mathcal{C}}(x) - \beta G(x))$  is proportional to  $h(x) \exp(\langle \theta, T(x) \rangle)$ , so  $p_{\mathcal{C}, \alpha, \beta} = p_\theta$ . Moreover, for any clause  $C_j$ , every monomial of  $H_{C_j}$  has coefficient at most 64 in absolute value, so every monomial of  $H_{\mathcal{C}}$  has coefficient at most  $64m$ . Similarly, every monomial of  $G$  has coefficient at most 2 in absolute value. Thus,  $\| \theta \|_{\infty} \leq 64m\alpha + 2\beta =: B$ , so  $p_{\mathcal{C}, \alpha, \beta} \in \mathcal{P}_{n,d,B}$ .
Given a point  $v \in \mathcal{H}$ , let  $\mathcal{O}(v) \coloneqq \{x \in \mathbb{R}^n : x_i v_i \geq 0; \forall i \in [n]\}$  denote the octant containing  $v$ , and let  $\mathcal{B}_r(v) \coloneqq \{x \in \mathbb{R}^n : \| x - v \|_{\infty} \leq r\}$  denote the ball of radius  $r$  with respect to  $\ell_{\infty}$  norm.
Lemma A.3. Let  $p \coloneqq p_{\mathcal{C},\alpha,\beta}$  and  $Z \coloneqq Z_{\mathcal{C},\alpha,\beta}$  for some 3- CNF C with m clauses and n variables, and some parameters  $\alpha, \beta > 0$ . Let  $r \in (0,1)$ . If  $\beta \geq 40r^{-2}\log(4n/r)$ , then for any  $v \in \mathcal{H}$  that is a satisfying assignment for  $\mathcal{C}$ ,
$$
\operatorname * {P r} _ {x \sim p} \left(x \in \mathcal {B} _ {r} (v)\right) \geq \frac {e ^ {- 1 - 8 1 m \alpha r ^ {2}}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n}.
$$
For any  $w\in \mathcal{H}$  that is not a satisfying assignment for  $\mathcal{C}$
$$
\operatorname * {P r} _ {x \sim p} (x \in \mathcal {O} (w)) \leq \frac {e ^ {- \alpha}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n}.
$$
Proof. We begin by lower bounding the probability over  $\mathcal{B}_r(v)$ . Pick any clause  $C_\ell$  included in  $\mathcal{C}$ . We claim that  $H_{C_\ell}(v') \leq 81r^2$  for all  $v' \in \mathcal{B}_r(v)$ . Indeed, say that  $C_\ell = \tilde{x}_i \vee \tilde{x}_j \vee \tilde{x}_k$ . Since  $v$  satisfies  $C_\ell$ , at least one of  $\{f_i(v_i), f_j(v_j), f_k(v_k)\}$  must be zero. Without loss of generality, say that  $f_i(v_i) = 0$ ; also observe that  $|f_j(v_j)|, |f_k(v_k)| \leq 2$ . It follows that for any  $v' \in \mathcal{B}_r(v), |f_i(v_i')| \leq r$  and  $|f_j(v_j')|, |f_j(v_k')| \leq 2 + r \leq 3$  (since  $r \leq 1$ ). Therefore, we have
$$
H _ {C _ {\ell}} \left(v ^ {\prime}\right) \leq r ^ {2} \cdot (3) ^ {2} \cdot (3) ^ {2} = 8 1 r ^ {2}.
$$
Summing over all  $m$  possible clauses, we have  $H_{\mathcal{C}}(v') \leq 81mr^2$  for all  $v' \in \mathcal{B}_r(v)$ . Hence,
$$
\begin{array}{l} \operatorname * {P r} _ {x \sim p} (x \in \mathcal {B} _ {r} (v)) = \frac {1}{Z} \int_ {\mathcal {B} _ {r} (v)} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \alpha H _ {\mathcal {C}} (x) - \beta G (x)\right) d x \\ \geq \frac {e ^ {- 8 1 m \alpha r ^ {2}}}{Z} \int_ {\mathcal {B} _ {r} (v)} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \beta G (x)\right) d x \\ = \frac {e ^ {- 8 1 m \alpha r ^ {2}}}{Z} \left(\int_ {1 - r} ^ {1 + r} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} \\ \geq \frac {e ^ {- 8 1 m \alpha r ^ {2}}}{Z} \left(1 + \frac {1}{n}\right) ^ {- n} \left(\int_ {0} ^ {\infty} \exp \left(- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}\right) d x\right) ^ {n} \tag {5} \\ \geq \frac {e ^ {- 1 - 8 1 m \alpha r ^ {2}}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} \\ \end{array}
$$
where the second inequality (5) is by Lemma A.8. Next, we upper bound the probability over  $\mathcal{O}(w)$ . Let  $C_{\ell}$  be any clause in  $\mathcal{C}$  that is not satisfied by  $w$ . Say that  $C_{\ell} = \tilde{x}_i \vee \tilde{x}_j \vee \tilde{x}_k$ . Then  $|f_i(w_i)| = |f_j(w_j)| = |f_k(w_k)| = 2$ . Furthermore, for any  $w' \in \mathcal{O}^d(w)$ , we have
$|f_i(w_i')|, |f_j(w_j')|, |f_k(w_k')| \geq 1$ , and hence  $H_{C_\ell}(w') \geq 1$ . Since  $H_{C'}(x) \geq 0$  for all  $x, C'$ , we conclude that  $H_{\mathcal{C}}(w') \geq H_{C_\ell}(w') \geq 1$  for all  $w' \in \mathcal{O}(w)$ . In particular, this gives us
$$
\begin{array}{l} \operatorname * {P r} _ {x \sim p} (x \in \mathcal {O} (w)) = \frac {1}{Z} \int_ {\mathcal {O} (w)} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \alpha H _ {\mathcal {C}} (x) - \beta G (x)\right) d x \\ \leq \frac {e ^ {- \alpha}}{Z} \int_ {\mathcal {O} (w)} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \beta G (x)\right) d x \\ = \frac {e ^ {- \alpha}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} \\ \end{array}
$$
A.1 Hardness of approximate sampling
Lemma A.4. Let  $\mathcal{C}$  be a satisfiable instance of 3-SAT with  $m$  clauses and  $n$  variables. Let  $\alpha, \beta > 0$  satisfy  $\alpha \geq 2(n + 1)$  and  $\beta \geq 6480m\log(13n\sqrt{m})$ . Set  $p := p_{\mathcal{C},\alpha,\beta}$  and  $Z := Z_{\mathcal{C},\alpha,\beta}$ . If  $\mathcal{V} \subseteq \mathcal{H}$  is the set of satisfiable assignments for  $\mathcal{C}$ , then
$$
\sum_ {v \in \mathcal {V}} \Pr_ {x \sim p} (x \in \mathcal {O} (v)) \geq \frac {1}{2}.
$$
Proof. Let  $v \in \mathcal{H}$  be any assignment that satisfies  $\mathcal{C}$ , and let  $w \in \mathcal{H}$  be any assignment that does not satisfy  $\mathcal{C}$ . By Lemma A.3 with  $r = 1 / \sqrt{162m}$ , we have