SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Pr_ {x \sim p _ {\mathcal {C}}} (x \in \mathcal {O} (v)) \geq \Pr_ {x \sim p _ {\mathcal {C}}} (x \in B _ {r} (v)) \\ \geq \frac {e ^ {- 1 - \alpha / 2}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} \\ \geq e ^ {- 1 + \alpha / 2} \Pr (x \in \mathcal {O} (w)). \\ \end{array}
$$
Since we chose  $\alpha$  sufficiently large that  $e^{-1 + \alpha /2}\geq 2^n$ , we get that
$$
\operatorname *{Pr}_{x\sim p_{\mathcal{C}}}\left(x\in \mathcal{O}(v)\right)\geq \sum_{w\in \mathcal{H}\setminus \mathcal{V}}\operatorname *{Pr}_{x\sim p_{\mathcal{C}}}\left(x\in \mathcal{O}(w)\right).
$$
$$
\sum_ {v \in \mathcal {V}} \Pr_ {x \sim p _ {\mathcal {C}}} (x \in \mathcal {O} (v)) \geq \sum_ {w \in \mathcal {H} \backslash \mathcal {V}} \Pr_ {x \sim p _ {\mathcal {C}}} (x \in \mathcal {O} (w)) = 1 - \sum_ {v \in \mathcal {V}} \Pr_ {x \sim p _ {\mathcal {C}}} (x \in \mathcal {O} (v)).
$$
The lemma statement follows.
Theorem A.5. Let  $B \geq Cn^2$  for a sufficiently large constant  $C$ . Unless  $RP = NP$ , there is no algorithm which takes as input an arbitrary  $\theta \in \Theta_B$  and outputs a sample from a distribution  $Q$  with  $\mathrm{TV}(P_{\theta}, Q) \leq 1/3$  in  $\mathrm{poly}(n)$  time.
Proof. Suppose that such an algorithm exists. For each  $n \in \mathbb{N}$  define  $\alpha = 2(n + 1)$  and  $\beta = 32400n\log(13n\sqrt{5n})$ . Given a 3-CNF formula  $\mathcal{C}$  with  $n$  variables and at most  $5n$  clauses, we can compute  $\theta = \theta(\mathcal{C}, \alpha, \beta)$ . By Lemma A.2 we have  $\theta \in \Theta_B$  so long as  $B \geq Cn^2$  for a sufficiently large constant  $C$ . Thus, by assumption we can generate a sample from a distribution  $Q$  with  $\mathrm{TV}(P_{\mathcal{C}, \alpha, \beta}, Q) \leq 1/3$ . But by Lemma A.4, we have  $\operatorname*{Pr}_{x \sim P_{\mathcal{C}, \alpha, \beta}}[\mathrm{sign}(x) \text{ satisfies } \mathcal{C}] \geq 1/2$ . Thus,  $\operatorname*{Pr}_{x \sim Q}[\mathrm{sign}(x) \text{ satisfies } \mathcal{C}] \geq 1/6$ . It follows that we can find a satisfying assignment with  $O(1)$  invocations of the sampling algorithm in expectation. By Theorem A.1 we get  $\mathsf{NP} = \mathsf{RP}$ .
A.2 Hardness of approximating zeroth-order oracle
Lemma A.6. Fix  $n, m \in \mathbb{N}$  and let  $\alpha \geq 2(n + 1)$  and  $\beta \geq 6480m\log (13n\sqrt{m})$ . There is a constant  $A = A(n,m,\alpha ,\beta)$  so that the following hold for every 3-CNF formula  $\mathcal{C}$  with  $n$  variables and  $m$  clauses:
- If  $\mathcal{C}$  is unsatisfiable, then  $Z_{\mathcal{C},\alpha,\beta} \leq A$
- If  $\mathcal{C}$  is satisfiable, then  $Z_{\mathcal{C},\alpha,\beta} \geq (e/2)^n A$ .
Proof. If  $\mathcal{C}$  is unsatisfiable, then by the second part of Lemma A.3, we have
$$
Z = Z \sum_ {w \in \mathcal {H}} \operatorname * {P r} _ {x \sim p} (x \in \mathcal {O} (w)) \leq 2 ^ {n} e ^ {- \alpha} \left(\int_ {0} ^ {\infty} \exp (- x ^ {d + 1} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} =: A _ {\mathrm {u n s a t}}.
$$
On the other hand, if  $\mathcal{C}$  is satisfiable, then by the first part of Lemma A.3 with  $r = 1 / \sqrt{162m}$ ,
$$
Z \geq Z \Pr_ {x \sim p} (x \in \mathcal {B} _ {r} (v)) \geq e ^ {- 1 - \alpha / 2} \left(\int_ {0} ^ {\infty} \exp (- x ^ {d + 1} - \beta (1 - x ^ {2}) ^ {2}) d x\right) ^ {n} =: A _ {\mathrm {s a t}}.
$$
Since  $\alpha \geq 2(n + 1)$ , we get
$$
A _ {\text {u n s a t}} \leq (2 / e) ^ {n} A _ {\text {s a t}}
$$
Proof of Theorem 3.4. First, observe that the following problem is NP-hard (under randomized reductions): given two 3-CNF formulas  $\mathcal{C},\mathcal{C}'$  each with  $n$  variables and at most  $10n$  clauses, where it is promised that exactly one of the formulas is satisfiable, determine which of the formulas is satisfiable. Indeed, this follows from Theorem A.1: given a 3-CNF formula  $\mathcal{C}$  with  $n$  variables, at most  $5n$  clauses, and at most one satisfying assignment, consider adjoining either the clause  $x_{i}$  or the clause  $\neg x_{i}$  to  $\mathcal{C}$ . If  $\mathcal{C}$  has a satisfying assignment  $v^{*}$ , then exactly one of the resulting formulas is satisfiable, and determining which one is satisfiable identifies  $v_{i}^{*}$ . Repeating this procedure for all  $i\in [n]$  yields an assignment  $v$ , which satisfies  $\mathcal{C}$  if and only if  $\mathcal{C}$  is satisfiable.
For each  $n \in \mathbb{N}$  define  $\alpha = 2(n + 1)$  and  $\beta = 64800n\log (13n\sqrt{10n})$ . Let  $B > 0$  be chosen later. Suppose that there is a poly  $(n)$ -time algorithm which, given  $\theta \in \Theta_B$ , computes an approximation of  $\log Z_{\theta}$  with additive error less than  $n \log 1.16$ . Then given two formulas  $\mathcal{C}$  and  $\mathcal{C}'$  with  $n$  variables and at most  $10n$  clauses each, we can compute  $\theta = \theta(\mathcal{C}, \alpha, \beta)$  and  $\theta' = \theta(\mathcal{C}', \alpha, \beta)$ . By Lemma A.2, we have  $\theta, \theta' \in \Theta_B$  so long as  $B \geq Cn^2$  for a sufficiently large constant  $C$ . Hence by assumption we can compute approximations  $\tilde{Z}_{\theta}$  and  $\tilde{Z}_{\theta'}$  of  $Z_{\theta}$  and  $Z_{\theta'}$  respectively, with multiplicative error less than  $1.16^n$ . However, by Lemma A.6 and the assumption that exactly one of  $\mathcal{C}$  and  $\mathcal{C}'$  is satisfiable, we know that  $\tilde{Z}_{\theta} > \tilde{Z}_{\theta'}$  if and only if  $\mathcal{C}$  is satisfiable. Thus,  $\mathsf{NP} = \mathsf{RP}$ .
A.3 Hardness of approximating first-order oracle
Lemma A.7. Let  $\mathcal{C}$  be a 3-CNF formula with  $m$  clauses and  $n$  variables, and exactly one satisfying assignment  $v^{*} \in \mathcal{H}$ . Let  $\alpha = 4n$  and  $\beta \geq 25920mn\log(102n\sqrt{mn})$ , and define  $p := p_{\mathcal{C},\alpha,\beta}$  and  $Z := Z_{\mathcal{C},\alpha,\beta}$ . Then  $\mathbb{E}_{x \sim p}[v_i^* x_i] \geq 1/20$  for all  $i \in [n]$ .
Proof. Without loss of generality take  $i = 1$  and  $v_{i}^{*} = 1$ . Set  $r = 1 / (\sqrt{648mn})$ ,  $\alpha = 4n$ , and  $\beta \geq 40r^{-2}\log (4n / r)$ . We want to show that  $\mathbb{E}_{x\sim p}[x_1]\geq 1 / 20$ . We can write
$$
\begin{array}{l} \mathbb {E} \left[ x _ {1} \right] = \mathbb {E} \left[ x _ {1} \mathbb {1} \left[ x \in B _ {r} \left(v ^ {*}\right) \right] \right] + \mathbb {E} \left[ x _ {1} \mathbb {1} \left[ x \in \mathcal {O} \left(v ^ {*}\right) \backslash B _ {r} \left(v ^ {*}\right) \right] \right] + \sum_ {v \in \mathcal {H} \backslash \{v ^ {*} \}} \mathbb {E} \left[ x _ {1} \mathbb {1} \left[ x \in \mathcal {O} (v) \right] \right] \\ \geq (1 - r) \Pr \left[ x \in B _ {r} \left(v ^ {*}\right) \right] - 2 ^ {n} \max  _ {v \in \mathcal {H} \backslash \{v ^ {*} \}} \mathbb {E} \left[ \left| x _ {1} \right| \mathbb {1} [ x \in \mathcal {O} (v) ] \right] \tag {6} \\ \end{array}
$$
since  $x_{1} \geq 1 - r$  for  $x \in B_r(v^*)$  and  $x_{1} \geq 0$  for  $x \in \mathcal{O}(v^{*})$ . Now observe that on the one hand,
$$
\Pr \left(x \in B _ {r} \left(v ^ {*}\right)\right) \geq \frac {e ^ {- 1 - 8 1 m \alpha r ^ {2}}}{Z} \left(\int_ {0} ^ {\infty} \exp \left(- x ^ {*} - \beta g (x)\right) d x\right) ^ {n} \tag {7}
$$
by Lemma A.3. On the other hand, for any  $v \in \mathcal{H} \setminus \{v^*\}$ ,
$$
\begin{array}{l} \mathbb {E} [ | x _ {1} | \mathbb {1} [ x \in \mathcal {O} (v) ] ] = \frac {1}{Z} \int_ {\mathcal {O} (v)} | x _ {1} | \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \alpha H (x) - \beta G (x)\right) d x \\ \leq \frac {e ^ {- \alpha}}{Z} \int_ {\mathcal {O} (v)} | x _ {1} | \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \beta G (x)\right) d x \\ = \frac {e ^ {- \alpha}}{Z} \left(\int_ {0} ^ {\infty} x \exp (- x ^ {8} - \beta g (x)) d x\right) \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta g (x)) d x\right) ^ {n - 1} \\ \leq \frac {2 e ^ {- \alpha}}{Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta g (x)) d x\right) ^ {n} \tag {8} \\ \end{array}
$$
where the second inequality is by Lemma A.9 with  $k = 1$ . Combining (7) and (8) with (6), we have
$$
\begin{array}{l} \mathbb {E} \left[ x _ {1} \right] \geq \frac {(1 - r) e ^ {- 1 - 8 1 m \alpha r ^ {2}} - 2 ^ {n + 1} e ^ {- \alpha}}{Z} \left(\int_ {0} ^ {\infty} \exp \left(- x ^ {8} - \beta g (x)\right) d x\right) ^ {n} \\ \geq \frac {1}{1 0 Z} \left(\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta g (x)) d x\right) ^ {n} \\ \geq \frac {1}{1 0 Z} \int_ {\mathcal {O} (v ^ {*})} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {8} - \alpha H (x) - \beta G (x)\right) d x \\ = \frac {1}{1 0} \Pr \left[ x \in \mathcal {O} \left(v ^ {*}\right) \right] \\ \geq \frac {1}{2 0} \\ \end{array}
$$
where the second inequality is by choice of  $\alpha$  and  $r$ ; the third inequality is by nonnegativity of  $H(x)$ ; and the fourth inequality is by Lemma A.4 and uniqueness of the satisfying assignment  $v^{*}$ .
Proof of Theorem 3.5. Suppose that such an algorithm exists. Set  $\alpha = 4n$  and  $\beta = 129600n^2\log (102n^2\sqrt{5})$ . Given a 3-CNF formula  $\mathcal{C}$  with  $n$  variables, at most  $5n$  clauses, and exactly one satisfying assignment  $v^{*} \in \mathcal{H}$ , we can compute  $\theta = \theta(\mathcal{C}, \alpha, \beta)$ . Let  $E \in \mathbb{R}^n$  be the algorithm's estimate of  $\nabla_{\theta}\log Z_{\theta} = \mathbb{E}_{x \sim p_{\mathcal{C}}, \alpha, \beta}T(x)$ . Then  $\left\|E - \mathbb{E}_{x \sim p_{\mathcal{C}}, \alpha, \beta}T(x)\right\|_{\infty} < 1/20$ . But by Lemma A.7, for each  $i \in [n]$ , the  $i$ -th entry of  $\mathbb{E}_{x \sim p_{\mathcal{C}}, \alpha, \beta}T(x)$ , which corresponds to the monomial  $x_i$ , has sign  $v_i^*$  and magnitude at least  $1/20$ . Thus,  $\mathrm{sign}(E_i) = v_i^*$ . So we can compute  $v^*$  in polynomial time. By Theorem A.1, it follows that  $\mathsf{NP} = \mathsf{RP}$ .
A.4 Integral bounds
Lemma A.8. Fix  $\beta > 150$  and  $\gamma \in [0,1]$ . Define  $f: \mathbb{R} \to \mathbb{R}$  by  $f(x) = \gamma x^8 + \beta (1 - x^2)^2$ . Pick any  $r \in (6/\beta, 0.04)$ . Then
$$
\int_ {0} ^ {\infty} \exp (- f (x)) d x \leq \left(\frac {1}{1 - \exp (- \beta r ^ {2} / 8)} + \frac {2 \exp (- \beta r / 4 0)}{r}\right) \int_ {1 - r} ^ {1 + r} \exp (- f (x)) d x.
$$
In particular, for any  $m \in \mathbb{N}$ , if  $\beta \geq 40r^{-2} \log(4m / r)$ , then
$$
\int_ {0} ^ {\infty} \exp (- f (x)) d x \leq \left(1 + \frac {1}{m}\right) \int_ {1 - r} ^ {1 + r} \exp (- f (x)) d x.
$$
Proof. Set  $a = 1 / \sqrt{2}$ . For any  $x \in [a, \infty)$  we have  $f''(x) = 56\gamma x^6 - 4\beta + 12\beta x^2 \geq \beta > 0$  for  $\beta > 150$ . Thus,  $f$  has at most one critical point in  $[a, \infty)$ ; call this point  $t_0$ . Since  $f'(x) = 8\gamma x^7 - 4\beta x(1 - x^2)$ , we have  $f'(1) = 8\gamma \geq 0$  and  $f'(1 - 3/\beta) \leq 8 - 4\beta (1 - 3/\beta)(3/\beta)(2 - 3/\beta) < 0$ . Thus,  $t_0 \in (1 - 3/\beta, 1]$ . Set  $r' = r - 3/\beta \geq r/2$ . Then
$$
\int_ {1 - r} ^ {1 + r} \exp (- f (x)) d x \geq \int_ {t _ {0} - r ^ {\prime}} ^ {t _ {0} + r ^ {\prime}} \exp (- f (x)) d x.
$$
For every  $t \in \mathbb{R}$  define  $I(t) = \int_t^{t + r'} \exp(-f(x)) dx$ . Since  $f$  is  $\beta$ -strongly convex on  $[a, \infty)$ , we have for any  $t \geq t_0$  that
$$
f (t + r ^ {\prime}) - f (t) \geq r ^ {\prime} f ^ {\prime} (t) + \frac {r ^ {\prime 2}}{2} \beta \geq \frac {r ^ {\prime 2}}{2} \beta
$$
where the final inequality is because  $f'(t) \geq 0$  for  $t \in [t_0, \infty)$ . Thus, for any  $t \geq t_0$ ,
$$
I (t + r ^ {\prime}) = \int_ {t + r ^ {\prime}} ^ {t + 2 r ^ {\prime}} \exp (- f (x)) d x = \int_ {t} ^ {t + r} \exp (- f (x + r ^ {\prime})) d x \leq \exp (- \beta r ^ {\prime 2} / 2) I (t).
$$
By induction, for any  $k \in \mathbb{N}$  it holds that  $I(t_0 + kr') \leq \exp(-\beta kr'^2 / 2)I(t_0)$ , so
$$
\int_ {t _ {0}} ^ {\infty} \exp (- f (x)) d x = \sum_ {k = 0} ^ {\infty} I \left(t _ {0} + k r ^ {\prime}\right) \leq I \left(t _ {0}\right) \sum_ {k = 0} ^ {\infty} \exp \left(- \beta k r ^ {\prime 2} / 2\right) = \frac {I \left(t _ {0}\right)}{1 - \exp \left(- \beta r ^ {\prime 2} / 2\right)}. \tag {9}
$$
Similarly, for any  $t \in [a + r', t_0]$ , we have
$$
f (t - r ^ {\prime}) - f (t) \geq - r ^ {\prime} f ^ {\prime} (t) + \frac {r ^ {\prime 2}}{2} \beta \geq \frac {r ^ {\prime 2}}{2} \beta
$$
using  $\beta$ -strong convexity on  $[a, \infty)$  and the bound  $f'(t) \leq 0$  on  $[a, t_0]$ . Thus, for any  $t \in [a, t_0 - r']$ ,
$$
I (t - r ^ {\prime}) = \int_ {t - r ^ {\prime}} ^ {t} \exp (- f (x)) d x = \int_ {t} ^ {t + r ^ {\prime}} \exp (- f (x - r ^ {\prime})) d x \leq \exp (- \beta r ^ {\prime 2} / 2) I (t),
$$
so by induction,  $I(t_0 - kr') \leq \exp(-\beta (k - 1)r'^2 / 2)I(t_0 - r')$  for any  $1 \leq k \leq K := \lfloor (t_0 - a) / r' \rfloor$ . It follows that
$$
\int_ {t _ {0} - K r ^ {\prime}} ^ {t _ {0}} \exp (- f (x)) d x = \sum_ {k = 1} ^ {K} I \left(t _ {0} - k r ^ {\prime}\right) \leq I \left(t _ {0} - r ^ {\prime}\right) \sum_ {k = 1} ^ {K} \exp \left(- \beta (k - 1) r ^ {\prime 2} / 2\right) \leq \frac {I \left(t _ {0} - r ^ {\prime}\right)}{1 - \exp \left(- \beta r ^ {\prime 2} / 2\right)}. \tag {10}
$$
Finally, note that  $t_0 - (K - 1)r' \leq a + 2r' \leq 0.8$ . For any  $x \in [0,0.8]$ , we have  $f'(x) \leq 8x^7 - 0.72\beta x = x(8x^6 - 1.44\beta) \leq 0$ , since  $\beta > 150$ . That is,  $f$  is non-increasing on  $[0, t_0 - (K - 1)r']$ . It follows that
$$
\begin{array}{l} \int_ {0} ^ {t _ {0} - K r ^ {\prime}} \exp (- f (x)) d x \leq \frac {t _ {0} - K r ^ {\prime}}{r ^ {\prime}} \int_ {t _ {0} - K r ^ {\prime}} ^ {t _ {0} - (K - 1) r ^ {\prime}} \exp (- f (x)) d x \\ \leq \frac {1}{r ^ {\prime}} I \left(t _ {0} - K r ^ {\prime}\right) \\ \leq \frac {\exp (- \beta (K - 1) r ^ {\prime 2} / 2)}{r ^ {\prime}} I (t _ {0} - r ^ {\prime}). \\ \end{array}
$$
Since  $(K - 1)r^{\prime}\geq t_{0} - 0.8\geq 1 - \frac{3}{\beta} -0.8\geq 0.1$  , we conclude that
$$
\int_ {0} ^ {t _ {0} - K r ^ {\prime}} \exp (- f (x)) d x \leq \frac {\exp \left(- \beta r ^ {\prime} / 2 0\right)}{r ^ {\prime}} I \left(t _ {0} - r ^ {\prime}\right). \tag {11}
$$
Combining (9), (10), and (11), we get
$$
\int_ {0} ^ {\infty} \exp (- f (x)) d x \leq \left(\frac {1}{1 - \exp (- \beta r ^ {\prime 2} / 2)} + \frac {\exp (- \beta r ^ {\prime} / 2 0)}{r ^ {\prime}}\right) \int_ {t _ {0} - r ^ {\prime}} ^ {t _ {0} + r ^ {\prime}} \exp (- f (x)) d x.
$$
Substituting in  $r' \geq r/2$  gives the claimed result.
Lemma A.9. Fix  $\beta \geq 160\log (8)$ . Then for any  $1\le k\le 8$
$$
\int_ {0} ^ {\infty} x ^ {k} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x \leq 2 ^ {k} \int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x.
$$
Proof. Define a distribution  $q(x) \propto \exp(-x^8 - \beta (1 - x^2)^2)$  for  $x \in [0, \infty)$ . We want to show that  $\mathbb{E}_q[x^k] \leq 2^k$ . Indeed,
$$
\begin{array}{l} \mathbb {E} _ {q} [ \exp (x ^ {8}) ] = \frac {\int_ {0} ^ {\infty} \exp (- \beta (1 - x ^ {2}) ^ {2}) d x}{\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x} \\ \leq \frac {2 \int_ {1 / 2} ^ {3 / 2} \exp (- \beta (1 - x ^ {2}) ^ {2}) d x}{\int_ {0} ^ {\infty} \exp (- x ^ {8} - \beta (1 - x ^ {2}) ^ {2}) d x} \\ = 2 \mathbb {E} _ {q} \left[ \exp \left(x ^ {8}\right) \mathbb {1} \left[ 1 / 2 \leq x \leq 3 / 2 \right] \right] \\ \leq 2 \exp ((3 / 2) ^ {8}) \\ \end{array}
$$
where the first inequality is by an application of Lemma A.8 with  $r = 1/2$  and  $m = 1$ . Now by Jensen's inequality we get
$$
\mathbb {E} _ {q} \left[ x ^ {8} \right] \leq \log \mathbb {E} _ {q} \left[ \exp \left(x ^ {8}\right) \right] = \log (2) + (3 / 2) ^ {8} \leq 2 ^ {8}
$$
and consequently, an application of Hölder inequality gives us  $\mathbb{E}_q[x^k] \leq 2^k$ , for any  $1 \leq k \leq 8$ .
A.5 Proof of Corollary 3.6
Proof of Corollary 3.6. Recall that  $\log p_{\theta}(x) = \log h(x) + \langle \theta, T(x) \rangle - \log Z_{\theta}$ . Therefore  $L_{\mathrm{MLE}}(\theta) = \hat{\mathbb{E}} \log h(x) + \langle \theta, \hat{\mathbb{E}} T(x) \rangle - \log Z_{\theta}$  and  $\nabla_{\theta} L_{\mathrm{MLE}}(\theta) = \hat{\mathbb{E}} T(x) - \nabla_{\theta} \log Z_{\theta}$ . Note that we can compute  $\hat{\mathbb{E}} \log h(x)$  and  $\hat{\mathbb{E}} T(x)$  exactly. It follows that if we can approximate  $L_{\mathrm{MLE}}(\theta)$  up to an additive error of  $n \log 1.16$ , then we can compute  $\log Z_{\theta}$  up to an additive error of  $n \log 1.16$ . Similarly, if we can compute  $\nabla_{\theta} L_{\mathrm{MLE}}(\theta)$  up to an additive error of  $1/20$ , then we can compute  $\nabla_{\theta} \log Z_{\theta}$  up to an additive error of  $1/20$ . This contradicts Theorems 3.4 and 3.5 respectively, completing the proof.
B Moment bounds
Lemma B.1 (Moment bound). For any  $\theta \in \Theta_B$ ,  $i \in [n]$ , and  $\ell \in \mathbb{N}$  it holds that
$$
\mathbb {E} _ {x \sim p _ {\theta}} x _ {i} ^ {\ell} \leq 3 2 ^ {\ell} \max  (2 \ell^ {\ell}, B ^ {\ell} M ^ {\ell} 2 ^ {\ell (d + 1) + 1}).
$$
Proof. Without loss of generality assume  $i = 1$ . Let  $L_0 \coloneqq 32\max(\ell, BM2^{d+1})$ . Then
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} x _ {1} ^ {\ell} \leq L _ {0} ^ {\ell} + \mathbb {E} _ {x \sim p _ {\theta}} x _ {1} ^ {\ell} \mathbb {1} [ \| x \| _ {\infty} > L _ {0} ] \\ = L _ {0} ^ {\ell} + \sum_ {k = 0} ^ {\infty} \mathbb {E} _ {x \sim p _ {\theta}} \left[ x _ {1} ^ {\ell} \mathbb {1} \left[ 2 ^ {k} L _ {0} <   \| x \| _ {\infty} \leq 2 ^ {k + 1} L _ {0} \right] \right] \\ \end{array}
$$
Now for any  $L \geq L_0$ ,
$$
\begin{array}{l} \mathbb {E} \left[ x _ {1} ^ {\ell} \mathbb {1} [ L <   \| x \| _ {\infty} \leq 2 L ] \right] \\ = \frac {1}{Z _ {\theta}} \int_ {B _ {2 L} (0) \backslash B _ {L} (0)} x _ {1} ^ {\ell} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {d + 1} + \langle \theta , T (x) \rangle\right) d x \\ \leq \frac {(2 L) ^ {n}}{Z _ {\theta}} (2 L) ^ {\ell} \exp \left(- L ^ {d + 1} + B M (2 L) ^ {d}\right) \\ \leq \frac {(2 L) ^ {n + \ell} \exp (- L ^ {d + 1} / 2)}{Z _ {\theta}}. \\ \end{array}
$$
We can lower bound  $Z_{\theta}$  as
$$
\begin{array}{l} Z _ {\theta} \geq \int_ {\mathcal {B} _ {1 / (B M)} (0)} \exp \left(- \sum_ {i = 1} ^ {n} x _ {i} ^ {d + 1} + \langle \theta , T (x) \rangle\right) d x \\ \geq (B M) ^ {- n} \exp (- n (B M) ^ {- d - 1} - B M (B M) ^ {- 1}) \\ \geq e ^ {- 2} (B M) ^ {- n}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ x _ {1} ^ {\ell} \mathbb {1} \left[ L <   \| x \| _ {\infty} \leq 2 L \right] \right] \leq \exp \left(\left(n + \ell\right) \log (2 L) - \frac {1}{2} L ^ {d + 1} + 2 + n \log (B M)\right) \\ \leq \exp \left(- \frac {1}{4} L ^ {d + 1}\right) \\ \end{array}
$$
where the last inequality uses the facts that  $L \geq L_0$ ,  $B \geq 1$ , and  $M \geq n$  to get
$$
(n + \ell) \log (2 L) + 2 + n \log (B M) \leq 2 \max (n, \ell) \cdot L + \frac {L ^ {2}}{1 6} + \frac {L ^ {2}}{1 6} \leq \frac {3 L ^ {2}}{1 6} \leq \frac {L ^ {d + 1}}{4}.
$$
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} x _ {1} ^ {\ell} \leq L _ {0} ^ {\ell} + \sum_ {k = 0} ^ {\infty} \exp \left(- \frac {1}{4} 2 ^ {k (d + 1)} L _ {0} ^ {d + 1}\right) \\ \leq L _ {0} ^ {\ell} + 1 \quad \leq 2 L _ {0} ^ {\ell} \\ \end{array}
$$
Lemma B.2 (Largest eigenvalue bound). For any  $\theta \in \Theta_B$ , it holds that
$$
\mathbb {E} _ {x \sim p _ {\theta}} T (x) T (x) ^ {\top} \preceq B ^ {2 d} M ^ {2 d + 1} 2 ^ {2 d (d + 6) + 1}.
$$
We also have the following consequences:
(a)  $\| \mathbb{E}_{x\sim p_{\theta}}T(x)\| _2^2\leq B^{2d}M^{2d + 2}2^{2d(d + 6) + 1},$
(b)  $\lambda_{max}(\mathcal{I}(\theta))\leq B^{2d}M^{2d + 1}2^{2d(d + 6) + 1},$
(c)  $\operatorname{Pr}_{x\sim p_{\theta}}[\| x\|_{\infty} > 2^{d + 8}nBM]\leq 1 / 2.$
Proof. Fix any  $u, v \in [M]$ . Then  $T(x)_u T(x)_v = \prod_{i=1}^n x_i^{\gamma_i}$  for some nonnegative integers  $\gamma_1, \ldots, \gamma_n$  where  $d' := \sum_{i=1}^n \gamma_i \leq 2d$ . Therefore
$$
\mathbb {E} _ {x \sim p _ {\theta}} T (x) _ {u} T (x) _ {v} = \mathbb {E} _ {x \sim p _ {\theta}} \prod_ {i = 1} ^ {n} x _ {i} ^ {\gamma_ {i}} \leq \prod_ {i = 1} ^ {n} \left(\mathbb {E} _ {x \sim p _ {\theta}} x _ {i} ^ {d ^ {\prime}}\right) ^ {\gamma_ {i} / d ^ {\prime}} \leq 3 2 ^ {2 d} B ^ {2 d} M ^ {2 d} 2 ^ {2 d (d + 1) + 1}
$$
by Holder's inequality and Lemma B.1 (with  $\ell = 2d$ ). The claimed spectral bound follows. To prove (a), observe that
$$
\| \mathbb {E} _ {x \sim p _ {\theta}} T (x) \| _ {2} ^ {2} \leq \mathbb {E} _ {x \sim p _ {\theta}} \| T (x) \| _ {2} ^ {2} = \mathrm {T r} \mathbb {E} _ {x \sim p _ {\theta}} T (x) T (x) ^ {\top} \leq M \lambda_ {\max} (\mathbb {E} _ {x \sim p _ {\theta}} T (x) T (x) ^ {\top})
$$
To prove (b), observe that  $\mathcal{I}(\theta) \preceq \mathbb{E}_{x \sim p_{\theta}} T(x) T(x)^{\top}$ . To prove (c), observe that for any  $i \in [n]$ ,
$$
\operatorname * {P r} _ {x \sim p _ {\theta}} \left[ | x _ {i} | > 2 ^ {d + 8} n B M \right] \leq \frac {\mathbb {E} _ {x \sim p _ {\theta}} x _ {i} ^ {2 d}}{(2 ^ {d + 8} n B M) ^ {2 d}} \leq \frac {1}{2 n}.
$$
A union bound over  $i \in [n]$  completes the proof.
Lemma B.3 (Smoothness bounds). For every  $\theta \in \Theta_B$ , it holds that
$$
\mathbb {E} _ {x \sim p _ {\theta}} \| \Delta T (x) \| _ {2} ^ {2} := \sum_ {j = 1} ^ {M} \mathbb {E} _ {x \sim p _ {\theta}} (\Delta T _ {j} (x)) ^ {2} \leq n ^ {2} d ^ {4} B ^ {4 d} M ^ {4 d + 3} 2 ^ {4 d (d + 6) + 2}
$$
$$
\mathbb {E} _ {x \sim p _ {\theta}} \| (J T) (x) \| _ {o p} ^ {2} \leq n ^ {2} d ^ {4} B ^ {4 d} M ^ {4 d + 2} 2 ^ {4 d (d + 6) + 2}.
$$
Proof. Fix any  $j \in [M]$ ; then there is a degree function  $\mathbf{d}$  with  $1 \leq |\mathbf{d}| \leq d$  so that  $T_{j}(x) = x_{\mathbf{d}} = \prod_{i=1}^{n} x_{i}^{\mathbf{d}(i)}$ . Therefore
$$
\Delta T _ {j} (x) = \sum_ {k \in [ n ]: \mathbf {d} (k) \geq 2} \mathbf {d} (k) (\mathbf {d} (k) - 1) x _ {\mathbf {d} - 2 \{k \}} =: \langle w, T (x) \rangle
$$
for some  $w \in \mathbb{R}^M$  with  $\| w\| _2^2 = \sum_{k\in [n]:\mathbf{d}(k)\geq 2}\mathbf{d}(k)^2 (\mathbf{d}(k) - 1)^2 \leq d^4$ . By Corollary B.2, we conclude that
$$
\mathbb {E} _ {x \sim p _ {\theta}} (\Delta T _ {j} (x)) ^ {2} = \mathbb {E} _ {x \sim p _ {\theta}} \langle w, T (x) \rangle^ {2} \leq n ^ {2} d ^ {4} B ^ {4 d} M ^ {4 d + 2} 2 ^ {4 d (d + 6) + 1}.
$$
Summing over  $j \in [M]$  gives the first claimed bound. For the second bound, observe that
$$
\mathbb {E} _ {x \sim p _ {\theta}} \| (J T) (x) \| _ {\mathsf {o p}} ^ {4} \leq \mathbb {E} _ {x \sim p _ {\theta}} \| (J T) (x) \| _ {F} ^ {4} = \mathbb {E} _ {x \sim p _ {\theta}} \left(\sum_ {j = 1} ^ {M} \sum_ {i = 1} ^ {n} \left(\frac {\partial}{\partial x _ {i}} T _ {j} (x)\right) ^ {2}\right) ^ {2}.
$$
For any  $j \in [M]$  and  $i \in [n]$ , there is some degree function  $\mathbf{d}$  with  $|\mathbf{d}| \leq d$  and  $\frac{\partial}{\partial x_i} T_j(x) = |\mathbf{d}| \cdot x_{\mathbf{d} - \{i\}}$ . Thus, by Holder's inequality and Lemma B.1 (with  $\ell = 4d$ ), we get
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} \left(\sum_ {j = 1} ^ {M} \sum_ {i = 1} ^ {n} \left(\frac {\partial}{\partial x _ {i}} T _ {j} (x)\right) ^ {2}\right) ^ {2} = \sum_ {j, j ^ {\prime} \in [ M ]} \sum_ {i, i ^ {\prime} \in [ n ]} \mathbb {E} _ {x \sim p _ {\theta}} \left(\frac {\partial}{\partial x _ {i}} T _ {j} (x)\right) ^ {2} \left(\frac {\partial}{\partial x _ {i ^ {\prime}}} T _ {j ^ {\prime}} (x)\right) ^ {2} \\ \leq M ^ {2} n ^ {2} d ^ {4} B ^ {4 d} M ^ {4 d} 2 ^ {4 d (d + 6) + 2} \\ \end{array}
$$
which proves the second bound.
The following regularity conditions are sufficient for consistency and asymptotic normality of score matching, assuming that the restricted Poincaré constant is finite and  $\lambda_{\min}(\mathcal{I}(\theta^{*})) > 0$  (see Proposition 2 in Forbes and Lauritzen (2015) together with Lemma 1 in Koehler et al. (2022)). We show that these conditions hold for our chosen exponential family.
Lemma B.4 (Regularity conditions). For any  $\theta \in \mathbb{R}^M$ , the quantities  $\mathbb{E}_{x \sim p_{\theta}} \| \nabla h(x) \|_2^4$ ,  $\mathbb{E}_{x \sim p_{\theta}} \| \Delta T(x) \|_2^2$ , and  $\mathbb{E}_{x \sim p_{\theta}} \| (JT)(x) \|_{\mathcal{O}P}^4$  are all finite. Moreover,  $p_{\theta}(x) \to 0$  and  $\| \nabla_x p_\theta(x) \|_2 \to 0$  as  $\| x \|_2 \to \infty$ .