SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Both of the quantities  $\| \nabla h(x)\| _2^4$  and  $\| \Delta T(x)\| _2^2$  can be written as a polynomial in  $x$ . Finiteness of the expectation under  $p_{\theta}$  follows from Holder's inequality and Lemma B.1 (with parameter  $B$  set to  $\| \theta \|_{\infty}$ ). Finiteness of  $\mathbb{E}_{x\sim p_{\theta}}\| (JT)(x)\|_{\mathrm{op}}^4$  is shown in Lemma B.3 (again, with  $B\coloneqq \| \theta \|_{\infty}$ ). The decay condition  $p_{\theta}(x)\to 0$  holds because  $\log p_{\theta}(x) + \log Z_{\theta} = -\sum_{i = 1}^{n}x_{i}^{d + 1} + \langle \theta ,T(x)\rangle$ . For  $x\in \mathbb{R}^n$  with  $L\leq \| x\|_{\infty}\leq 2L$ , the RHS is at most  $-L^{d + 1} + M\| \theta \|_{\infty}(2L)^{d}$ , which goes to  $-\infty$  as  $L\to \infty$ . A similar bound shows that  $\| \nabla_x p_\theta (x)\|_2\to 0$ .
C Omitted Proofs from Section 4
Proof of Lemma 4.2. We use the fact that the Legendre polynomials
$$
L _ {k} (x) = \frac {1}{2 ^ {k}} \sum_ {j = 0} ^ {k} \binom {k} {j} ^ {2} (x - 1) ^ {k - j} (x + 1) ^ {j},
$$
for integers  $0 \leq k \leq d$ , form an orthogonal basis for the vector space  $\mathbb{R}[x]_{\leq d}$  with respect to  $L^2[-1,1]$  (see e.g. Koepf (1998)). We consider the normalized versions  $\hat{L}_k = \sqrt{\frac{2k + 1}{2}} L_k$ , so that  $\left\| \hat{L}_k \right\|_{L^2[-1,1]} = 1$ . By tensorization, the set of products of Legendre polynomials
$$
\hat {L} _ {\mathbf {d}} (x) = \prod_ {i = 1} ^ {n} \hat {L} _ {\mathbf {d} (i)} (x _ {i}),
$$
as  $\mathbf{d}$  ranges over degree functions with  $|\mathbf{d}| \leq d$ , form an orthonormal basis for  $\mathbb{R}[x_1, \ldots, x_n]_{\leq d}$  with respect to  $L^2([-1, 1]^n)$ .
Using the formula for  $L_{k}$ , we obtain that the sum of absolute values of coefficients of  $L_{k}$  (in the monomial basis) is at most  $\frac{1}{2^k}\sum_{j = 0}^{k}\binom{k}{j}^2 2^k\leq 2^{2k}$ . By the bound  $\| \cdot \| _2\leq \| \cdot \| _1$  and the definition of  $\hat{L}_k$ ,
$$
\left\| \hat {L} _ {k} \right\| _ {\text {m o n}} ^ {2} \leq \frac {2 k + 1}{2} \| L _ {k} \| _ {\text {m o n}} ^ {2} \leq \frac {2 k + 1}{2} 2 ^ {4 k}
$$
and hence for any degree function  $\mathbf{d}$  with  $|\mathbf{d}|\leq d$
$$
\begin{array}{l} \left\| \hat {L} _ {\mathbf {d}} \right\| _ {\text {m o n}} ^ {2} = \prod_ {i = 1} ^ {n} \left\| \hat {L} _ {\mathbf {d} (i)} \right\| _ {\text {m o n}} ^ {2} \leq \prod_ {i = 1} ^ {n} \frac {2 \mathbf {d} (i) + 1}{2} 2 ^ {4 \mathbf {d} (i)} \\ \leq \prod_ {i = 1} ^ {n} e ^ {\mathbf {d} (i)} 2 ^ {4 \mathbf {d} (i)} \leq (1 6 e) ^ {d}. \\ \end{array}
$$
Consider any polynomial  $f \in \mathbb{R}[x_1, \ldots, x_n]_{\leq d}$ , and write  $f = \sum_{|\mathbf{d}| \leq d} a_{\mathbf{d}} \hat{L}_{\mathbf{d}}$ . By orthonormality, it holds that  $\sum_{|\mathbf{d}| \leq d} a_{\mathbf{d}}^2 = \| f \|_{L^2([-1,1]^n)}^2$ . Thus, by the triangle inequality and Cauchy-Schwarz,
$$
\begin{array}{l} \| p \| _ {\mathrm {m o n}} ^ {2} = \left\| \sum_ {| \mathbf {d} | \leq d} a _ {\mathbf {d}} \hat {L} _ {\mathbf {d}} \right\| _ {\mathrm {m o n}} ^ {2} \leq \sum_ {| \mathbf {d} | \leq d} a _ {\mathbf {d}} ^ {2} \cdot \sum_ {| \mathbf {d} | \leq d} \left\| \hat {L} _ {\mathbf {d}} \right\| _ {\mathrm {m o n}} ^ {2} \\ \leq \| p \| _ {L ^ {2} ([ - 1, 1 ] ^ {n})} ^ {2} \left( \begin{array}{c} n + d \\ d \end{array} \right) (1 6 e) ^ {d} \\ \end{array}
$$
Proof of Lemma 4.3. Let  $f \in \mathbb{R}[x_1, \ldots, x_n]_{\leq d}$  be a polynomial of degree at most  $d$  in  $x_1, \ldots, x_n$ . Define  $g(x) = f(x) - \mathbb{E}_{x \sim p} f(x)$ . Set  $\epsilon = 1 / (2(d + 1) MR^d(n + B))$  and let  $(W_i)_{i \in I}$  be  $\ell_{\infty}$ -balls of radius  $\epsilon$  partitioning  $\{x \in \mathbb{R}^n : \|x\|_{\infty} \leq R\}$ . Define random variable  $X \sim p|\{\|X\|_{\infty} \leq R\}$  and let  $\iota \in I$  be the random index so that  $X \in W_{\iota}$ . Then
$$
\begin{array}{l} \mathrm {V a r} _ {p} (f) = \mathbb {E} _ {x \sim p} [ g (x) ^ {2} ] \\ \geq \frac {1}{2} \mathbb {E} [ g (X) ^ {2} ] \\ = \frac {1}{2} \mathbb {E} _ {\iota} \mathbb {E} _ {X} [ g (X) ^ {2} | X \in W _ {\iota} ] \\ \end{array}
$$
where the inequality uses guarantee (c) of Lemma B.2 that  $\operatorname{Pr}_{x\sim p}[\| x\|_{\infty} > R]\leq 1 / 2$
Thus, there exists some  $\iota^{*} \in I$  such that  $\mathbb{E}_X[g(X)^2 | X \in W_{\iota^{*}}] \leq 2 \operatorname{Var}_p(f)$ . Let  $q: \mathbb{R}^n \to \mathbb{R}_+$  be the density function of  $X | X \in W_{\iota^{*}}$ . Since  $q(x) \propto p(x) \mathbb{1}[x \in W_{\iota^{*}}]$ , for any  $u, v \in W_{\iota^{*}}$  we have that
$$
\begin{array}{l} \frac {q (u)}{q (v)} = \frac {p (u)}{p (v)} = \frac {h (u) \exp (\langle \theta , T (u) \rangle)}{h (v) \exp (\langle \theta , T (v) \rangle)} \\ = \exp \left(\sum_ {i = 1} ^ {n} v _ {i} ^ {d + 1} - u _ {i} ^ {d + 1} + \langle \theta , T (u) - T (v) \rangle\right). \\ \end{array}
$$
Applying Lemma C.1, we get that
$$
\begin{array}{l} \frac {q (u)}{q (v)} \leq \exp \left(n (d + 1) R ^ {d} \| u - v \| _ {\infty} + M B \| T (u) - T (v) \| _ {\infty}\right) \\ \leq \exp \left((n + B) \cdot M (d + 1) R ^ {d} \| u - v \| _ {\infty}\right) \\ \leq \exp (2 \epsilon (n + B) \cdot M (d + 1) R ^ {d}) \\ \leq \exp (1) \\ \end{array}
$$
by choice of  $\epsilon$ . It follows that if  $\tilde{\mathcal{U}}$  is the uniform distribution on  $W_{\iota^{*}}$ , then  $q(x) \geq e^{-1}\tilde{\mathcal{U}}(x)$  for all  $x \in \mathbb{R}^n$ . Thus,
$$
\operatorname {V a r} _ {p} (f) \geq \frac {1}{2} \mathbb {E} _ {X} [ g (X) ^ {2} | X \in W _ {\iota^ {*}} ] \geq \frac {1}{2 e} \mathbb {E} _ {x \sim \tilde {\mathcal {U}}} [ g (x) ^ {2} ] \geq \frac {1}{2 e} \operatorname {V a r} _ {\tilde {\mathcal {U}}} (g) = \frac {1}{2 e} \operatorname {V a r} _ {\tilde {\mathcal {U}}} (f)
$$
Lemma C.1. Fix  $R > 0$ . For any degree function  $\mathbf{d}:[n]\to \mathbb{N}$  with  $|\mathbf{d}|\leq d$ , and for any  $u,v\in \mathbb{R}^n$  with  $\| u\|_{\infty},\| v\|_{\infty}\leq R$ , it holds that
$$
\left| u _ {\mathbf {d}} - v _ {\mathbf {d}} \right| \leq d R ^ {d - 1} \left\| u - v \right\| _ {\infty}.
$$
Proof. Define  $m(x) = x_{\mathbf{d}} = \prod_{i=1}^{n} x_i^{\mathbf{d}(i)}$ . Then
$$
\begin{array}{l} |m(u) - m(v)|\leq \| u - v\|_{\infty}\sup_{x\in \mathcal{B}_{R}(0)}\| \nabla_{x}m(x)\|_{1} \\ = \| u - v \| _ {\infty} \sup _ {x \in \mathcal {B} _ {R} (0)} \sum_ {i \in [ n ]: \mathbf {d} (i) > 0} \mathbf {d} (i) \prod_ {j = 1} ^ {n} x _ {i} ^ {\mathbf {d} (i) - \mathbb {1} [ i = j ]} \\ \leq \| u - v \| _ {\infty} \cdot d R ^ {d - 1} \\ \end{array}
$$
Proof of Lemma 4.4. By Lemma 4.3, there is some  $z \in \mathbb{R}^n$  with  $\|z\|_{\infty} \leq R$  and some  $\epsilon \geq 1/(2(d+1)MR^d(n+B))$  so that if  $\tilde{\mathcal{U}}$  is the uniform distribution on  $\{x \in \mathbb{R}^n : \|x-z\|_{\infty} \leq \epsilon\}$ , then
$$
\operatorname {V a r} _ {p} (f) \geq \frac {1}{2 e} \operatorname {V a r} _ {\tilde {\mathcal {U}}} (f).
$$
Define  $g:\mathbb{R}^n\to \mathbb{R}$  by  $g(x) = f(\epsilon x + z) - \mathbb{E}_{\tilde{\mathcal{U}}}f$  . Then by Lemma 4.2,
$$
\begin{array}{l} \| g \| _ {\operatorname {m o n}} ^ {2} \leq (1 6 e) ^ {d} M \mathbb {E} _ {x \sim \operatorname {U n i f} ([ - 1, 1 ] ^ {n})} g (x) ^ {2}. \\ = (1 6 e) ^ {d} M \operatorname {V a r} _ {\tilde {\mathcal {U}}} (f) \\ \leq (1 6 e) ^ {d + 1} M \operatorname {V a r} _ {p} (f). \\ \end{array}
$$
Write  $f(x) = \sum_{1\leq |\mathbf{d}|\leq d}\alpha_{\mathbf{d}}x_{\mathbf{d}}$  and  $g(x) = \sum_{1\leq |\mathbf{d}|\leq d}\beta_{\mathbf{d}}x_{\mathbf{d}}$ . We know that  $f(x) = g(\epsilon^{-1}(x - z)) + \mathbb{E}_{\tilde{\mathcal{U}}}f$ . Thus, for any nonzero degree function  $\mathbf{d}$ , we have
$$
\alpha_{\mathbf{d}} = \sum_{\substack{\mathbf{d}^{\prime}\geq \mathbf{d}\\ |\mathbf{d}^{\prime}|\leq d}}\epsilon^{-|\mathbf{d}^{\prime}|}(-z)^{\mathbf{d}^{\prime} - \mathbf{d}}\beta_{\mathbf{d}^{\prime}}.
$$
Thus  $|\alpha_{\mathbf{d}}| \leq \epsilon^{-d} R^{d} \| \beta \|_{1} \leq \epsilon^{-d} R^{d} \sqrt{M} \| g \|_{\mathrm{mon}}$ , and so summing over monomials gives
$$
\| f \| _ {\operatorname {m o n}} ^ {2} \leq M ^ {2} \epsilon^ {- 2 d} R ^ {2 d} \| g \| _ {\operatorname {m o n}} ^ {2} \leq (1 6 e) ^ {d + 1} M ^ {3} \epsilon^ {- 2 d} R ^ {2 d} \operatorname {V a r} _ {p} (f).
$$
Substituting in the choice of  $\epsilon$  from Lemma 4.3 completes the proof.
D Omitted Proofs from Section 5
Proof of Lemma 5.1. We have
$$
\begin{array}{l} \operatorname {V a r} _ {p _ {\theta}} (f) = \mathbb {E} _ {x \sim p _ {\theta}} [ f (x) ^ {2} ] - \mathbb {E} _ {x \sim p _ {\theta}} [ f (x) ] ^ {2} \\ = w ^ {\top} \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) T (x) ^ {\top} ] w - w ^ {\top} \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] ^ {\top} w \\ = w ^ {\top} \mathcal {I} (\theta) w, \\ \end{array}
$$
$$
\| w \| _ {2} ^ {2} \lambda_ {\min } (\mathcal {I} (\theta)) \leq w ^ {\top} \mathcal {I} (\theta) w \leq \| w \| _ {2} ^ {2} \lambda_ {\max } (\mathcal {I} (\theta),
$$
the lemma statement follows.
Proof of Lemma 5.3. Since  $f(x) = \sum_{1 \leq |\mathbf{d}| \leq d} w_{\mathbf{d}} x_{\mathbf{d}}$ , we have for any  $u \in \mathbb{R}^n$  that
$$
\langle u, \nabla_ {x} f (x) \rangle = \sum_ {i = 1} ^ {n} u _ {i} \sum_ {0 \leq | \mathbf {d} | <   d} (1 + \mathbf {d} (i)) w _ {\mathbf {d} + \{i \}} x _ {\mathbf {d}} = c (u) + \sum_ {1 \leq | \mathbf {d} | <   d} \tilde {w} (u) _ {\mathbf {d}} x _ {\mathbf {d}}
$$
where  $c(u) \coloneqq \sum_{i=1}^{n} u_i w_{\{i\}}$  and  $\tilde{w}(u)_{\mathbf{d}} \coloneqq \sum_{i=1}^{n} u_i (1 + \mathbf{d}(i)) w_{\mathbf{d} + \{i\}}$ . But now
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} [ \| \nabla_ {x} f (x) \| _ {2} ^ {2} ] = \mathbb {E} _ {x \sim p _ {\theta}} \mathbb {E} _ {u \sim N (0, I _ {n})} \langle u, \nabla_ {x} f (x) \rangle^ {2} \\ = \mathbb {E} _ {u \sim N (0, I _ {n})} \mathbb {E} _ {x \sim p _ {\theta}} \big (c (u) + \langle \tilde {w} (u), T (x) \rangle \big) ^ {2} \\ \geq \mathbb {E} _ {u \sim N (0, I _ {n})} \frac {c (u) ^ {2} + \| \tilde {w} (u) \| _ {2} ^ {2}}{4 + 4 \| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \| _ {2} ^ {2}} \min (1, \lambda_ {\min} (\mathcal {I} (\theta))). \\ \end{array}
$$
where the last inequality is by Lemma 5.2. Finally,
$$
\begin{array}{l} \mathbb {E} _ {u \sim N (0, I _ {n})} \left[ c (u) ^ {2} + \| \tilde {w} (u) \| _ {2} ^ {2} \right] = \sum_ {0 \leq | \mathbf {d} | <   d} \mathbb {E} _ {u \sim N (0, I _ {n})} \left[ \left(\sum_ {i = 1} ^ {n} u _ {i} (1 + \mathbf {d} (i)) w _ {\mathbf {d} + \{i \}}\right) ^ {2} \right] \\ = \sum_ {0 \leq | \mathbf {d} | <   d} \sum_ {i = 1} ^ {n} (1 + \mathbf {d} (i)) ^ {2} w _ {\mathbf {d} + \{i \}} ^ {2} \quad \geq \| w \| _ {2} ^ {2} \\ \end{array}
$$
where the second equality is because  $\mathbb{E}[u_i u_j] = \mathbb{1}[i = j]$  for all  $i, j \in [n]$ , and the last inequality is because every term  $w_{\mathbf{d}}^2$  in  $\| w \|_2^2$  appears in at least one of the terms of the previous summation (and has coefficient at least one). Putting everything together gives
$$
\begin{array}{l} \mathbb {E} _ {x \sim p _ {\theta}} [ \| \nabla_ {x} f (x) \| _ {2} ^ {2} ] \geq \frac {\| w \| _ {2} ^ {2}}{4 + 4 \| \mathbb {E} _ {x \sim p _ {\theta}} [ T (x) ] \| _ {2} ^ {2}} \min  (1, \lambda_ {\min } (\mathcal {I} (\theta))) \\ \geq \frac {1}{4 + 4 \| \mathbb {E} [ T (x) ] \| _ {2} ^ {2}} \frac {\min  \left(1 , \lambda_ {\min } (\mathcal {I} (\theta))\right)}{\lambda_ {\max } (\mathcal {I} (\theta))} \operatorname {V a r} _ {p _ {\theta}} (f) \\ \end{array}
$$
where the last inequality is by Lemma 5.1.