SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \nabla^ {2} \ell (\boldsymbol {\theta}) = \frac {(\int_ {S} T (\mathbf {x}) T (\mathbf {x}) ^ {\top} h (\mathbf {x}) \exp (\boldsymbol {\theta} ^ {\top} T (\mathbf {x}) - A (\boldsymbol {\theta})) d \mathbf {x})}{(\int_ {S} h (\mathbf {x}) \exp (\boldsymbol {\theta} ^ {\top} T (\mathbf {x}) - A (\boldsymbol {\theta})) d \mathbf {x})} \\ - \frac {\left(\int_ {S} T (\mathbf {x}) h (\mathbf {x}) \exp \left(\boldsymbol {\theta} ^ {\top} T (\mathbf {x}) - A (\boldsymbol {\theta})\right) d \mathbf {x}\right)}{\left(\int_ {S} h (\mathbf {x}) \exp \left(\boldsymbol {\theta} ^ {\top} T (\mathbf {x}) - A (\boldsymbol {\theta})\right) d \mathbf {x}\right)} \cdot \left(\frac {\left(\int_ {S} T (\mathbf {x}) h (\mathbf {x}) \exp \left(\boldsymbol {\theta} ^ {\top} T (\mathbf {x}) - A (\boldsymbol {\theta})\right) d \mathbf {x}\right)}{\left(\int_ {S} h (\mathbf {x}) \exp \left(\boldsymbol {\theta} ^ {\top} T (\mathbf {x}\right) - A (\boldsymbol {\theta})\right) d \mathbf {x}}\right) ^ {\top} \\ = \mathbf {C o v} _ {\mathbf {x} \sim p _ {\theta} ^ {S}} [ T (\mathbf {x}), T (\mathbf {x}) ] \\ \end{array}
$$
We can similarly define the population negative log-likelihood as
$$
\overline {{\ell}} (\boldsymbol {\theta}) := \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} \left[ - \log h (\mathbf {x}) - \boldsymbol {\theta} ^ {\top} T (\mathbf {x}) \right] + \log \int_ {S} h (\mathbf {x}) \exp (\theta^ {\top} T (\mathbf {x})) d \mathbf {x}),
$$
$$
\nabla^ {\ell} (\boldsymbol {\theta}) = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta} ^ {*}} ^ {s}} [ - T (\mathbf {x}) ] + \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {s}} [ T (\mathbf {x}) ],
$$
$$
\nabla^ {2} \bar {\ell} (\boldsymbol {\theta}) = \nabla^ {2} \ell (\boldsymbol {\theta})
$$
Proof. Define the following quantities:
$$
\mathbf {R} ^ {*} = \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} \left[ (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (\mathbf {x}) ]) \cdot (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (x) ]) ^ {\top} \right]
$$
$$
\mathbf {R} ^ {\prime} = \mathbb {E} _ {x \sim p _ {\boldsymbol {\theta}}} \left[ \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) \cdot \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) ^ {\top} \right]
$$
$$
\mathbf {R} = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} \left[ \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) \cdot \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) ^ {\top} \right]
$$
Claim 2.  $R^{\prime}\succeq R^{*}$  . (Proof in Appendix A.4.)
Now, let  $\xi \in \mathbb{R}^k$  with  $\| \xi \|_2^2 = 1$  arbitrary. Then
$$
\xi^ {\top} \mathbf {R} ^ {*} \xi = \xi^ {\top} \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} \left[ (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (\mathbf {x}) ]) \cdot (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (\mathbf {x}) ]) ^ {\top} \right] \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} \left[ p _ {\xi} (\mathbf {x}) \right]
$$
$$
\xi^ {\top} \mathbf {R} ^ {\prime} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} \left[ p _ {\xi} ^ {\prime} (\mathbf {x}) \right]
$$
$$
\xi^ {\top} \mathbf {R} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {s}} \left[ p _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {x}) \right]
$$
where  $p_{\xi}(\mathbf{x}), p_{\xi}'(\mathbf{x})$  are polynomials of degree at most  $2d$  whose coefficients depend on  $\xi$  (under A3). Furthermore, note that for any  $\xi \in \mathbb{R}^k$ ,  $p_{\xi}(\mathbf{x}) \geq 0$  and  $p_{\xi}'(\mathbf{x}) \geq 0$  (due to the rank one matrix inside the expectation being PSD).
First, since  $\mathbf{R}' \succeq \mathbf{R}^*\iff \xi^\top \mathbf{R}'\xi \geq \xi^\top \mathbf{R}^*\xi$  we have
$$
\mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}}} \left[ p _ {\xi} ^ {\prime} (\mathbf {z}) \right] \geq \mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}}} \left[ p _ {\xi} (\mathbf {z}) \right] \geq \lambda .
$$
Now define the set  $A \coloneqq \{\mathbf{x}: p_{\xi}'(\mathbf{x}) \leq \gamma\}$  for  $\gamma = \left(\frac{\beta}{4Cd}\right)^{2d} \lambda$  where  $p_{\boldsymbol{\theta}}(S) = \beta > 0$ . Theorem 8 of [5] says
$$
p _ {\boldsymbol {\theta}} (A) \leq \frac {C q \gamma^ {1 / (2 d)}}{\left(\mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}}} \left[ p _ {\xi} ^ {\prime} (\mathbf {z}) \right] ^ {q / 2 d}\right) ^ {1 / q}} \overset {q = 2 d} {=} \frac {2 C d \gamma^ {1 / (2 d)}}{\left(\underbrace {\mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}}} \left[ p _ {\xi} ^ {\prime} (\mathbf {z}) \right]} _ {\geq \lambda}\right) ^ {1 / (2 d)}} \leq \frac {2 C d \cdot \gamma^ {1 / (2 d)}}{\lambda^ {1 / (2 d)}} = \frac {\beta}{2}.
$$
Now we can split  $\mathbb{E}_{\mathbf{z}\sim p_{\theta}^{S}}\left[p_{\xi}'(\mathbf{z})\right]$  into the part on  $S\cap A$  and  $S\cap A^c$ . Note that if  $p_{\theta}(S) = \beta$  and  $p_{\theta}(A)\leq \frac{\beta}{2}$ , this implies  $p_{\theta}(S\cap A^c)\geq \frac{\beta}{2}$  as
$$
p _ {\boldsymbol {\theta}} (S \cap A ^ {c}) \geq p _ {\boldsymbol {\theta}} (S) + p _ {\boldsymbol {\theta}} (A ^ {c}) - p _ {\boldsymbol {\theta}} (S \cup A ^ {c}) \geq \beta + \left(1 - \frac {\beta}{2}\right) - 1 = \frac {\beta}{2}.
$$
$$
\mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}} ^ {S \cap A}} \left[ p _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {z}) \right] + E _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}} ^ {S \cap A ^ {c}}} \left[ p _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {z}) \right] \geq \frac {p _ {\boldsymbol {\theta}} (S \cap A)}{p _ {\boldsymbol {\theta}} (S)} \cdot 0 + \frac {p _ {\boldsymbol {\theta}} (S \cap A ^ {c})}{p _ {\boldsymbol {\theta}} (S)} \cdot \gamma \geq \frac {1}{2} \gamma \Rightarrow \mathbb {E} _ {\mathbf {z} \sim p _ {\boldsymbol {\theta}} ^ {S}} \left[ p _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {z}) \right] \geq \frac {1}{2} \left(\frac {\beta}{4 C d}\right) ^ {2 d} \lambda
$$
Proof. Similar to the proof of the previous lemma, define the following quantities:
$$
\begin{array}{l} \mathbf {R} ^ {*} = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} \left[ (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} [ T (\mathbf {x}) ]) \cdot (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} [ T (\mathbf {x}) ]) ^ {\top} \right] \\ \mathbf {R} ^ {\prime \prime} = \mathbb {E} _ {\mathbf {x} \sim p _ {\theta} ^ {S}} \left[ (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (\mathbf {x}) ]) \cdot (T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ T (\mathbf {x}) ]) ^ {\top} \right] \\ \mathbf {R} = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} \left[ \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) \cdot \left(T (\mathbf {x}) - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ]\right) ^ {\top} \right] \\ \end{array}
$$
Claim 3. It holds that  $\mathbf{R}''\succeq \mathbf{R}$  (Similar proof to Claim 2.)
Let  $\xi \in \mathbb{R}^k$  with  $\| \xi \| _2^2 = 1$  arbitrary. Then
$$
\xi^ {\top} \mathbf {R} ^ {*} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\theta}} [ f _ {\xi} (\mathbf {x}) ]
$$
$$
\xi^ {\top} \mathbf {R} ^ {\prime \prime} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ f _ {\xi} (\mathbf {x}) ]
$$
$$
\xi^ {\top} \mathbf {R} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ f _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {x}) ]
$$
where  $f_{\xi}(\mathbf{x}), f_{\xi}'(\mathbf{x})$  are some functions which depend on  $\mathbf{x}$  and  $\xi$  (e.g., polynomials of degree at most  $2d$  under A3). By the previous claim, we also have
$$
\mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {s}} [ f _ {\boldsymbol {\xi}} (\mathbf {x}) ] \geq \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {s}} [ f _ {\boldsymbol {\xi}} ^ {\prime} (\mathbf {x}) ].
$$
$$
\mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}} ^ {S}} [ f _ {\boldsymbol {\xi}} (\mathbf {x}) ] = \int_ {\mathcal {X}} p _ {\boldsymbol {\theta}} ^ {S} (\mathbf {x}) \cdot f _ {\boldsymbol {\xi}} (\mathbf {x}) d \mathbf {x} = \int_ {\mathcal {X}} \frac {1}{p _ {\boldsymbol {\theta}} (S)} p _ {\boldsymbol {\theta}} (\mathbf {x}) \cdot f _ {\boldsymbol {\xi}} (\mathbf {x}) \cdot \mathbb {1} \{\mathbf {x} \in S \} d \mathbf {x} \leq \frac {1}{p _ {\boldsymbol {\theta}} (S)} \underbrace {\int p _ {\boldsymbol {\theta}} (\mathbf {x}) f _ {\boldsymbol {\xi}} (\mathbf {x}) d \mathbf {x}} _ {= \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} [ f _ {\boldsymbol {\xi}} (\mathbf {x}) ]}.
$$
Since  $\lambda I \preceq \mathbf{R}^{*} \preceq LI$  by A1, it holds that  $\xi^{\top} \mathbf{R}^{*}\xi = \mathbb{E}_{\mathbf{x} \sim p_{\theta}}[f_{\xi}(\mathbf{x})] \leq L$ , thus the following inequalities hold:
$$
\xi^ {\top} \mathbf {R} \xi = \mathbb {E} _ {\mathbf {x} \sim p _ {\pmb {\theta}} ^ {S}} [ f _ {\xi} ^ {\prime} (\mathbf {x}) ] \leq \mathbb {E} _ {\mathbf {x} \sim p _ {\pmb {\theta}} ^ {S}} [ f _ {\xi} (\mathbf {x}) ] \leq \frac {1}{p _ {\pmb {\theta}} (S)} L.
$$
A.4 Proof of Claim 2
We will prove a general claim which should take care of both claims in Lemmas 3.2 and 3.3.
Claim 4. Let  $\mathbf{x} \sim \rho$  be a random vector with mean  $\mu$ . Let  $\mathbf{b}$  be another vector such that  $\mathbf{b} \neq \mu$ . Then
$$
\mathbf {C o v} _ {\mathbf {x} \sim \rho} [ \mathbf {x}, \mathbf {x} ] = \mathbb {E} _ {\mathbf {x} \sim \rho} [ (\mathbf {x} - \boldsymbol {\mu}) (\mathbf {x} - \boldsymbol {\mu}) ^ {\top} ] = \mathbb {E} _ {\mathbf {x} \sim \rho} [ (\mathbf {x} - \mathbf {b}) (\mathbf {x} - \mathbf {b}) ^ {\top} ] - (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top}.
$$
$$
\begin{array}{l} \mathbb {E} \left[ (\mathbf {x} - \boldsymbol {\mu}) (\mathbf {x} - \boldsymbol {\mu}) ^ {\top} \right] \\ = \mathbb {E} [ (\mathbf {x} - \mathbf {b} + \mathbf {b} - \boldsymbol {\mu}) (\mathbf {x} - \mathbf {b} + \mathbf {b} - \boldsymbol {\mu}) ^ {\top} ] \\ = \mathbb {E} [ (\mathbf {x} - \mathbf {b}) (\mathbf {x} - \mathbf {b}) ^ {\top} ] + \underbrace {\mathbb {E} [ (\mathbf {x} - \mathbf {b}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} ]} _ {= (- 1) \cdot \mathbb {E} [ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} ]} + \underbrace {\mathbb {E} [ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {x} - \mathbf {b}) ^ {\top} ]} _ {= (- 1) \cdot \mathbb {E} [ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} ]} + \underbrace {\mathbb {E} [ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} ]} _ {= \mathbb {E} [ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} ]} \\ = \mathbb {E} \left[ (\mathbf {x} - \mathbf {b}) (\mathbf {x} - \mathbf {b}) ^ {\top} \right] - \mathbb {E} \left[ (\mathbf {b} - \boldsymbol {\mu}) (\mathbf {b} - \boldsymbol {\mu}) ^ {\top} \right] \\ \end{array}
$$
As a corollary, since the second term is a rank-1 matrix (thus PSD), we have that  $\mathbb{E}[(\mathbf{x} - \mathbf{b})(\mathbf{x} - \mathbf{b})^\top] \succeq \mathbb{E}[(\mathbf{x} - \boldsymbol{\mu})(\mathbf{x} - \boldsymbol{\mu})^\top]$ .
B Examples of Other Distributions which Satisfy Assumptions
Example 1 (Exponential Distribution). The exponential distribution density can be written
$$
p _ {\lambda} (x) = \lambda \exp (- \lambda x) = \exp (- \lambda x + \log (\lambda)),
$$
defined on  $x \in \mathbb{R}^+$  which is a convex set and for  $\lambda > 0$ . In natural form, it is
$$
p _ {\theta} (x) = \exp (\theta x + \log (- \theta))),
$$
defined for  $\theta < 0$ . Note that
-  $T(x) = x$  is a polynomial in  $x$ .
- This is log-linear in  $x$  (so log-concave in  $x$ ).
- Variance of the sufficient statistic is simply the variance, which is  $1 / \theta^2 > 0$  for any  $\theta < 0$ . If we restrict  $\theta$  in a bounded set, the negative log-likelihood will be strongly convex and smooth in  $\theta$ .
Example 2 (Weibull Distribution with known shape  $k$ ). The Weibull distribution with known shape  $k > 0$  has density
$$
p _ {\lambda} (x) = \exp ((k - 1) \log x + \left(- \frac {1}{\lambda^ {k}}\right) x ^ {k} + \log k - k \log \lambda)
$$
defined on  $x \in \mathbb{R}^+$  and  $\lambda > 0$ . We can re-parameterize this in terms of  $\theta = -\frac{1}{\lambda^k}$  with  $\theta < 0$  as
$$
p _ {\theta} (x) = x ^ {k - 1} \exp \left(\theta \cdot x ^ {k} + \log k + \log (- \theta)\right).
$$
-  $T(x) = x^{k}$  is polynomial in  $x$ .
-  $p_{\theta}(x)$  is log-concave in  $x$  if  $k > 1$  (where recall  $x \in \mathbb{R}^+$  and  $\theta < 0$ ).
- The variance of the sufficient statistic can also be found by taking the second derivative of  $A(\theta) = -\log k - \log (-\theta)$  w.r.t.  $\theta$ , which is also  $1 / \theta^2 > 0$ .
Example 3 (Continuous Bernoulli). The continuous Bernoulli density [31] can be written
$$
p _ {\lambda} (x) = \exp \left(\log \frac {\lambda}{1 - \lambda} - \log \frac {1 - 2 \lambda}{(1 - \lambda) \log \frac {1 - \lambda}{\lambda}}\right)
$$
with support  $x \in [0,1]$  and  $\lambda \in (0,1)$ . We can re-parameterize this in terms of  $\theta = \log \frac{\lambda}{1 - \lambda}$  with  $\theta \in [0,\infty)$  so
$$
p _ {\theta} (x) = \exp \left(\theta x - \log \frac {e ^ {\theta} - 1}{\theta}\right).
$$
-  $T(x) = x$  is polynomial in  $x$ .
$p_{\theta}(x)$  is log-linear in  $x$  (so log-concave).
- The variance of sufficient statistic is simply the variance again, which is given by
$$
\mathbf {V a r} (X) = \left\{ \begin{array}{l l} 1 / 1 2 & \text {i f} \lambda = 1 / 2 \\ \frac {(\lambda - 1) \lambda}{(1 - 2 \lambda) ^ {2}} + \frac {1}{(2 \tanh  ^ {- 1} (1 - 2 \lambda)) ^ {2}} & \text {o t h e r w i s e} \end{array} \right.
$$
This is strictly positive and bounded for all values of  $\lambda$  (thus all values of  $\theta$ ).
Example 4 (Continuous Poisson). A continuous version of the Poisson distribution (although there can be others [23]) can be written
$$
p _ {\lambda} (x) = \frac {1}{Z (\lambda)} \frac {e ^ {- \lambda} \lambda^ {x}}{\Gamma (x + 1)}
$$
with support  $x \in [0, \infty)$  and  $\lambda \in (0, \infty)$ . We can write this with  $\theta = \log \lambda$  so
$$
p _ {\theta} (x) = \frac {1}{\Gamma (x + 1)} \exp (\theta x - A (\theta)).
$$
-  $T(x) = x$  is polynomial in  $x$ .
-  $p_{\theta}(x)$  is log-concave in  $x$  for  $x \in \mathbb{R}^{+}$ .
- In  $\lambda$  parameters, the mean of this distribution is  $\lambda$  through usual calculations (e.g., similar to those of the Gamma distribution). Note: we can absorb the  $e^{-\lambda}$  term into the partition function.
$$
\begin{array}{l} \mathbb {E} [ X ] = \frac {1}{Z (\lambda)} \int_ {0} ^ {\infty} \frac {x \lambda^ {x}}{\Gamma (x + 1)} d x \\ = \frac {1}{Z (\lambda)} \int_ {0} ^ {\infty} \frac {x \lambda^ {x}}{x \cdot \Gamma (x)} d x \qquad \qquad \qquad \Gamma (x + 1) = x \cdot \Gamma (x) \\ = \frac {\lambda}{Z (\lambda)} \int_ {1} ^ {\infty} \frac {\lambda^ {x - 1}}{\Gamma (x)} d x \quad \text {P a r t i t i o n f u n c t i o n , c h a n g e v a r .} z = x - 1 \\ = \lambda \\ \end{array}
$$
Similarly, we should be able to show the variance is  $\lambda$  as usual. In  $\theta$  space, this means the variance is  $\exp (\theta)$  for  $\theta \in \mathbb{R}$  which is always positive. Again, we can make it bounded by restricting  $\theta$  to some set.
Example 5 (Multivariate Gaussian). The multivariate Gaussian also satisfies all of these properties. Recall that the sufficient statistics of the multivariate Gaussian has
-  $T(\mathbf{x}) = [\mathbf{x}, \mathbf{xx}^{\top}]$  is a polynomial in the components of  $\mathbf{x}$  with degree at most 2 (where the  $\mathbf{xx}^{\top}$  term can be thought of as the vector after standard vectorization).
- The multivariate Gaussian density is strongly log-concave.
- The covariance matrix (of the sufficient statistics) has a complicated form, which the authors of [10] have analyzed the lower bound for, e.g., in their Claims 1 and 2. As before, we can restrict our parameter space to ensure upper bounds.
Example 6 (Generalized Linear Models). This example is the same as the one given in [25] for generalized linear models. It is restated here for completeness.
Consider when we have some covariance, response pair  $(X,Y)$  drawn from some distribution  $D$ . Suppose that we have a family of distributions  $P(\cdot \mid \theta; X)$  such that, for each  $X$ , it is an exponential family with sufficient statistic  $t_{y,X}$
$$
P (y \mid \theta ; X) = h (y) \exp \left(\langle \theta , t _ {y, X} \rangle - A (\theta , X)\right).
$$
We can consider a one-dimensional exponential family  $q_{\nu}$  with parameterization  $\nu = \langle \theta ,X\rangle$  , then
$$
P (y \mid \theta ; X) = h (y) \exp (y \langle \theta , X \rangle - \log Z (\langle \theta , X \rangle))
$$
where we see that  $t_{y,X} = yX$  and the log partition function  $A(\theta, X) = \log Z(\langle \theta, X \rangle)$ . When  $q_{\nu}$  is Bernoulli family or unit variance Gaussian family, this corresponds to logistic regression or least squares regression, respectively.
We can appropriately generalize this to beyond linear models (e.g., polynomials) provided that we can keep the distribution log-concave.
Comment on A3. We mentioned in the main paper that this assumption combined with log-concavity provides the anti-concentration property that we need for Lemma 3.2. We assume it for simplicity of exposition, but it should be noted that as long as we have the type of anti-concentration property to control how much the covariance can shrink under truncation, we do not necessarily need  $T(\mathbf{x})$  to be polynomial. However, we've provided examples of exponential families which already satisfy this above (and there are potentially more which can be addressed by this framework that do not have polynomial sufficient statistics but nonetheless exhibit similar anti-concentration properties).
C Proofs Relating Truncated and Non-Truncated Quantities
C.1 General Truncated Densities
Let  $\rho$  be a probability distribution on  $\mathbb{R}^d$ . Let  $S \subseteq \mathbb{R}^d$  be such that  $\rho(S) = \alpha$  for some  $\alpha \in (0,1]$ . Let  $\rho^S \coloneqq \rho(\cdot \mid \cdot \in S)$  be the conditional distribution of  $\mathbf{x} \sim \rho$  given that  $\mathbf{x} \in S$ .
Note that the relative density is
$$
\frac {\rho^ {S} (\mathbf {x})}{\rho (\mathbf {x})} = \frac {\mathbf {1} \{\mathbf {x} \in S \}}{\rho (S)}.
$$
Then we can compute that the Rényi divergence is a constant for any order  $1 \leq q \leq \infty$ .
$$
\mathsf {K L} (\rho^ {S} \| \rho) = \mathbb {E} _ {\rho^ {S}} \left[ \log \frac {\rho^ {S}}{\rho} \right] = \mathbb {E} _ {\rho^ {S}} \left[ \log \frac {1}{\rho (S)} \right] = \log \frac {1}{\alpha}.
$$
$$
\chi^ {2} (\rho^ {S} \| \rho) = \mathbb {E} _ {\rho^ {S}} \left[ \frac {\rho^ {S}}{\rho} \right] - 1 = \frac {1}{\rho (S)} - 1 = \frac {1}{\alpha} - 1.
$$
$$
\mathsf {R} _ {q} (\rho^ {S} \| \rho) = \frac {1}{q - 1} \log \mathbb {E} _ {\rho^ {S}} \left[ \left(\frac {\rho^ {S}}{\rho}\right) ^ {q - 1} \right] = \frac {1}{q - 1} \log \frac {1}{\rho (S) ^ {q - 1}} = \log \frac {1}{\rho (S)} = \log \frac {1}{\alpha}.
$$
$$
\mathsf {R} _ {\infty} (\rho^ {S} \| \rho) = \log \sup _ {x} \frac {\rho^ {S} (x)}{\rho (x)} = \log \frac {1}{\rho (S)} = \log \frac {1}{\alpha}.
$$
Note  $\mathsf{R}_2(\rho^S\| \rho) = \log (1 + \chi^2 (\rho^S\| \rho))$
We recall the following general estimates.
Lemma C.1. For any probability distributions  $\rho, \pi$  (such that the quantities below are finite):
1.  $\| \mathbb{E}_{\rho}[\mathbf{x}] - \mathbb{E}_{\pi}[\mathbf{x}]\| \leq \sqrt{\chi^2(\rho\|\pi)}\cdot \sqrt{\mathbf{Var}_{\pi}(\mathbf{x})}.$
2.  $|\mathbb{E}_{\rho}[\| \mathbf{x}\| ^2 ] - \mathbb{E}_{\pi}[\| \mathbf{x}\| ^2 ]|\leq \sqrt{\chi^2(\rho\|\pi)}\cdot \sqrt{\mathbb{E}_{\pi}[\| \mathbf{x}\| ^4]}$
3.  $|\mathbf{Var}_{\rho}(\mathbf{x}) - \mathbf{Var}_{\pi}(\mathbf{x})|\leq \sqrt{(\chi^{2}(\rho\|\pi) + 1)^{2} - 1}\cdot \sqrt{2\mathbb{E}_{\pi}[[\| \mathbf{x} - \mathbb{E}_{\pi}[x]\|^{4}]}}.$
Proof. The first two claims are immediate by Cauchy-Schwarz. For the third one, recall we can write
$$
\mathbf {V a r} _ {\rho} (\mathbf {x}) = \frac {1}{2} \mathbb {E} _ {\rho^ {\otimes 2}} [ \| \mathbf {x} - \mathbf {y} \| ^ {2} ].
$$
Then by applying part (1) to  $\rho^{\otimes 2}$  and  $(\pi)^{\otimes 2}$ , we get
$$
\begin{array}{l} \left| \mathbf {V a r} _ {\rho} (\mathbf {x}) - \mathbf {V a r} _ {\pi} (\mathbf {x}) \right| \leq \frac {1}{2} \sqrt {\chi^ {2} \left(\rho^ {\otimes 2} \| \pi^ {\otimes 2}\right)} \cdot \sqrt {\mathbb {E} _ {\pi^ {\otimes 2}} [ \| \mathbf {x} - \mathbf {y} \| ^ {4} ]} \\ = \frac {1}{2} \sqrt {\left(\chi^ {2} (\rho \| \pi) + 1\right) ^ {2} - 1} \cdot \sqrt {2 \mathbb {E} _ {\pi} [ \| \mathbf {x} - \mathbb {E} _ {\pi} [ \mathbf {x} ] \| ^ {4} ] + 6 \mathbb {E} _ {\pi} [ \| \mathbf {x} - \mathbb {E} _ {\pi} [ \mathbf {x} ] \| ^ {2} ] ^ {2}} \\ \leq \frac {1}{2} \sqrt {\left(\chi^ {2} (\rho \| \pi) + 1\right) ^ {2} - 1} \cdot \sqrt {8 \mathbb {E} _ {\pi} [ \| \mathbf {x} - \mathbb {E} _ {\pi} [ \mathbf {x} ] \| ^ {4} ]}. \\ \end{array}
$$
For our application, we have the following. Given a probability distribution  $\rho$  on  $\mathbb{R}^d$ , we let  $\mu(\rho) = \mathbb{E}_{\rho}[\mathbf{x}]$  be its mean, and for  $k \in \mathbb{N}$ ,
$$
M _ {k} (\rho) := \mathbb {E} _ {\rho} [ \| \mathbf {x} - \mu (\rho) \| ^ {k} ] ^ {1 / k}.
$$
So for example we have  $M_2(\rho) = \sqrt{\mathbf{Var}_\rho(\mathbf{x})}$ . We also have  $M_k(\rho) \leq M_\ell(\rho)$  if  $k \leq \ell$ .
Lemma C.2. Let  $\rho$  be a probability distribution on  $\mathbb{R}^d$ . Let  $S \subseteq \mathbb{R}^d$  with  $\rho(S) = \alpha \in (0,1]$ . Then
1.  $\| \mathbb{E}_{\rho^S}[\mathbf{x}] - \mathbb{E}_\rho [\mathbf{x}]\|\leq \sqrt{\frac{1 - \alpha}{\alpha}}\cdot \sqrt{\mathbf{Var}_\rho(\mathbf{x})}.$
2.  $|\mathbf{Var}_{\rho^S}(\mathbf{x}) - \mathbf{Var}_\rho (\mathbf{x})|\leq \frac{\sqrt{2(1 - \alpha^2)}}{\alpha} M_4(\rho)^2.$
In particular, if  $\alpha \in (0,1]$  is such that  $\frac{1}{\alpha^2} \leq 1 + \frac{c^2 M_2(\rho)^4}{2 M_4(\rho)^4}$  for some  $0 \leq c < 1$ , then
$$
\mathbf {V a r} _ {\rho^ {S}} (\mathbf {x}) \geq (1 - c) \mathbf {V a r} _ {\rho} (\mathbf {x}).
$$
Note that the constraint on  $\alpha$  above implies  $\frac{1}{\alpha^2} \leq \frac{3}{2}$ , so  $\alpha \geq \sqrt{2/3}$ . But if  $M_2(\rho) \ll M_4(\rho)$ , then  $1 - \alpha$  will be very small.
Recall also that under some conditions, e.g. if  $\rho$  is log-concave, then we have the reverse bound that
$$
M _ {2} (\rho) \geq C _ {2, 4} M _ {4} (\rho)
$$
for a universal constant  $C_{2,4}$ , so the constraint above is not too restrictive, as it allows  $1 - \alpha$  of constant size.
C.2 Exponential Families with Strongly Convex and Smooth Log-Partition Functions are Sub-Exponential
Let  $\pmb{\theta} \in \Theta$  such that  $\pmb{\theta} + \frac{1}{\beta}\mathbf{u} \in \Theta$  for some  $\beta > 0$  for all unit vectors  $\mathbf{u}$  and such that Assumption A1 holds for  $p_{\pmb{\theta}}$ . Then  $X := \mathbf{u}^{\top}(T(\mathbf{x}) - \mathbb{E}_{\mathbf{x} \sim p_{\pmb{\theta}}} [T(\mathbf{x})])$  is  $SE(L, \beta)$ .
Proof. WLOG, consider  $p_{\theta}$  in the transformed space  $\mathbf{x} \mapsto T(\mathbf{x})$  so that
$$
p _ {\boldsymbol {\theta}} (\mathbf {t}) = h (\mathbf {t}) \exp (\boldsymbol {\theta} ^ {\top} \mathbf {t} - A (\boldsymbol {\theta})) d \mathbf {t},
$$