SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, to make use of this property we want to establish a procedure to initialize a parameter  $\theta_0$  such that its distance to the optimum  $\theta^*$  is bounded. Then during the optimization procedure we will make sure to make progress toward  $\theta^*$ . The following is the result we will be able to prove after proving some results between truncated and non-truncated quantities in the following Section 3.3.
Corollary 3.5 (Parameter space with non-trivial mass on truncation set). Given  $\theta_0$  such that  $E_{p_{\theta_0}}[T(\mathbf{x})] = \overline{T}$  where  $\overline{T} = \frac{1}{n}\sum_{i=1}^{n} T(\mathbf{x}_i)$  is the empirical mean sufficient statistics given our samples  $\{\mathbf{x}_i\}_{i=1}^n$  for each  $\mathbf{x}_i \sim p_{\boldsymbol{\theta}^*}^S$ , if we define
$$
K = B \left(\boldsymbol {\theta} _ {0}, \frac {\epsilon_ {S} + \mathcal {O} (\log 1 / \alpha)}{\lambda}\right) \cap \Theta
$$
then  $p_{\theta}(S) \geq \alpha^2 \exp \left(-\frac{6L}{\lambda^2} (\epsilon_S + \mathcal{O}(\log 1 / \alpha))^2\right) > 0$  holds  $\forall \theta \in K$  (with probability at least  $1 - \delta$  for  $n \geq \Omega(\log(1 / \delta))$ ). Furthermore,  $\theta^{*} \in K$  (as long as  $\theta^{*} \in \Theta$  satisfies conditions of Claim 1).
This result will follow from Lemma 3.4, Claim 1, Lemma 3.7, and Lemma 3.9 in the next section.
3.3 Initialization with Empirical Samples and Non-truncated MLE
Given samples from the truncated density  $p_{\theta^{*}}^{S}$ , one may first try to solve the non-truncated empirical MLE problem to find a parameter  $\theta_0$  without truncation and hope that it is good enough. (We will show that it is.) In order to understand how good this initial guess is, we need to establish some relationships between the truncated and non-truncated density.
Lemma 3.6 (Truncated vs. Non-truncated Mean Sufficient Statistics for General Densities). Let  $\rho$  be a probability distribution on  $\mathbb{R}^d$  (not necessarily from an exponential family). Let  $S \subseteq \mathbb{R}^d$  with  $\rho(S) > 0$ . Then
$$
\| \mathbb {E} _ {\rho^ {S}} [ \mathbf {x} ] - \mathbb {E} _ {\rho} [ \mathbf {x} ] \| \leq \sqrt {\frac {1 - \rho (S)}{\rho (S)}} \cdot \sqrt {\mathbf {V a r} _ {\rho} (\mathbf {x})}.
$$
Proof of this lemma and several related quantities for general truncated densities is in Appendix C.1. In low dimensions, this variance term may effectively be a constant; however, in high-dimensional settings this term can grow with dimension (which is undesirable if we want an efficient algorithm). Given more assumptions about the density, we can get better dimension-free bounds which generalize the results from the Gaussian case.
Claim 1. Let  $\pmb{\theta} \in \Theta$  such that  $\pmb{\theta} + \frac{1}{\beta}\mathbf{u} \in \Theta$  for some  $\beta > 0$  for all unit vectors  $\mathbf{u}$  and such that Assumption A1 holds for  $p_{\theta}$  from an exponential family. Then  $X := \mathbf{u}^{\top}(T(\mathbf{x}) - \mathbb{E}_{\mathbf{x} \sim p_{\theta}}[T(\mathbf{x})])$  is  $SE(L, \beta)$ . (Proof provided in Appendix C.2.)
Remark. This claim allows us to have concentration of the empirical mean sufficient statistics to its population mean for the non-truncated distribution. The prior work [10] analyzed the mean and covariance of a Gaussian separately, but for instance to establish bounds on distances between the truncated and non-truncated mean parameter, it made use of Gaussian concentration inequalities (which are tighter than sub-exponential ones). The relationship between the truncated density and the non-truncated one will allow us to say that the empirical truncated mean sufficient statistics is also somewhat close to the non-truncated population mean.
Lemma 3.7 (Concentration of Empirical vs. Non-truncated Mean Sufficient Statistics). Suppose  $\theta^{*}$  satisfies the conditions of Claim 1 and  $p_{\theta^{*}}(S) = \alpha >0$ . Let  $\overline{T} = \frac{1}{n}\sum_{i = 1}^{n}T(\mathbf{x}_i)$  be the empirical mean sufficient statistics given our samples  $\{\mathbf{x}_i\}_{i = 1}^n$  each  $\mathbf{x}_i\sim p_{\theta^*}^S$ . Let  $\epsilon_S > 0$ . For  $n\geq \Omega \left(\frac{2\beta}{\epsilon_S}\log \left(\frac{1}{\delta}\right)\right)$ , with probability at least  $1 - \delta$ ,
$$
\left\| \bar {T} - \mathbb {E} _ {p _ {\theta^ {*}}} [ T (\mathbf {x}) ] \right\| \leq \epsilon_ {S} + \mathcal {O} (\log 1 / \alpha).
$$
It should be noted that it suffices to consider  $\epsilon_S$  as some independent constant (and not related to the  $\epsilon$  accuracy parameter in the main theorem). However, by taking  $n$  at least  $\widetilde{\mathcal{O}}(k/\epsilon^2)$  as stated in the main theorem, this will be small. See proof in Appendix C.4.
Remark. At a high level, the truncated samples can be thought of as  $\mathcal{O}(n / \alpha)$  samples from the non-truncated distribution (keeping only those in  $S$ ), and each are "not too far" (depending on how much mass the set  $S$  has under the non-truncated distribution) from the non-truncated mean due to concentration. Note that the  $\mathcal{O}(\log 1 / \alpha)$  term will never disappear even as we increase  $n$ , which quantifies the inherent bias that the truncated mean sufficient statistics will have with respect to the non-truncated one (and is large if the mass  $\alpha$  is small). From this, we can also say something about the population mean sufficient statistics, one on the truncated distribution and one on the non-truncated.
Corollary 3.8 (Truncated vs. Non-truncated Mean Sufficient Statistics). Let  $\theta$  satisfy the conditions of Claim 1 and  $p_{\theta}(S) > 0$ . Then
$$
\| \mathbb {E} _ {p _ {\boldsymbol {\theta}} ^ {S}} [ T (\mathbf {x}) ] - \mathbb {E} _ {\mathbf {x} \sim p _ {\boldsymbol {\theta}}} [ T (\mathbf {x}) ] \| \leq \mathcal {O} (\log 1 / p _ {\boldsymbol {\theta}} (S)).
$$
The proof follows from the preceding lemma, replacing  $\alpha$  with  $p_{\theta}(S)$  and taking  $n\to \infty$ . Compare this to the Gaussian case [10], where the mean and truncated means were bounded as  $\| \pmb {\mu} - \pmb{\mu}_S\| \leq \mathcal{O}(\sqrt{\log 1 / p_\theta(S)})$  and separately the covariances were bounded as  $\| \Sigma^{-1 / 2}\Sigma_S\Sigma^{-1 / 2} - \mathbf{I}\| _F\leq$ $\mathcal{O}(\log 1 / p_{\theta}(S))$ . Note the smaller  $\mathcal{O}(\sqrt{\log 1 / p_{\theta}(S)})$  quantity due to tighter Gaussian concentration vs. the sub-exponential rate.
Once we have bounds on the norm of the difference between the truncated and non-truncated mean sufficient statistics, we can bound distance in parameter space. The following completes this.
Lemma 3.9 (Non-truncated MLE Solution Distance to  $\pmb{\theta}^{*}$ ). Suppose  $\pmb{\theta}^{*}$  satisfies the conditions of Claim 1 and  $p_{\pmb{\theta}^{*}}(S) = \alpha > 0$ . Let  $\pmb{\theta}_{0}$  be such that  $\mathbb{E}_{p_{\pmb{\theta}_{0}}}[T(\mathbf{x})] = \overline{T}$  where  $\overline{T} = \frac{1}{n}\sum_{i=1}^{n} T(\mathbf{x}_{i})$  given each  $\mathbf{x}_{i} \sim p_{\pmb{\theta}^{*}}^{S}$ . Let  $\epsilon_{S} > 0$  and  $n > \Omega\left(\frac{2\beta}{\epsilon_{S}}\log(1/\delta)\right)$ . Then w.p. at least  $1 - \delta$ ,
$$
\left\| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} ^ {*} \right\| \leq \frac {1}{\lambda} \left(\mathcal {O} (\log 1 / \alpha) + \epsilon_ {S}\right).
$$
Proof. Define  $\bar{\ell}^{\mathrm{unr}}(\pmb {\theta})\coloneqq \mathbb{E}_{\mathbf{x}\sim p_{\pmb{\theta}_0}}[-\log p_{\pmb{\theta}}(\mathbf{x})]$ . Its gradient and Hessian calculations can be done similarly to  $\bar{\ell} (\pmb {\theta})$ , the truncated version, but with  $S = \mathcal{X}$  the full support of the distribution.
Since  $E_{\mathbf{z}\sim p_{\boldsymbol{\theta}^*}}[T(\mathbf{z})] - \mathbb{E}_{\mathbf{x}\sim p_{\boldsymbol{\theta}_0}}[T(\mathbf{x})] = \nabla \bar{\ell} (\boldsymbol{\theta}^*)^{\mathrm{unr}}$  is the gradient of the untruncated negative log-likelihood whose optimum is at  $\pmb{\theta}_0$ , by A1 this gives
$$
\| \nabla \overline {{\ell}} ^ {\mathrm {u n t r}} (\boldsymbol {\theta} ^ {*}) - \underbrace {\nabla \overline {{\ell}} ^ {\mathrm {u n t r}} (\boldsymbol {\theta} _ {0})} _ {0} \| \geq \lambda \| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} ^ {*} \| \Rightarrow \| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} ^ {*} \| \leq \frac {1}{\lambda} \| \nabla \overline {{\ell}} ^ {\mathrm {u n t r}} (\boldsymbol {\theta} ^ {*}) \|
$$
where the result follows from the fact that  $\mathbb{E}_{\mathbf{z}\sim p_{\theta_0}}[T(\mathbf{z})] = \overline{T}$  and  $\| \mathbb{E}_{\mathbf{z}\sim p_{\theta^*}}[T(\mathbf{z})] - \overline{T}\| \leq \mathcal{O}(\log 1 / \alpha +\epsilon_S)$  w.p.  $1 - \delta$  from Lemma 3.7.
Note that this result combined with Lemma 3.4 gives Corollary 3.9.
3.4 Analysis of Projected Stochastic Gradient Descent Algorithm
Now we have all the tools we need to analyze the main algorithm. For ease of notation, define  $d(\alpha) \coloneqq \epsilon_S + \mathcal{O}(\log 1 / \alpha)$  which is a constant that depends on  $\alpha$ . The following describes the projected stochastic gradient descent algorithm referenced by Theorem 3.1.
Algorithm 1 Projected SGD Algorithm Given Truncated Samples
Given  $\{\mathbf{x}_i\}_{i = 1}^n$  , each  $\mathbf{x}_i\sim p_{\theta^*}^S$    
Initial  $\pmb {\theta}_0\in \mathbb{R}^k$  s.t.  $\mathbb{E}_{\mathbf{z}\sim p_{\theta_0}}[T(\mathbf{z})] = \overline{T}$  , where  $\overline{T} = \frac{1}{n}\sum_{i}T(\mathbf{x}_i)$    
for  $i = 0,\dots ,N$  do  
 $\mathbf{v}_i =$  SampleGradient  $(\mathbf{x}_i,\pmb {\theta}_i)$ $\pmb{\theta}_{i + 1}\gets \pmb{\theta}_i - \eta \mathbf{v}_i$    
Project  $\pmb{\theta}_{i + 1}$  onto  $K = B(\pmb {\theta}_0,\frac{d(\alpha)}{\lambda})\cap \Theta$    
end for   
Return  $\pmb{\theta}_T$
Algorithm 2 SampleGradient
Input:  $\mathbf{x},\theta$    
while True do Sample  $\mathbf{z}\sim p_{\theta}$  if  $\mathbb{1}\{\mathbf{z}\in S\}$  via membership oracle then Return  $T(\mathbf{z}) - T(\mathbf{x})$  end if   
end while
Given the results from the previous sections, we can now prove the main result. The analysis is based on that of Chapter 5 (Theorem 5.7) of [20] which we modify and state below:
Theorem 3.10 (SGD Convergence). Let  $f$  be a  $\lambda$ -strongly convex function. Let  $\pmb{\theta}^{*} \in \arg \min_{\pmb{\theta} \in K} f(\pmb{\theta})$ . Consider the sequence  $\{\pmb{\theta}_t\}_{t=1}^N$  generated by SGD (Algorithm 3) and  $\{\mathbf{v}_t\}_{t=1}^N$  the sequence random vectors satisfying  $\mathbb{E}[\mathbf{v}_t \mid \pmb{\theta}_t] = \nabla f(\pmb{\theta}_t)$  and  $\mathbb{E}[||\mathbf{v}_t||^2 \mid \pmb{\theta}_t] < \rho^2$  for all  $t$ , with a constant step size  $\eta$  satisfying  $0 < \eta < \frac{1}{\lambda}$ . It follows that for  $t \geq 0$ ,
$$
\mathbb {E} \left\| \boldsymbol {\theta} _ {t} - \boldsymbol {\theta} ^ {*} \right\| ^ {2} \leq (1 - 2 \eta \lambda) ^ {t} \left\| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} ^ {*} \right\| ^ {2} + \frac {\eta}{\lambda} \rho^ {2}.
$$
The proof is adapted and reproduced in Appendix D.1 for completeness. To apply the above theorem we need to take care of statistical problems:
(i) strong convexity  $f$  is a strongly convex function over  $K$  a convex set
(ii) smoothness  $f$  is a Lipschitz-smooth function over  $K$
(iii) feasibility of optimal solution  $\theta^{*}\in K$
(iv) bounded variance step for all  $t$ ,  $\mathbb{E}[\| \mathbf{v}_t\|^2\mid \pmb {\theta}_t] < \rho^2$  for some  $\rho^2$
and algorithmic ones:
(a) initial feasible point efficiently compute an initial feasible point  $\theta_0$
(b) unbiased gradient estimation efficiently sample an unbiased estimate of  $\nabla f (= \nabla \bar{\ell})$
(c) efficient projection efficiently project to parameter space  $K$
Statistical problems. Firstly, (iii) is assumed. (i-ii) is addressed by Lemmas 3.2, 3.3, 3.6, 3.9, 3.7, 3.4. In particular, we can initialize with  $\pmb{\theta}_0$  such that  $\| \pmb{\theta}_0 - \pmb{\theta}^*\| \leq \frac{d(\alpha)}{\lambda}$  by Lemmas 3.6, 3.9, 3.7, with probability at least  $1 - \delta$ . Given this  $\pmb{\theta}_0$ , we can construct  $K = B(\pmb{\theta}_0, \frac{d(\alpha)}{\lambda}) \cap \Theta$  which has the property that
$$
\left\| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {*} \right\| \leq \frac {2}{\lambda} d (\alpha), \forall \boldsymbol {\theta} \in K.
$$
Thus by Lemma 3.4, we will also have
$$
p _ {\boldsymbol {\theta}} (S) \geq \alpha^ {2} \exp \left(- 6 \frac {\kappa}{\lambda} \cdot (d (\alpha)) ^ {2}\right) > 0, \forall \boldsymbol {\theta} \in K
$$
where  $\kappa = L / \lambda$  is the condition number. Since we are projecting to  $K$  in which all parameters have non-trivial mass, our objective remains strongly convex. In particular, our objective  $\bar{\ell} (\pmb {\theta})$  has
$$
\lambda_ {S} I \preceq \nabla^ {2} \bar {\ell} (\boldsymbol {\theta}) \preceq L _ {S} I, \forall \boldsymbol {\theta} \in K,
$$
where  $\lambda_{S} = \frac{1}{2}\left(\frac{\alpha^{2}\exp\left(-6\frac{\kappa}{\lambda}\cdot(d(\alpha))^{2}\right)}{4C\deg}\right)^{2\deg}\lambda$  and  $L_{S} = \frac{\exp\left(6\frac{\kappa}{\lambda}\cdot(d(\alpha))^{2}\right)}{\alpha^{2}} L$  are some constants which depend on  $\alpha, \lambda, L$ , and the maximum degree, deg, of the sufficient statistics. It remains to address (iv), which is done by the following lemma.
Lemma 3.11 (Bounded variance step). Let  $\mathbf{v}_i$  denote the output of SampleGradient  $(\mathbf{x}_i, \boldsymbol{\theta}_i)$  at any iteration  $i \in [N]$ . Provided that  $\| \mathbb{E}_{p_\theta}^S[T(\mathbf{x})] - \mathbb{E}_{p_\theta}[T(\mathbf{x})]\| \leq \mathcal{O}(\log 1 / p_\theta(S))$  for all  $\boldsymbol{\theta} \in K$ ,
$$
\mathbb {E} \left[ \left\| \mathbf {v} _ {i} \mid \boldsymbol {\theta} _ {i} \right\| ^ {2} \right] \leq k L _ {S} + k L + (1 + 2 \kappa) ^ {2} \left(\mathcal {O} \left(\log 1 / p _ {\boldsymbol {\theta} _ {i}} (S)\right)\right) ^ {2}.
$$
Proof. At any iteration  $i$  (arbitrary),
$$
\begin{array}{l} \mathbb {E} [ \| \mathbf {v} _ {i} \| ^ {2} \mid \boldsymbol {\theta} _ {i} ] = \mathbb {E} _ {(\mathbf {z}, \mathbf {x}) \sim p _ {\boldsymbol {\theta} _ {i}} ^ {S} \otimes p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} \left[ \| T (\mathbf {z}) - T (\mathbf {x}) \| ^ {2} \right] \\ = \mathbb {E} _ {(\mathbf {z}, \mathbf {x}) \sim p _ {\boldsymbol {\theta} _ {i}} ^ {S} \otimes p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} \left[ \| T (\mathbf {z}) \| ^ {2} - 2 \langle T (\mathbf {z}), T (\mathbf {x}) \rangle + \| T (\mathbf {x}) \| ^ {2} \right] \\ = \operatorname {T r} (\mathbf {C o v} [ T (\mathbf {z}) ]) + (\mathbb {E} [ \| T (\mathbf {z}) \| ]) ^ {2} + \operatorname {T r} (\mathbf {C o v} [ T (\mathbf {x}) ]) + (\mathbb {E} [ \| T (\mathbf {x}) \| ]) ^ {2} - 2 \langle \mathbb {E} [ T (\mathbf {z}) ], \mathbb {E} [ T (\mathbf {x}) ] \rangle \\ = \operatorname {T r} \left(\mathbf {C o v} [ T (\mathbf {z}) ]\right) + \operatorname {T r} \left(\mathbf {C o v} [ T (\mathbf {x}) ]\right) + \left\| \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} [ T (\mathbf {x}) ] \right\| ^ {2} \\ \leq k L _ {S} + k L + (1 + 2 \kappa) ^ {2} (\mathcal {O} (\log 1 / p _ {\boldsymbol {\theta} _ {i}} (S))) ^ {2} \\ \end{array}
$$
In the last step, we've used the fact that
$$
\begin{array}{l} \| \mathbb {E} _ {p _ {\pmb {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\pmb {\theta} ^ {*}} ^ {S}} [ T (\mathbf {x}) ] \| \leq \| \mathbb {E} _ {p _ {\pmb {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\pmb {\theta} _ {i}}} [ T (\mathbf {z}) ] \| + \| \mathbb {E} _ {p _ {\pmb {\theta} _ {i}}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\pmb {\theta} ^ {*}} ^ {S}} [ T (\mathbf {z}) ] \| \\ = \| \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}}} [ T (\mathbf {z}) ] \| + \| \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {0}}} [ T (\mathbf {z}) ] \| \\ \leq \mathcal {O} (\log 1 / p _ {\boldsymbol {\theta} _ {i}} (S)) + (2 L / \lambda) (\mathcal {O} (\log 1 / p _ {\boldsymbol {\theta} _ {i}} (S)) + \epsilon_ {S}) \\ \end{array}
$$
by assumption, smoothness, Cor. 3.8 and Lemma 3.9.
Algorithmic problems. For the algorithmic problems, by Cor. 3.8 and Lemmas 3.9, 3.7, we can address (a) by solving the empirical MLE problem with no truncation. Given that we can efficiently sample exactly (or approximately see Appendix D.2) from the non-truncated  $p_{\theta}$  for any  $\theta$ , we can sample unbiased gradients via Algorithm 2 with expected  $\mathcal{O}(1 / p_{\theta_i}(S)) = \mathcal{O}\left(\frac{\exp(6\kappa \cdot (d(\alpha))^2)}{\alpha^2}\right)$  samples at each step  $t$  to address (b). Point (c) can be done efficiently, since our parameter space is a simple intersection of Euclidean balls if we choose  $\Theta$  to be a Euclidean ball that sits inside the whole parameter space which contains  $\theta^*$ .
Let  $D(k, L, \lambda, \alpha) = \frac{k(L_S + L) + (1 + 2\kappa)^2(6\kappa(d(\alpha))^2 - 2\log\alpha)^2}{\lambda_S^2\epsilon^2}$ . Putting everything together, to get  $\mathbb{E}\|\pmb{\theta}_N - \pmb{\theta}^*\|^2 \leq \epsilon^2$ , the number of iterations and samples should be
$$
N \geq \max \left\{D (k, L, \lambda , \alpha), \frac {1}{2} \right\} \log \left(\frac {2 d (\alpha)}{\lambda \epsilon^ {2}}\right),
$$
provided that  $\eta = \min \left\{\frac{\lambda_S\epsilon^2}{2\rho^2},\frac{1}{\lambda_S}\right\}$ , applying Lemma D.1 to the bound from Theorem 3.10 with  $A = \frac{\rho^2}{\lambda_S}$ ,  $C = \lambda_S$ ,  $\mu = 2\lambda_S$ , and  $\rho^2 = kL_S + kL + (1 + 2\kappa)^2 (\mathcal{O}(\log 1 / p_\theta (S)))^2$  by Lemma 3.11. Further, Lemma 3.4 guarantees  $\mathcal{O}(\log 1 / p_{\pmb{\theta}}(S))\leq \mathcal{O}\left(\log \frac{\exp(6\kappa\cdot(d(\alpha))^2)}{\alpha^2}\right)$  for all  $\pmb {\theta}\in K$
In probability, we get  $\mathbb{P}(\| \pmb{\theta}_N - \pmb{\theta}^*\|^2 \geq 3\epsilon^2) \leq 1/3$  by Markov's inequality. Then we can amplify the probability of success to  $1 - \delta$  by repeating the procedure from scratch  $\log 1/\delta$  times, as in [10]. Given a polynomial time (poly  $(m, k, 1/\epsilon)$ ) algorithm  $A_S$  to sample from  $p_\theta$  for all  $\pmb{\theta}$ , each iteration takes expected  $\mathcal{O}(1/p_{\pmb{\theta}_i}(S)) = \mathcal{O}\left(\frac{\exp(6\frac{\kappa}{\lambda} \cdot (d(\alpha))^2)}{\alpha^2}\right)$  times the running time of  $A_S$  plus the projection step (which is also efficient). This completes the result of Theorem 3.1.
Remark. In the Gaussian case, the sample complexity was given in terms of  $m$ , the dimension of the  $\mathbf{x}$  and was stated in [10] as  $\widetilde{\mathcal{O}}(m^2/\epsilon^2)$ . For multivariate Gaussian, the dimension  $k$  of  $\pmb{\theta}$  is the dimension of  $[\mathbf{x},\mathbf{x}\mathbf{x}^\top]$  (vectorized) which is  $m + m^2$ , thus we can recover the previous result.
Numerical example. While a lot of the analyses shown can seem complicated, they give some guarantees for a rather simple algorithm: simply initialize your parameters using MLE as if there is no truncation at all, then run projected stochastic gradient descent using rejection sampling. We do not even need to describe the truncation set  $S$  as long as we can query whether a point is inside  $S$  or not efficiently. As a proof of concept for the use of this algorithm, we've implemented our simple projected SGD algorithm to learn the parameters of multivariate exponential distributions, given truncated samples. To save space in the main body for exposition of the theoretical results, we've included the results and details in Appendix E.
To our knowledge, this work is the first which develops a computationally and statistically efficient algorithm for learning from samples truncated to very general sets  $S$  of the form in [10] in high dimensions whose distribution does not rely on Gaussianity. This work also has interesting implications for learning general log-concave distributions through applying the Taylor theorem to the log density as in [12] and for sampling (e.g., as in [30]). Through generalizing the previous Gaussian results to general exponential families, we also extract the broader abstract properties (e.g., concentration and anti-concentration) of distributions for which the previously proposed projected stochastic gradient descent procedure still applies. It would be interesting to understand how much these results can be generalized to densities beyond exponential families, or even to extend the current work presented from expected polynomial running time to deterministic polynomial running time. We hope that our work provides the foundation for future work in this direction.
We thank Tim Kunisky for insightful discussions during the preparation of this work. Jane Lee was supported by a GFSD fellowship sponsored by the U.S. National Security Agency (NSA).
[36] Edwin James George Pitman. Sufficient statistics and intrinsic accuracy. In Mathematical Proceedings of the Cambridge Philosophical society, volume 32, pages 567-579. Cambridge University Press, 1936.
[37] Orestis Plevrakis. Learning from censored and dependent data: The case of linear dynamics. CoRR, abs/2104.05087, 2021.
[38] Mathias Raschke. Inference for the truncated exponential distribution. Stochastic environmental research and risk assessment, 26(1):127-138, 2012.
[39] Adrien Saumard and Jon A. Wellner. Log-concavity and strong log-concavity: A review. Statistics Surveys, 8(none):45 - 114, 2014.
[40] Abhin Shah, Raaz Dwivedi, Devavrat Shah, and Gregory W Wornell. On counterfactual inference with unobserved confounding. arXiv preprint arXiv:2211.08209, 2022.
[41] Abhin Shah, Devavrat Shah, and Gregory Wornell. A computationally efficient method for learning exponential family distributions. Advances in neural information processing systems, 34:15841-15854, 2021.
[42] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014.
[43] John W Tukey. Sufficiency, truncation and selection. The Annals of Mathematical Statistics, pages 309-311, 1949.
[44] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.
A Proofs and Calculations Regarding the Objective
A.1 The Truncated Negative Expected Log-Likelihood Function
The negative log-likelihood that  $\mathbf{x} \in S$  is a sample of  $p_{\boldsymbol{\theta}}^{S}(\mathbf{x})$  is
$$
\underbrace{\ell(\theta,\mathbf{x})}_{-\log p_{\boldsymbol{\theta}}^{S}(\mathbf{x})}:= -\log h(\mathbf{x}) - \boldsymbol{\theta}^{\top}T(\mathbf{x}) + \log \int_{S}h(\mathbf{x}^{\prime})\exp (\theta^{\top}T(\mathbf{x}^{\prime}))d\mathbf{x}^{\prime}.
$$
Its gradient w.r.t.  $\theta$  is
$$
\begin{array}{l} \nabla \ell (\pmb {\theta}, \mathbf {x}) = - T (\mathbf {x}) + \frac {\int_ {S} T (\mathbf {x} ^ {\prime}) h (\mathbf {x} ^ {\prime}) \exp (\theta^ {\top} T (\mathbf {x} ^ {\prime})) d \mathbf {x} ^ {\prime}}{\int_ {S} h (\mathbf {x} ^ {\prime}) \exp (\theta^ {\top} T (\mathbf {x} ^ {\prime})) d \mathbf {x} ^ {\prime}} \\ = - T (\mathbf {x}) + \frac {\int_ {S} T (\mathbf {x} ^ {\prime}) h (\mathbf {x} ^ {\prime}) \exp (\theta^ {\top} T (\mathbf {x} ^ {\prime}) - A (\boldsymbol {\theta})) d \mathbf {x} ^ {\prime}}{\int_ {S} h (\mathbf {x} ^ {\prime}) \exp (\boldsymbol {\theta} ^ {\top} T (\mathbf {x} ^ {\prime}) - A (\boldsymbol {\theta})) d \mathbf {x} ^ {\prime}} \\ = - T (\mathbf {x}) + \mathbb {E} _ {\mathbf {z} \sim p _ {\theta} ^ {\mathrm {S}}} [ T (\mathbf {z}) ] \\ \end{array}
$$
The Hessian is