SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Efficient, Approximate Sampling. There exist several results in sampling which give bounds in TV distance in polynomial time (mixing time bounds) for log-concave distributions, e.g., [15], [3], [33] but which also usually require that the log density is also smooth (in  $\mathbf{x}$ , not  $\theta$ ). There are also proofs for Langevin Monte Carlo when the log density is convex and Lipschitz, not necessarily smooth (e.g., see Chapter 4 of [7]), or under LSI (which is implied by strong log-concavity) with convergence in Renyi divergence (e.g., Chapter 5 of [7]). We can also use the proximal sampler to achieve convergence in KL divergence under log-concavity (e.g., Chapter 8.4 of [7]), which by Pinsker's inequality can bound the TV distance.
Proposition D.4 (Bounded variance step with bias). If  $\mathbb{E}[\| \mathbf{v}_i\| \mid \boldsymbol {\theta}_i]\leq \rho^2$  where  $\mathbf{v}_i = T(\mathbf{z}) - T(\mathbf{x})$  with  $\mathbf{z}\sim p_{p_{\theta_i}^S}$  and  $\mathbf{x}\sim p_{\theta^{*}}^{S}$ , then  $\mathbb{E}[\| \widetilde{\mathbf{v}}_i\| |\boldsymbol {\theta}_i]\leq \rho^{\prime 2}$  for  $\widetilde{\mathbf{v}}_i = T(\bar{\mathbf{z}}) - T(\mathbf{x})$  with  $\widetilde{\mathbf{z}}\sim \widetilde{p}_{p_{\theta_i}^S}$  and  $\mathbf{x}\sim p_{\theta^{*}}^{S}$ , where
$$
\rho^ {\prime 2} = \mathbf {V a r} _ {\tilde {p} _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) - \mathbf {V a r} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) + \rho^ {2} + B ^ {2},
$$
where  $\| \mathbf{b}_i\| = \| \mathbb{E}_{\widetilde{\mathbf{z}}\sim \widetilde{p}_{\boldsymbol{\theta}_i}^S}[T(\widetilde{\mathbf{z}})] - \mathbb{E}_{\mathbf{z}\sim p_{\boldsymbol{\theta}_i}^S}[T(\mathbf{z})]\| < B.$
Proof. As in the proof of the exact sampling version, we can write
$$
\begin{array}{l} \mathbb {E} [ \| \widetilde {\mathbf {v}} _ {i} \| | \boldsymbol {\theta} _ {i} ] = \mathbf {V a r} _ {\widetilde {p} _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) + \mathbf {V a r} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} (T (\mathbf {x})) + \| \mathbb {E} _ {\widetilde {p} _ {\boldsymbol {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} ^ {*}}} [ T (\mathbf {x}) ] \| ^ {2} \\ \leq \mathbf {V a r} _ {\widetilde {p} _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) + \mathbf {V a r} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} (T (\mathbf {x})) + \| \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} [ T (\mathbf {x}) ] \| ^ {2} + \| \mathbf {b} _ {i} \| ^ {2} \\ = \mathbf {V a r} _ {\widetilde {p} _ {\theta_ {i}} ^ {S}} (T (\mathbf {z})) - \mathbf {V a r} _ {p _ {\theta_ {i}} ^ {S}} (T (\mathbf {z})) \\ + \underbrace {\mathbf {V a r} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) + \mathbf {V a r} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} (T (\mathbf {x})) + \| \mathbb {E} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} [ T (\mathbf {z}) ] - \mathbb {E} _ {p _ {\boldsymbol {\theta} ^ {*}} ^ {S}} [ T (\mathbf {x}) ] \| ^ {2}} _ {\leq \rho^ {2}} + \underbrace {\| \mathbf {b} _ {i} \| ^ {2}} _ {\leq B ^ {2}} \\ \leq \mathbf {V a r} _ {\widetilde {p} _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) - \mathbf {V a r} _ {p _ {\boldsymbol {\theta} _ {i}} ^ {S}} (T (\mathbf {z})) + \rho^ {2} + B ^ {2} \\ \end{array}
$$
We can bound the difference  $\mathbf{Var}_{\widetilde{p}_{\theta_i}^S}(T(\mathbf{z})) - \mathbf{Var}_{p_{\theta_i}^S}(T(\mathbf{z}))$  if we have bounds on  $\mathbf{Var}_{\widetilde{p}_{\theta_i}^S}(T(\mathbf{z}))$ , through bounds like in Lemma C.1 if we can say something about the chi-square divergence, or through similar arguments to the bias bound if we assume some bounds on  $\| T(\mathbf{x})\|^2$  over its support.
E Numerical Example
To illustrate how the algorithm performs in different dimensions, we implemented our algorithm for 2-, 5-, 10-, and 20-dimensional exponential distributions. In all cases, the truncation set is the (hyper-)cube  $[0, 2]^d$ . We chose true parameters in all cases which resulted in an initial error at most 2.5. In all cases, we use 1500 iterations and step size 0.01, each repeated 10 times. In the end, all have (average) L2 error at most 0.15. For stability (and to bypass repeating the algorithm multiple times as stated in the analysis), we instead calculated gradients using the average of 10 samples which was sufficient to have stable training results. See Figure 2.
Learning Multi-Dim Exponential Distribution w/ Truncated Data
Figure 2: Learning 2-, 5-, 10-, and 20-dimensional truncated exponential distributions. In all cases, the truncation set is the (hyper-)cube  $[0,2]^d$ .
The wall clock time to finish all 1500 iterations of training for 5-, 10-, and 20-dimensions was  $42.9 \pm 2.2$ ,  $49.1 \pm 6.5$ , and  $61.2 \pm 3.0$  seconds, respectively. We can see that the running time is not doubling with the doubling of dimensions.