SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1.  $\left| {\mathbb{E}}_{S}\left\lbrack  {{p}_{ii}{p}_{ij}\left( {{\eta }_{i}^{\prime }\left( {{\mu }_{{k}^{\prime }}^{T}{u}_{{k}^{\prime }} - x{\mu }_{{k}^{\prime }}^{T}{v}_{{k}^{\prime }}}\right) }\right) \left( {{\theta }_{k}^{T}{u}_{k}{\xi }_{j}}\right) }\right\rbrack  \right|$
2.  $\left| \mathbb{E}_S \left[ p_{ii} p_{ij} \left( \eta_i' \left( \mu_{k'}^T u_{k'} - x \mu_{k'}^T v_{k'} \right) \right) \left( \theta_k^T u_k \xi_i \right) \right] \right| \leq C q_{ii} q_{ij} \sigma_{\eta_i'}^2 \sigma_{\xi_i}^2 \| \theta_{k'} \| \| \theta_k \| \| \theta_k \| \leq C q_{ii} q_{ij} \| \theta_{k'} \| ^2 \| \theta_{k'} \| \| \theta_k \| ^4$ .
3.  $\left| {\mathbb{E}}_{S}\left\lbrack  {{p}_{ii}{p}_{ij}\left( {{y}^{\prime }\left( {{\mu }_{{k}^{\prime }}^{T}{u}_{{k}^{\prime }} - x{\mu }_{{k}^{\prime }}^{T}{v}_{{k}^{\prime }}}\right) }\right) \left( {{\theta }_{k}^{T}{u}_{k}{\xi }_{j}}\right) }\right\rbrack  \right|$
4.  $\left|\mathbb{E}_S\left[p_{ii}p_{ij}\left(y'\left(\mu_{k'}^T u_{k'} - x\mu_{k'}^T v_{k'}\right)\right)\left(\theta_k^T u_k\xi_i\right)\right]\right|$ $\leq$ $C q_{ii}q_{ij}\sigma_{\xi_i}^2\| \theta_k\| \| \theta_k\| \| \theta_{k'}^{\parallel}\| \quad \leq$ $C q_{ii}q_{ij}\| \theta_k\| ^4\| \theta_{k'}^{\parallel}\|$
5.  $\left| \alpha_{k} \mathbb{E}_{S} \left[ p_{ii} p_{ij} \left( \eta_{i}' \left( \mu_{k'}^{T} u_{k'} - x \mu_{k'}^{T} v_{k'} \right) \right) \left( \theta_{k}^{T} u_{k} y \right) \right] \right| \leq C \alpha_{k} q_{ii} q_{ij} \sigma_{\eta_{i}'}^{2} \| \theta_{k'} \| \| \theta_{k} \| \| \theta_{k}^{\parallel} \| = C \alpha_{k} q_{ii} q_{ij} \| \theta_{k'}^{\perp} \|^{2} \| \theta_{k'} \| \| \theta_{k} \| \| \theta_{k}^{\parallel} \|$
6.  $\left| \alpha_{k} \mathbb{E}_{S} \left[ p_{ii} p_{ij} \left( y' \left( \mu_{k'}^T u_{k'} - x \mu_{k'}^T v_{k'} \right) \right) (\eta_i y) \right] \right| \leq C \alpha_{k} q_{ii} q_{ij} \sigma_{\eta_i}^2 \| \theta_k \| \| \theta_{k'}^{\parallel} \| \| \theta_k^{\parallel} \| = C \alpha_{k} q_{ii} q_{ij} \| \theta_k^{\perp} \| ^2 \| \theta_k \| \| \theta_{k'}^{\parallel} \| \| \theta_k^{\parallel} \|.$
7.  $\left| \alpha_{k} \mathbb{E}_{S} \left[ p_{ii} p_{ij} \left( y'(-x\zeta_{i}') \right) \left( \theta_{k}^{T} u_{k} y \right) \right] \right| \leq C \alpha_{k} q_{ii} q_{ij} \sigma_{\zeta_{i}}^{2} \| \theta_{k'} \| \| \theta_{k} \| \| \theta_{k}^{\parallel} \| \leq C \alpha_{k} q_{ii} q_{ij} \| \theta_{k'}^{\parallel} \|^{2} \| \theta_{k'} \| \| \theta_{k} \| \| \theta_{k}^{\parallel} \|$ .
Combining the bounds on these 7 terms, proves the lemma:
We now prove the lemmas on the non-junk terms.
Proof of Lemma E.9.
$$
\begin{array}{l} \mathbb {E} _ {S} \left[ p _ {i i} p _ {i j} \left(\left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(2 \mu_ {k} ^ {T} u _ {k} \alpha_ {k} \left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) \right] \\ = \mathbb {E} _ {S} \left[ q _ {i i} q _ {i j} \left((\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right)\left(2 \mu_ {k} ^ {T} u _ {k} \alpha_ {k} (\theta_ {k} ^ {\parallel}) ^ {T} u _ {k}\right)\right] + \mathbb {E} _ {S} \left[ \right. (p _ {i i} p _ {i j} - q _ {i i} q _ {i j}) \left((\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right)\left(2 \mu_ {k} ^ {T} u _ {k} \alpha_ {k} (\theta_ {k} ^ {\parallel}) ^ {T} u _ {k}\right) \biggr ] \\ = 2 \alpha_ {k} q _ {i i} q _ {i j} \theta_ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}} \theta_ {k} ^ {T} \mu_ {k} + 2 \alpha_ {k} q _ {i i} q _ {i j} \mathbb {E} _ {S} \left[ \left(\frac {p _ {i i} p _ {i j}}{q _ {i i} q _ {i j}} - 1\right) \left((\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(\mu_ {k} ^ {T} u _ {k} (\theta_ {k} ^ {\parallel}) ^ {T} u _ {k}\right) \right]. \\ \end{array}
$$
Now by Claim E.16, we have  $\left|\frac{p_{ii}p_{ij}}{q_{ii}q_{ij}} - 1\right| \leq Z_iZ_j - 1$  (where the variable's  $Z_i, Z_j$  are defined in the Claim E.16) so
$$
\begin{array}{l} \left| \mathbb {E} _ {S} \left[ \left(\frac {p _ {i i} p _ {i j}}{q _ {i i} q _ {i j}} - 1\right) \left((\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(\mu_ {k} ^ {T} u _ {k} (\theta_ {k} ^ {\parallel}) ^ {T} u _ {k}\right) \right] \right| \leq \mathbb {E} _ {S} \left[ (Z _ {i} Z _ {j} - 1) \left| (\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}} \right| \left| \mu_ {k} ^ {T} u _ {k} (\theta_ {k} ^ {\parallel}) ^ {T} u _ {k} \right| \right] \\ \leq C \left(\| \theta_ {k} \| ^ {2} + \| \theta_ {k ^ {\prime}} \| ^ {2}\right) \| \theta_ {k ^ {\prime}} ^ {\parallel} \| \| \theta_ {k} ^ {\parallel} \|. \\ \end{array}
$$
Here the second inequality follows from applying Lemma E.5 first, and then Lemma E.4 repeatedly for the remainder of the variables in  $S$ . This proves the lemma. Note that we need to apply Lemma E.5 several times to a single variable  $X \in S$ . Indeed we can write
$$
\begin{array}{l} \left(Z _ {i} Z _ {j} - 1\right) \left| \left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}} \right| \left| \mu_ {k} ^ {T} u _ {k} \left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k} \right| = (\mathbb {E} _ {\ell} \exp \left(| t _ {\ell} X |\right) S _ {\ell} - 1) B | X | ^ {c} \\ = \left(\mathbb {E} _ {\ell} S _ {\ell} \left(\exp \left(| t _ {\ell} X |\right) - 1\right)\right) B | X | ^ {c} + \left(\mathbb {E} _ {\ell} S _ {\ell} - 1\right)) B | X | ^ {c} \\ \end{array}
$$
for some distribution on  $\ell$ , and for some terms  $S_{\ell}, t_{\ell}$ , and  $B$  that are independent of  $X$ , and  $c \in \{0,1,2\}$ . Then to take the expectation of this term over  $X$ , we first apply Lemma E.5 to on  $X$  to the first term, and iteratively apply Lemma E.5 to the random variables appearing in the next terms.
Proof of Lemma E.11.
$$
\begin{array}{l} \frac {1}{1 - x ^ {2}} \mathbb {E} _ {S} \left[ p _ {i i} p _ {i j} \theta_ {k} ^ {T} h _ {1} (S) \right] = \mathbb {E} _ {S} \left[ p _ {i i} p _ {i j} \left(\left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(2 \left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k} \alpha_ {k} \left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) \right] \\ = \mathbb {E} _ {S} \left[ q _ {i i} q _ {i j} \left(\left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(2 \alpha_ {k} \left(\left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) ^ {2}\right) \right] \\ + \mathbb {E} _ {S} \left[ \left(p _ {i i} p _ {i j} - q _ {i i} q _ {i j}\right) \left(\left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(2 \alpha_ {k} \left(\left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) ^ {2}\right) \right] \\ = 2 \alpha_ {k} q _ {i i} q _ {i j} \theta_ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}} \| \theta_ {k} ^ {\parallel} \| ^ {2} + 2 \alpha_ {k} q _ {i i} q _ {i j} \mathbb {E} _ {S} \left[ \left(\frac {p _ {i i} p _ {i j}}{q _ {i i} q _ {i j}} - 1\right) \left((\theta_ {k ^ {\prime}} ^ {\parallel}) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left((\theta_ {k} ^ {\parallel}) ^ {T} u _ {k}\right) ^ {2} \right]. \\ \end{array}
$$
Now by Claim E.16, we have  $\left|\frac{p_{ii}p_{ij}}{q_{ii}q_{ij}} - 1\right| \leq Z_iZ_j - 1$ , so
$$
\begin{array}{l} \left| \mathbb {E} _ {S} \left[ \left(\frac {p _ {i i} p _ {i j}}{q _ {i i} q _ {i j}} - 1\right) \left(\left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}}\right) \left(\left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) ^ {2} \right] \right| \leq \mathbb {E} _ {S} \left[ \left(Z _ {i} Z _ {j} - 1\right) \left| \left(\theta_ {k ^ {\prime}} ^ {\parallel}\right) ^ {T} u _ {k ^ {\prime}} u _ {k ^ {\prime}} ^ {T} \mu_ {k ^ {\prime}} \right| \left(\left(\theta_ {k} ^ {\parallel}\right) ^ {T} u _ {k}\right) ^ {2} \right] \\ \leq C \left(\| \theta_ {k} \| ^ {2} + \| \theta_ {k ^ {\prime}} \| ^ {2}\right) \| \theta_ {k ^ {\prime}} ^ {\|} \| \theta_ {k} ^ {\|} \| ^ {2}, \\ \end{array}
$$
Again the second inequality follows from applying Lemma E.5 first (several times as described in the previous lemma), and then Lemma E.4 repeatedly for the remainder of the variables in  $S$ . Taking absolute values proves the lemma.