SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now complete the proof of Lemma C.1 (with  $\delta$  instead of  $\delta / 2$  for convenience). By Lemma 3.1, taking  $m \geq O\left((d / \varepsilon_1^2)O(\ell^2)\log (d / \delta)\right)$  ensures that  $\| \mathbf{E}_B\| _2 \leq \varepsilon_1\lambda_{\max}$  and  $\| \mathbf{E}_D\| _2 \leq \varepsilon_1\lambda_{\max}$  w.p. at least  $1 - \delta$  where  $\mathbf{E}_B = \hat{\pmb{\Sigma}}_B - \pmb{\Sigma}_B$  and  $\mathbf{E}_D = \hat{\pmb{\Sigma}}_D - \pmb{\Sigma}_D$ . We start by defining  $\rho(\hat{\mathbf{r}}) := \frac{\mathbf{r}^\mathrm{T}\hat{\pmb{\Sigma}}_D\mathbf{r}}{\mathbf{r}^\mathrm{T}\hat{\pmb{\Sigma}}_B\mathbf{r}}$  which is the equivalent of  $\rho$  using the estimated matrices. Observe that it can be written as  $\hat{\rho}(\mathbf{r}) = \frac{\mathbf{r}^\mathrm{T}\pmb{\Sigma}_B\mathbf{r} + \mathbf{r}^\mathrm{T}\pmb{E}_B\mathbf{r}}{\mathbf{r}^\mathrm{T}\pmb{\Sigma}_D\mathbf{r} + \mathbf{r}^\mathrm{T}\pmb{E}_D\mathbf{r}}$ . Using these we can obtain the following bound on  $\hat{\rho}$ : for any  $\mathbf{r} \in \mathbb{R}^d$ ,  $|\hat{\rho}(\mathbf{r}) - \rho(\mathbf{r})| \leq \theta\varepsilon_1|\rho(\mathbf{r})|$  w.p. at least  $1 - \delta (*)$  as long as  $\varepsilon_1 \leq \frac{(1 - \max(0,\kappa_1))}{2}\frac{\lambda_{\min}}{\lambda_{\max}}$ , which we shall ensure (see Appendix B.4).
For convenience we denote the normalized projection of any vector  $\mathbf{r}$  as  $\tilde{\mathbf{r}} := \frac{\pmb{\Sigma}^{1/2}\mathbf{r}}{\|\pmb{\Sigma}^{1/2}\mathbf{r}\|_2}$ . Now let  $\tilde{\mathbf{r}} \in \mathbb{R}^d$  be a unit vector such that  $\min\{\|\tilde{\mathbf{r}} - \tilde{\mathbf{r}}_*\|_2, \|\tilde{\mathbf{r}} + \tilde{\mathbf{r}}_*\|_2\} \geq \varepsilon_2$ . Hence, using the definitions from Lemma C.4,  $|\gamma(\mathbf{r})| \leq 1 - \varepsilon_2^2/2$  while  $\gamma(\mathbf{r}_*) = 1$  which implies  $\rho(\mathbf{r}_*) - \rho(\mathbf{r}) \geq \kappa_3\varepsilon_2^2/2$ . Note that  $\rho(\mathbf{r}) \leq \rho(\mathbf{r}_*) = 2 + \kappa_3(1 - \max(0, \kappa_1))$ . Choosing  $\varepsilon_1 < \frac{\kappa_3}{2\theta(2 + \kappa_3(1 - \max(0, \kappa_1))}\varepsilon_2^2$ , we obtain that  $\rho(\mathbf{r}_*)(1 - \theta\varepsilon_1) > \rho(\mathbf{r})(1 + \theta\varepsilon_1)$ . Using this along with the bound (*) we obtain that w.p. at least  $1 - \delta$ ,  $\hat{\rho}(\mathbf{r}_*) > \hat{\rho}(\mathbf{r})$  when  $\varepsilon_2 > 0$ . Since our algorithm returns  $\hat{\mathbf{r}}$  as the maximizer of  $\hat{\rho}$ , w.p. at least  $1 - \delta$  we get  $\min\{\|\tilde{\mathbf{r}} - \tilde{\mathbf{r}}_*\|_2, \|\tilde{\mathbf{r}} + \tilde{\mathbf{r}}_*\|_2\} \leq \varepsilon_2$ . Using Lemma 2.1,  $\min\{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\} \leq 4\sqrt{\frac{\lambda_{\mathrm{max}}}{\lambda_{\mathrm{min}}}}\varepsilon_2$ . Substituting  $\varepsilon_2 = \frac{\varepsilon}{4}\sqrt{\frac{\lambda_{\mathrm{min}}}{\lambda_{\mathrm{max}}}}$ ,  $\|\mathbf{r} - \mathbf{r}_*\|_2 \leq \varepsilon$  w.p. at least  $1 - \delta$ . The conditions on  $\varepsilon_1$  are satisfied by taking it to be  $\leq O\left(\frac{\kappa_3\varepsilon^2\lambda_{\mathrm{min}}}{\theta(2 + \kappa_3(1 - \max(0, \kappa_1))\lambda_{\mathrm{max}}}\right)$ , and thus we can take  $m \geq O\left((d/\varepsilon^4)O(\ell^2)\log(d/\delta)\left(\frac{\lambda_{\mathrm{max}}}{\lambda_{\mathrm{min}}}\right)^2\theta^2\left(\frac{2 + \kappa_3(1 - \max(0, \kappa_1)}{\kappa_3}\right)^2\right) = O\left((d/\varepsilon^4)O(\ell^2)\log(d/\delta)\left(\frac{\lambda_{\mathrm{max}}}{\lambda_{\mathrm{min}}}\right)^4q^4\right)$  since  $1/\kappa_3 \leq q^2(1 + \ell^2/4)/\ell^2$  whenever  $\ell > 1$  (Defn. C.3). This completes the proof of Lemma C.1.
D Generalization error Bounds
We show that if a hypothesis LTF  $h$  satisfies close to 1 fraction of sufficient number of bags sampled from a bag oracle then with high probability  $h$  is a good approximator for the target LTF  $f$  (or its
complement in the case of balanced bags). The first step is to prove a generalization bound from the sample bag-level accuracy to the oracle bag-level accuracy.
Theorem D.1. Let  $\mathcal{O} \coloneqq \mathsf{Ex}(f, \mathcal{D}, q, k)$  be any bag oracle for an LTF  $f$  in  $d$ -dimensions, and let  $\mathcal{M}$  be a collection of  $m$  bags sampled iid from the oracle. Then, there is an absolute constant  $C_0 \leq 1000$  s.t. w.p. at least  $1 - \delta$ ,
$$
\operatorname {B a g E r r} _ {\text {o r a c l e}} (h, f, \mathcal {D}, q, k) \leq \operatorname {B a g E r r} _ {\text {s a m p l e}} (h, \mathcal {M}) + \varepsilon \tag {8}
$$
when  $m \geq C_0 d\left(\log q + \log (1 / \delta)\right) / \varepsilon^2$ , for any  $\delta, \varepsilon > 0$ .
Proof. The proof follows from the arguments similar to the ones used in Appendix M of [26] to prove bag satisfaction generalization bounds. Consider a bag loss function of the form  $\ell (\phi^{\zeta}(h,B),(\phi^{\zeta}(f,B))$ , where  $\phi^{\zeta}(g,B)\coloneqq \zeta (\mathbf{y}_{g,B})$  where  $\mathbf{y}_{g,B}$  is the vector of  $(g(\mathbf{x}))_{\mathbf{x}\in B}$ , for  $\zeta :\{0,1\} ^q\to \mathbb{R}$ . The result of [35] showed generalization error bounds when (i)  $\zeta$  is 1-Lipschitz w.r.t. to  $\infty$ -norm, and (ii)  $\ell$  is 1-Lipschitz in the first coordinate. We can thus apply their results using the bound of  $(d + 1)$  on the VC-dimension of LTFs in  $d$ -dimensions to show that the above bound on  $m$  holds (with  $C_0 / 8$  instead of  $C_0$ ) for the generalization of the following bag error:
$$
\left| \gamma (B, f, t) - \gamma (B, h, t) \right|,
$$
$$
\gamma (B, g, t) := \left\{ \begin{array}{l l} 0 & \text {i f} \sum_ {\mathbf {x} \in B} g (\mathbf {x}) \leq t \\ 1 & \text {o t h e r w i s e .} \end{array} \right. \tag {9}
$$
for  $t \in \{0, \dots, q - 1\}$ . We can bound our bag satisfaction error by the sum of the generalization errors of  $|\gamma(B, f, k) - \gamma(B, h, k)|$ , and  $|\gamma(B, f, k - 1) - \gamma(B, h, k - 1)|$ , which can each be bounded by  $\varepsilon / 2$  thus completing the proof.
Next we show that if the oracle-level bag accuracy of  $h$  is high then this translates to  $h$  is being a low error instance-level approximator for the target LTF  $f$  (or its complement in the case of balanced bags). With the setup as used in the previous theorem, let us define the regions  $S_{a} := \{\mathbf{x} \text{ s.t } f(\mathbf{x}) = a\}$  for  $a \in \{0,1\}$ , and  $S_{ab} := \{\mathbf{x} \text{ s.t } f(\mathbf{x}) = a, h(\mathbf{x}) = b\}$ . Let  $\mu$  be the measure induced by  $\mathcal{D}$ ,  $\mu_{a}$  and  $\mu_{ab}$  be the respectively conditional measures induced on  $S_{a}$  and  $S_{ab}$ . The oracle  $\mathcal{O}$  for a random bag  $B$ , samples  $k$  points iid from  $(S_{1}, \mu_{1})$  and  $(q - k)$  points from  $(S_{0}, \mu_{0})$ .
we have the following theorem.
Theorem D.2. Suppose  $k \in \{1, \dots, q\}$ , and  $0 < \operatorname{BagAcc}_{\mathrm{oracle}}(h, f, \mathcal{D}, q, k) \leq \varepsilon' < 1/(4q)$ , then
(i)  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$  if  $k \neq q/2$ , and
(ii)  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$  or  $\operatorname{Pr}[f(\mathbf{x}) \neq (1 - h(\mathbf{x}))] \leq \varepsilon$ , if  $k = q/2$ ,
where  $\varepsilon = 4\varepsilon^{\prime}$
Before we prove the above them, we need the following lemma bounding the probability that two independent binomial random variables take the same value. Let  $\operatorname{Binomial}(n,p)$  be the sum of  $n$  iid  $\{0,1\}$  random variables each with expectation  $p$ .
Lemma D.3. Let  $u, v \in \{1, \ldots, q\}$ , and  $X_1 \sim \mathrm{Binomial}(u, p_1)$  and  $X_2 \sim \mathrm{Binomial}(v, p_2)$  be independent binomial distributions for some probabilities  $p_1$  and  $p_2$ . Let  $p^* = \min \{\max \{p_1, (1 - p_1)\}, \max \{p_2, (1 - p_2)\}\}$ , then,  $\operatorname{Pr}[X_1 \neq X_2] \geq 1 - \sqrt{p^*}$ .
Proof. We first begin by bounding a binomial coefficient by the sum of its adjacent binomial coefficients. Let  $n \geq 1$  and  $n > r > 0$ . We begin with the standard identity and proceed further:
$$
\begin{array}{l} \left( \begin{array}{c} n \\ r \end{array} \right) = \left( \begin{array}{c} n - 1 \\ r \end{array} \right) + \left( \begin{array}{c} n - 1 \\ r - 1 \end{array} \right) \\ \leq \frac {n}{r + 1} \binom {n - 1} {r} + \frac {n}{n - r + 1} \binom {n - 1} {r - 1} (10) \\ = \binom {n} {r + 1} + \binom {n} {r - 1} (11) \\ \end{array}
$$
Moreover, it is trivially true that  $\binom{n}{0} \leq \binom{n}{1}$  and  $\binom{n}{n} \leq \binom{n}{n-1}$ , therefore (11) holds even for  $r \in \{0, n\}$  whenever the binomial coefficients exist. Now, for some probability  $p$  define  $\nu(p, n, r) := \binom{n}{r} p^r (1 - p)^{1-r}$  which is pdf at  $r$  of Binomial  $(n, p)$ . The above implies the following:
$$
\nu (p, n, r) \leq p ^ {\prime} (\nu (p, n, r - 1) + \nu (p, n, r + 1)) \tag {12}
$$
where  $p' = \max \{p / (1 - p), (1 - p) / p\}$ . Using the fact that  $\nu(p, n, r - 1) + \nu(p, n, r) + \nu(p, n, r + 1) \leq 1$ , we obtain that
$$
\begin{array}{l} (1 / p ^ {\prime}) \nu (p, n, r) + \nu (p, n, r) \leq 1 \\ \Rightarrow \quad \nu (p, n, r) \leq \left(p ^ {\prime} / 1 + p ^ {\prime}\right) = \max  \{p, 1 - p \} \tag {13} \\ \end{array}
$$
The above allows us to complete the proof of the lemma as follows.
$$
\begin{array}{l} \Pr \left[ X _ {1} = X _ {2} \right] \\ = \sum_ {r = 0} ^ {\min  \{u, v \}} \nu (p _ {1}, u, r) \nu (p _ {2}, v, r) \\ \leq \left(\sum_ {r = 0} ^ {u} \nu (p _ {1}, u, r) ^ {2}\right) ^ {\frac {1}{2}} \left(\sum_ {r = 0} ^ {v} \nu (p _ {2}, v, r) ^ {2}\right) ^ {\frac {1}{2}} \\ \leq \left(\max  _ {r} \{\nu (p _ {1}, u, r) \} \sum_ {r = 0} ^ {u} \nu (p _ {1}, u, r)\right) ^ {\frac {1}{2}} \\ \cdot \left(\max  _ {r} \{\nu (p _ {2}, v, r) \} \sum_ {r = 0} ^ {u} \nu (p _ {2}, v, r)\right) ^ {\frac {1}{2}} \\ \leq \left(\max  \left\{p _ {1}, 1 - p _ {1} \right\}\right) ^ {\frac {1}{2}} \left(\max  \left\{p _ {2}, 1 - p _ {2} \right\}\right) ^ {\frac {1}{2}} \\ \leq \sqrt {p ^ {*}}, \tag {14} \\ \end{array}
$$
where we use Cauchy-Schwarz for the first inequality.
Proof. (of Theorem D.2) We now consider three cases, for each one we shall prove points (i) and (ii) of the theorem.
Case  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \geq 1 - \varepsilon$ . This condition means that  $\mu(S_{10}) + \mu(S_{01}) \geq 1 - \varepsilon$ , which implies that at least one of  $\mu_1(S_{10}), \mu_0(S_{01})$  is  $\geq 1 - \varepsilon$ . Assume that  $\mu_0(S_{01}) \geq 1 - \varepsilon$  (the other case is analogous).
Consider unbalanced bags i.e.,  $k \neq q/2$ . Now in case that  $\mu_1(S_{10}) \geq 1 - \varepsilon$ , all the points sampled from  $S_{1}$  are sampled from  $S_{10}$  w.p.  $(1 - k\varepsilon)$  and those sampled from  $S_{0}$  are all sampled from  $S_{01}$  w.p.  $(1 - (q - k)\varepsilon)$ . Therefore, with probability at least  $(1 - q\varepsilon)$  all the points are sampled from  $S_{10} \cup S_{01}$ . Since  $k \neq q/2$  this implies that  $h$  does not satisfy the random bags with probability at least  $(1 - q\varepsilon) > \varepsilon'$ . If  $\mu_1(S_{10}) \leq \varepsilon$ , we can show with similar arguments that with probability  $\geq (1 - q\varepsilon) > \varepsilon'$  no points are sampled from  $S_{10}$  and  $q - k$  points are sampled from  $S_{01}$  which means (since  $k \geq 1$ ) that  $h$  does not satisfy the bag. Finally, let  $\mu_1(S_{10}) \in (\varepsilon, 1 - \varepsilon)$ . In this case, the number of points sampled from  $S_{10}$  is distributed as  $\mathrm{Binomial}(k, \mu_1(S_{10}))$  and those sampled from  $S_{01}$  is independently distributed as  $\mathrm{Binomial}(k, \mu_0(S_{01}))$ . If these two numbers are different then  $h$  does not satisfy the bag. We can apply Lemma D.3 and using the bounds on  $\mu_1(S_{10})$ , we get that  $p^* \leq 1 - \varepsilon$ . Therefore, the probability of  $h$  not satisfying a randomly sampled bag is at least
$$
1 - \sqrt {1 - \varepsilon} \geq \frac {(1 - (1 - \varepsilon))}{1 + \sqrt {1 - \varepsilon}} \geq \varepsilon / 2 > \varepsilon^ {\prime}. \tag {15}
$$
For balanced bags, the case  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \geq 1 - \varepsilon$  implies that  $\operatorname{Pr}[f(\mathbf{x}) \neq (1 - h(\mathbf{x}))] \leq \varepsilon$ , so the condition (ii) of the theorem holds.
Case  $\varepsilon < \operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] < 1 - \varepsilon$ . This case violates the conditions (i) and (ii). First observe that,  $\mu_1(S_{10})$  and  $\mu_0(S_{01})$  both cannot be  $\geq (1 - \varepsilon)$  or  $\leq \varepsilon$ , otherwise  $\mu(S_{10}) + \mu(S_{01})$  is either  $\geq 1 - \varepsilon$  or  $\leq \varepsilon$  violating the assumption of this case. For the subcase that  $\mu_1(S_{10}) \leq \varepsilon$  and  $\mu_0(S_{01}) \geq (1 - \varepsilon)$ ,
we can show using arguments similar to the previous case that w.p.  $\geq (1 - q\varepsilon)$  no points are sampled from  $S_{10}$  and  $q - k$  points are sampled from  $S_{01}$ , and thus,  $h$  will not satisfy a random bag with probability at least  $(1 - q\varepsilon) > \varepsilon'$ . The subcase when  $\mu_1(S_{01}) \leq \varepsilon$  and  $\mu_0(S_{10}) \geq (1 - \varepsilon)$  is analogous. Finally, we have the subcase that  $\mu_1(S_{10})$  or  $\mu_0(S_{01})$  both lie in the range  $(\varepsilon, 1 - \varepsilon)$ . Now, the number of points sampled from  $S_{10}$  is distributed as  $\mathrm{Binomial}(k, \mu_1(S_{10}))$  and those sampled from  $S_{01}$  is independently distributed as  $\mathrm{Binomial}(k, \mu_0(S_{01}))$ . If these two numbers are different then  $h$  does not satisfy the bag. We can apply Lemma D.3 and using the bounds on one of  $\mu_1(S_{10})$  or  $\mu_0(S_{01})$ , we get that  $p^* \leq 1 - \varepsilon$ . Therefore, the probability of  $h$  not satisfying a randomly sampled bag is at least  $1 - \sqrt{1 - \varepsilon} \geq \varepsilon / 2 > \varepsilon'$ , using (15).
D.1 Proof of Theorem 2.2
The proof follows directly from Theorems D.1 and D.2.
D.2 Proof of Lemma 2.3
$$
\begin{array}{l} \Pr [ \operatorname {p o s} (\mathbf {r} ^ {\mathsf {T}} \mathbf {X} + c) \neq \operatorname {p o s} (\hat {\mathbf {r}} ^ {\mathsf {T}} \mathbf {X} + c) ] = \Pr [ \operatorname {p o s} (\mathbf {r} ^ {\mathsf {T}} \tilde {\mathbf {X}} + \| \boldsymbol {\Gamma} \mathbf {r} \| _ {2} \zeta) \neq \operatorname {p o s} (\hat {\mathbf {r}} ^ {\mathsf {T}} \tilde {\mathbf {X}} + \| \boldsymbol {\Gamma} \hat {\mathbf {r}} \| _ {2} \hat {\zeta}) ] \\ = \Pr [ \operatorname {p o s} (\mathbf {a} ^ {\mathsf {T}} \mathbf {Z} + \zeta) \neq \operatorname {p o s} (\hat {\mathbf {a}} ^ {\mathsf {T}} \mathbf {Z} + \hat {\zeta}) ] \tag {16} \\ \end{array}
$$
where  $\boldsymbol{\Gamma} = \boldsymbol{\Sigma}^{1/2}$ ,  $\mathbf{a} = \boldsymbol{\Gamma}\mathbf{r} / \|\boldsymbol{\Gamma}\mathbf{r}\|_2$  and  $\hat{\mathbf{a}} = \boldsymbol{\Gamma}\hat{\mathbf{r}} / \|\boldsymbol{\Gamma}\hat{\mathbf{r}}\|_2$  and  $\mathbf{Z} = \boldsymbol{\Gamma}^{-1}\tilde{\mathbf{X}} \sim N(\mathbf{0},\mathbf{I})$  and  $\tilde{\mathbf{X}} + \boldsymbol{\mu} = \mathbf{X} \sim N(\boldsymbol{\mu},\boldsymbol{\Sigma})$ ,  $\zeta = (c + \mathbf{r}^{\top}\boldsymbol{\mu}) / \|\boldsymbol{\Gamma}\mathbf{r}\|_2$  and  $\hat{\zeta} = (c + \hat{\mathbf{r}}^{\top}\boldsymbol{\mu}) / \|\boldsymbol{\Gamma}\hat{\mathbf{r}}\|_2$ . By Lemma 2.1,  $\|\mathbf{a} - \hat{\mathbf{a}}\|_2 \leq 4\sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}} \varepsilon$ .
Now, the RHS of (16) can be bounded as,
$$
\Pr [ \operatorname {p o s} (\mathbf {a} ^ {\mathsf {T}} \mathbf {Z} + \zeta) \neq \operatorname {p o s} (\mathbf {a} ^ {\mathsf {T}} \mathbf {Z} + \hat {\zeta}) ] + \Pr [ \operatorname {p o s} (\mathbf {a} ^ {\mathsf {T}} \mathbf {Z} + \hat {\zeta}) \neq \operatorname {p o s} (\hat {\mathbf {a}} ^ {\mathsf {T}} \mathbf {Z} + \hat {\zeta}) ] \tag {17}
$$
Now,  $g \coloneqq \mathbf{a}^{\top}\mathbf{Z} \sim N(0,1)$ . Thus, the first term in the above is bounded by the probability that  $g$  lies in a range of length  $|\zeta -\hat{\zeta}| = \| \pmb {\mu}\| _2\| \mathbf{r} - \hat{\mathbf{r}}\| _2 / \| \pmb {\Gamma}\mathbf{r}\| _2 \leq \varepsilon \| \pmb {\mu}\| _2 / \sqrt{\lambda_{\min}}$ . This probability is at most  $\varepsilon \| \pmb {\mu}\| _2 / \sqrt{2\pi\lambda_{\min}}$ .
For the second term, that is at most the probability that  $\mathrm{pos}(\mathbf{a}^{\top}\mathbf{Z})\neq \mathrm{pos}(\hat{\mathbf{a}}^{\top}\mathbf{Z})$ . Now  $\| \hat{\mathbf{a}} -\mathbf{a}\| _2\leq 4\sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}}\varepsilon \Rightarrow \angle \hat{\mathbf{a}},\mathbf{a}\leq \pi 4\sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}}\varepsilon \Rightarrow \operatorname *{Pr}[\mathrm{pos}(\mathbf{a}^{\top}\mathbf{Z})\neq \mathrm{pos}(\hat{\mathbf{a}}^{\top}\mathbf{Z})]\leq 2\sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}}\varepsilon$ . Hence,
$$
\Pr [ \operatorname {p o s} (\mathbf {r} ^ {\mathsf {T}} \mathbf {X} + c) \neq \operatorname {p o s} (\hat {\mathbf {r}} ^ {\mathsf {T}} \mathbf {X} + c) ] \leq \varepsilon \left(2 \sqrt {\frac {\lambda_ {\max }}{\lambda_ {\min }}} + \frac {\| \boldsymbol {\mu} \| _ {2}}{\sqrt {2 \pi \lambda_ {\min }}}\right)
$$
E Subgaussian concentration with thresholded Gaussian random variables
Let  $\Phi(.)$  be the standard Gaussian cdf i.e.,  $\Phi(t) \coloneqq \operatorname{Pr}_{X \sim N(0,1)}[X \geq t]$ . We also define  $\overline{\Phi}(t) \coloneqq \operatorname{Pr}[X > t] = 1 - \Phi(t)$ . We begin by defining the subgaussian norm of a random variable.
Definition E.1. The subgaussian norm of a random variable  $X$  denoted by  $\| X\|_{\psi_2}$  and is defined as:  $\| X\|_{\psi_2} \coloneqq \inf \{t > 0 : \mathbb{E}\left[\exp \left(X^2 / t^2\right)\right] \leq 2\}$ . Further, there is an absolute constant  $K_0$  such that  $\| X\|_{\psi_2} \leq K_0K$  if  $X$  satisfies,
$$
\Pr \left[ | X | \geq t \right] \leq 2 \exp \left(- t ^ {2} / K ^ {2}\right), \quad \text {f o r a l l} t \geq 0. \tag {18}
$$
Let  $X \sim N(0,1)$ . It is easy to see that  $\mathbb{E}\left[\exp (X^2 /2^2)\right] = (1 / \sqrt{2\pi})\int_{-\infty}^{\infty}\exp (-x^2 /4)dx = \sqrt{2} (1 / \sqrt{2\pi})\int_{-\infty}^{\infty}\exp (-z^2 /2)dz = \sqrt{2}$ . Thus,  $X \sim N(0,1)$  is subgaussian with subgaussian norm  $\leq 2$ . In the analysis of this section, we shall use the following proposition from [32].
Proposition E.2 (Prop. 2.1.2 of [32]). Let  $X \sim N(0,1)$ . Then, for any  $t > 0$ ,
$$
\left(\frac {1}{t} - \frac {1}{t ^ {3}}\right) \frac {1}{\sqrt {2 \pi}} \exp (- t ^ {2} / 2) \leq \overline {{\Phi}} (t) \leq \frac {1}{t \sqrt {2 \pi}} \exp (- t ^ {2} / 2).
$$
In particular for  $t \geq 1$ ,
$$
\overline {{\Phi}} (t) \leq \frac {1}{\sqrt {2 \pi}} \mathrm {e x p} \left(- t ^ {2} / 2\right).
$$
Using the above, and by symmetry it is easy to see that for  $t \geq 1$ ,  $\operatorname*{Pr}\left[|X| > t\right] \leq \left(\sqrt{2 / \pi}\right)\exp \left(-t^2 /2\right) \leq 2 \cdot \exp \left(-t^2 /2\right)$ . On the other hand,  $2 \cdot \exp \left(-t^{2} / 2\right) \geq 1$  for  $0 \leq t < 1$ . Thus,
$$
\Pr \left[ | X | > t \right] \leq 2 \cdot \exp \left(- t ^ {2} / 2\right), \quad \forall t \geq 0. \tag {19}
$$
Consider the normal distribution conditioned on a threshold defined by letting  $\mathcal{D}_{\ell}$  be the distribution of  $\{X\sim N(0,1)\mid X > \ell \}$ . We shall show that  $\tilde{X}\sim \mathcal{D}_{\ell}$  is a subgaussian random variable. Let us handle the (relatively) easy case of  $\ell \leq 0$  first.
Lemma E.3. Let  $\tilde{X} \sim \mathcal{D}_{\ell}$  for some  $\ell \leq 0$ . Then,  $\operatorname*{Pr}\left[\left|\tilde{X}\right| > t\right] \leq 2 \cdot \exp\left(-t^{2}/2\right)$ , for any  $t > 0$ .
Proof. Let  $E$  be the event that  $\tilde{X} \geq 0$ . Conditioned on  $E$ ,  $\tilde{X}$  is distributed as  $|X|$  where  $X \sim N(0,1)$  and (19) implies that
$$
\Pr \left[ | \tilde {X} | > t \mid E \right] \leq 2 \cdot \exp (- t ^ {2} / 2), \quad \forall t > 0.
$$
On the other hand, conditioned on  $\overline{E}$ ,  $\tilde{X}$  is sampled as  $-|Z|$  where  $\{Z\sim N(0,1)\mid |Z|\leq -\ell = |\ell |\}$ . Thus,
$$
\Pr \left[ | \tilde {X} | > t \mid \bar {E} \right] = \underset {Z \sim N (0, 1)} {\operatorname * {P r}} [ | Z | > t \mid | Z | \leq | \ell | ]. \tag {20}
$$
Now, if  $t > |\ell|$  then  $\operatorname{Pr}\left[\left|\tilde{X}\right| > t\mid \overline{E}\right] = 0$ , otherwise if  $t\leq \ell$  the LHS of (20) is,
$$
\begin{array}{l} \operatorname * {P r} _ {Z \sim N (0, 1)} \left[ | Z | > t \mid | Z | \leq | \ell | \right] = 1 - \operatorname * {P r} \left[ | Z | \leq t \mid | Z | \leq | \ell | \right] = 1 - \frac {\operatorname* {P r} \left[ | Z | \leq t \right]}{\operatorname* {P r} \left[ | Z | \leq \ell \right]} \\ \leq 1 - \Pr \left[ | Z | \leq t \right] = \Pr \left[ | Z | > t \right] \\ \end{array}
$$
which is bounded by  $2 \cdot \exp\left(-t^{2} / 2\right)$  and combining the probability bounds conditioned on  $E$  and  $\overline{E}$  we complete the proof.
The case of  $\ell > 0$  is proved below.
Lemma E.4. Let  $\tilde{X} \sim \mathcal{D}_{\ell}$  for some  $\ell > 0$ . Then,  $\operatorname*{Pr}\left[\left|\tilde{X}\right| > t\right] \leq 2 \cdot \exp \left(-t^{2} / K_{1}^{2}\right)$ , for any  $t > 0$  where  $K_{1} = \max \{\sqrt{20}, |\ell| \sqrt{10}\}$ .
Proof. Let us first explicitly define the pdf of  $\hat{X}$  as:
$$
f _ {\mathcal {D} _ {\ell}} (x) = \left\{ \begin{array}{l l} 0 & \text {i f} x \leq \ell \\ f _ {N (0, 1)} (x) / \overline {{\Phi}} (\ell) & \text {o t h e r w i s e ,} \end{array} \right. \tag {21}
$$
where  $f_{N(0,1)}$  is the pdf of  $N(0,1)$ . We prove this in two cases.
Case  $\ell \leq 2$ . In this case, one can use the lower bound from Prop. E.2 to show that  $\overline{\Phi}(\ell) \geq \overline{\Phi}(2) > 1/50$  by explicit calculation. Thus,  $f_{\mathcal{D}_{\ell}}(x) \leq 50 f_{N(0,1)}(x)$  for  $x > 0$ . We can now obtain the desired bound as follows. Letting  $X \sim N(0,1)$ , for any  $t > 0$ ,
$$
\begin{array}{l} \Pr \left[ | \tilde {X} | > t \right] = \Pr \left[ \tilde {X} > t \right] \\ \leq 5 0 \Pr [ X > t ] \leq 5 0 \Pr [ | X | > t ] \leq 1 0 0 \exp (- t ^ {2} / 2) \tag {22} \\ \end{array}
$$
using (19). Now, it is easy to check that for  $t \geq 3$ ,
$$
\begin{array}{l} \frac {\exp \left(- \frac {t ^ {2}}{2 0}\right)}{\exp \left(- \frac {t ^ {2}}{2}\right)} \geq \exp \left(9 \cdot \left(\frac {1}{2} - \frac {1}{2 0}\right)\right) \geq \exp (4) > 5 0 \\ \Rightarrow 2 \cdot \exp \left(- \frac {t ^ {2}}{2 0}\right) > 1 0 0 \cdot \exp \left(- \frac {t ^ {2}}{2}\right) \tag {23} \\ \end{array}
$$
On the other hand, for  $0 \leq t < 3$ ,  $2\exp(-t^2/20) > 1$ . Thus,
$$
\Pr \left[ | \tilde {X} | > t \right] \leq 2 \cdot \exp (- t ^ {2} / 2 0)
$$
Case  $\ell > 2$ . In this case using the easily verifiable facts that hold for  $\ell > 2$ :
-  $(1 / \ell - 1 / \ell^3) > 1 / (2\ell)$  and
-  $\exp \left(-3\ell^{2} / 2\right) \leq \frac{1}{2\ell\sqrt{2\pi}}$
Prop. E.2 yields
$$
\bar {\Phi} (\ell) \geq \left(\frac {1}{\ell} - \frac {1}{\ell^ {3}}\right) \frac {1}{\sqrt {2 \pi}} \exp (- \ell^ {2} / 2) \geq \frac {1}{2 \ell \sqrt {2 \pi}} \exp (- \ell^ {2} / 2) \geq \exp (- 2 \ell^ {2}). \tag {24}
$$
Observe that  $\operatorname*{Pr}\left[\left|\tilde{X}\right| > t\right] = \operatorname*{Pr}_{Z\sim N(0,1)}[Z > t\mid Z\geq \ell ]$  . If  $t <   \ell$  , then this probability vanishes. Otherwise  $t\geq \ell >2$  , and from Prop. E.2,  $\operatorname *{Pr}[Z > t]\leq \exp (-t^2 /2)$  and therefore  $\operatorname *{Pr}\left[\left|\tilde{X}\right| > t\right]$  can be bounded by
$$
\leq \frac {\Pr [ Z > t ]}{\Pr [ Z \geq \ell ]} \leq 2 \overline {{\Phi}} (\ell) ^ {- 1} \exp (- t ^ {2} / 2) \leq 2 \exp (- t ^ {2} / 2 + 2 \ell^ {2}) \tag {25}
$$
using (24). Now, if  $t^2 = 5\ell^2 + \kappa$  for some  $\kappa \geq 0$ , then using  $\ell > 2$  we have
$$
- \frac {t ^ {2}}{2} + 2 \ell^ {2} = - \frac {\ell^ {2} + \kappa}{2} \leq - 1 - \frac {\kappa}{5 \ell^ {2}} = - \frac {5 \ell^ {2} + \kappa}{5 \ell^ {2}} \leq - \frac {t ^ {2}}{5 \ell^ {2}} \leq - \frac {t ^ {2}}{1 0 \ell^ {2}}. \tag {26}
$$