SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \tilde {\mathbf {r}} - \tilde {\mathbf {r}} _ {*} \| _ {2} \geq \frac {\| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} - \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} _ {*} \| _ {2}}{\| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \| _ {2}} \\ \geq \frac {\lambda_ {\operatorname* {m i n}}}{\lambda_ {\operatorname* {m a x}}} \sqrt {\frac {2 (1 - \operatorname* {m a x} (0 , \kappa_ {1} (q , k , \ell)))}{1 - \operatorname* {m i n} (0 , \kappa_ {1} (q , k , \ell))}} \\ \end{array}
$$
$$
\gamma (\mathbf {r}) = \tilde {\mathbf {r}} ^ {T} \tilde {\mathbf {r}} _ {*} \leq 1 - \left(\frac {\lambda_ {\min}}{\lambda_ {\max}}\right) ^ {2} \frac {1 - \max (0 , \kappa_ {1} (q , k , \ell))}{1 - \min (0 , \kappa_ {1} (q , k , \ell))}
$$
If  $\ell = 0$ , then  $\kappa_{1}(q,k,0) = \kappa_{1}(q,k)\geq 0$ . Hence,
$$
\gamma (\mathbf {r}) \leq 1 - \left(\frac {\lambda_ {\operatorname* {m i n}}}{\lambda_ {\operatorname* {m a x}}}\right) ^ {2} (1 - \kappa_ {1} (q, k))
$$
B.4 Bounding error in  $\hat{\rho}$
Lemma B.4. If  $\| E_B\| _2\leq \varepsilon \lambda_{\mathrm{max}}$  and  $\| E_D\| _2\leq \varepsilon \lambda_{\mathrm{max}}$  where  $E_{B} = \hat{\Sigma}_{B} - \hat{\Sigma}_{B}$  and  $E_{D} = \hat{\Sigma}_{D} - \hat{\Sigma}_{D}$  then,
$$
\left| \left(\rho (\mathbf {r})\right) - \rho (\mathbf {r}) \right| \leq \left| \rho (\mathbf {r}) \right| \theta \varepsilon
$$
for  $\theta$  as defined in Defn. 4.3 or Defn. C.3.
Proof. Let  $a(\mathbf{r}) \coloneqq \mathbf{r}^{\top}\pmb{\Sigma}_{D}\mathbf{r}, b(\mathbf{r}) \coloneqq \mathbf{r}^{\top}\pmb{\Sigma}_{B}\mathbf{r}, e(\mathbf{r}) \coloneqq \mathbf{r}^{\top}\pmb{E}_{D}\mathbf{r}, f(\mathbf{r}) \coloneqq \mathbf{r}^{\top}\pmb{E}_{B}\mathbf{r}$ . Using the bounds on the spectral norms of  $E_{B}$  and  $E_{D}$ , we get that  $|e(\mathbf{r})| \leq \varepsilon \lambda_{\max}$  and  $|f(\mathbf{r})| \leq \varepsilon \lambda_{\max}$ . Also, using variances in Lemma C.4,  $a(\mathbf{r}) \geq \lambda_{\min}(2 - \max(0, 2\kappa_{1} - \kappa_{2})) \geq 0$  and  $b(\mathbf{r}) \geq \lambda_{\min}(1 - \max(0, \kappa_{1})) \geq 0$ . Hence, we can conclude that
$$
\left| \frac {e (\mathbf {r})}{a (\mathbf {r})} \right| \leq \varepsilon \frac {\lambda_ {\max }}{\lambda_ {\min }} \left(\frac {1}{2 - \max  \left(0 , 2 \kappa_ {1} - \kappa_ {2}\right)}\right) \text {a n d} \left| \frac {f (\mathbf {r})}{b (\mathbf {r})} \right| \leq \varepsilon \frac {\lambda_ {\max }}{\lambda_ {\min }} \left(\frac {1}{1 - \max  \left(0 , \kappa_ {1}\right)}\right)
$$
Observe that  $\hat{\rho} (\mathbf{r}) / \rho (\mathbf{r}) = \frac{1 + e(\mathbf{r}) / a(\mathbf{r})}{1 + b(\mathbf{r}) / f(\mathbf{r})}$ . Hence,
$$
\frac {1 - \varepsilon \frac {\lambda_ {\max}}{\lambda_ {\min}} \left(\frac {1}{2 - \max \left(0 , 2 \kappa_ {1} - \kappa_ {2}\right)}\right)}{1 + \varepsilon \frac {\lambda_ {\max}}{\lambda_ {\min}} \left(\frac {1}{1 - \max \left(0 , \kappa_ {1}\right)}\right)} \leq \frac {\hat {\rho} (\mathbf {r})}{\rho (\mathbf {r})} \leq \frac {1 + \varepsilon \frac {\lambda_ {\max}}{\lambda_ {\min}} \left(\frac {1}{2 - \max \left(0 , 2 \kappa_ {1} - \kappa_ {2}\right)}\right)}{1 - \varepsilon \frac {\lambda_ {\max}}{\lambda_ {\min}} \left(\frac {1}{1 - \max \left(0 , \kappa_ {1}\right)}\right)}
$$
Now, whenever  $\varepsilon \leq \frac{1 - \max(0,\kappa_1)}{2}\frac{\lambda_{\min}}{\lambda_{\max}}, 1 - \varepsilon \frac{\lambda_{\max}}{\lambda_{\min}}\left(\frac{1}{1 - \max(0,\kappa_1)}\right) \geq 1/2$  and,
$1 + \varepsilon \frac{\lambda_{\mathrm{max}}}{\lambda_{\mathrm{min}}}\left(\frac{1}{1 - \max(0,\kappa_1)}\right)\leq 3 / 2.$  Thus, we obtain that
$$
\left| \frac {\hat {\rho} (\mathbf {r}) - \rho (\mathbf {r})}{\rho (\mathbf {r})} \right| \leq \theta \varepsilon
$$
where  $\theta$  is as defined in Defn. C.3. If we substitute  $\ell = 0$ , we get that  $\kappa_{1} \geq 0$  and we get  $\theta$  as defined in Defn. 4.3.
B.5 Ratio maximisation as a PCA problem
Theorem B.5. If  $\mathbf{A}$  and  $\mathbf{B}$  are positive definite matrices, then for all  $\mathbf{r} \in \mathbb{R}^d$
1.  $\frac{\mathbf{r}^{\mathsf{T}}\mathbf{Ar}}{\mathbf{r}^{\mathsf{T}}\mathbf{Br}} = \tilde{\mathbf{r}}^{\mathsf{T}}\mathbf{B}^{-1 / 2}\mathbf{AB}^{-1 / 2}\tilde{\mathbf{r}},\tilde{\mathbf{r}} = \frac{\mathbf{B}^{1 / 2}\mathbf{r}}{\|\mathbf{B}^{1 / 2}\mathbf{r}\|_{2}}$
2.  $\underset {\| \mathbf{r}\| _2 = 1}{\mathrm{argmax}}\frac{\mathbf{r}^\top\mathbf{A}\mathbf{r}}{\mathbf{r}^\top\mathbf{B}\mathbf{r}} = \frac{\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}^*}{\|\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}^*\|_2}$  where  $\tilde{\mathbf{r}}^{*} = \operatorname *{argmax}_{\| \tilde{\mathbf{r}}\|_{2} = 1}\tilde{\mathbf{r}}^{\top}\mathbf{B}^{-1 / 2}\mathbf{A}\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}$
Proof. The first statement comes from the substitution. Now,  $\frac{\mathbf{r}^{\mathrm{T}}\mathbf{A}\mathbf{r}}{\mathbf{r}^{\mathrm{T}}\mathbf{B}\mathbf{r}}$  is homogeneous in  $\mathbf{r}$ . Let  $a\simeq b\Rightarrow a = kb,k\in \mathbb{R}$
$$
\begin{array}{l} \underset {\| \mathbf {r} \| _ {2} = 1} {\operatorname {a r g m a x}} \frac {\mathbf {r} ^ {\top} \mathbf {A} \mathbf {r}}{\mathbf {r} ^ {\top} \mathbf {B} \mathbf {r}} \simeq \underset {\mathbf {r}} {\operatorname {a r g m a x}} \frac {\mathbf {r} ^ {\top} \mathbf {A} \mathbf {r}}{\mathbf {r} ^ {\top} \mathbf {B} \mathbf {r}} \\ \simeq \operatorname * {a r g m a x} _ {\mathbf {B} ^ {- 1 / 2} \tilde {\mathbf {r}}} \frac {\tilde {\mathbf {r}} ^ {\mathsf {T}} \mathbf {B} ^ {- 1 / 2} \mathbf {A} \mathbf {B} ^ {- 1 / 2} \tilde {\mathbf {r}}}{\| \tilde {\mathbf {r}} \| _ {2} ^ {2}} \\ \simeq \mathbf {B} ^ {- 1 / 2} _ {\tilde {\mathbf {r}}} \frac {\tilde {\mathbf {r}} ^ {\mathsf {T}} \mathbf {B} ^ {- 1 / 2} \mathbf {A} \mathbf {B} ^ {- 1 / 2} \tilde {\mathbf {r}}}{\| \tilde {\mathbf {r}} \| _ {2} ^ {2}} \\ \simeq \mathbf{B}^{-1 / 2}\operatorname *{argmax}_{\| \tilde{\mathbf{r}}\|_{2} = 1}\tilde{\mathbf{r}}^{\top}\mathbf{B}^{-1 / 2}\mathbf{A}\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}} \\ \end{array}
$$
Hence,  $\underset {\| \mathbf{r}\| _2 = 1}{\mathrm{argmax}}\frac{\mathbf{r}^\top\mathbf{Ar}}{\mathbf{r}^\top\mathbf{Br}} = \frac{\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}^*}{\|\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}^*\|_2}$  where  $\tilde{\mathbf{r}}^{*} = \operatorname *{argmax}_{\| \tilde{\mathbf{r}}\|_{2} = 1}\tilde{\mathbf{r}}^{\top}\mathbf{B}^{-1 / 2}\mathbf{A}\mathbf{B}^{-1 / 2}\tilde{\mathbf{r}}$
B.6 Proof of Lemma 2.1
Proof. Let  $\mathbf{r}_1' = \mathbf{A}\mathbf{r}_1$  and  $\mathbf{r}_2' = \mathbf{B}\mathbf{r}_2$ . As  $\mathbf{r}_1$  and  $\mathbf{r}_2$  are unit vectors,  $\frac{1}{\|\mathbf{A}^{-1}\|_2} \leq \|\mathbf{r}_1'\|_2 \leq \|\mathbf{A}\|_2$ .
$$
\begin{array}{l} \left\| \mathbf {r} _ {1} ^ {\prime} - \mathbf {r} _ {2} ^ {\prime} \right\| _ {2} = \left\| \mathbf {A r} _ {1} - \mathbf {A r} _ {2} + \mathbf {A r} _ {2} - \mathbf {B r} _ {2} \right\| _ {2} \\ \leq \| \mathbf {A} \| _ {2} \| \mathbf {r} _ {1} - \mathbf {r} _ {2} \| _ {2} + \| \mathbf {A} - \mathbf {B} \| _ {2} \| \mathbf {r} _ {2} \| _ {2} \\ \leq \left\| \mathbf {A} \right\| _ {2} \left(\varepsilon_ {2} + \varepsilon_ {1}\right) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \frac {\mathbf {r} _ {1} ^ {\prime}}{\| \mathbf {r} _ {1} ^ {\prime} \| _ {2}} - \frac {\mathbf {r} _ {2} ^ {\prime}}{\| \mathbf {r} _ {2} ^ {\prime} \| _ {2}} \right\| _ {2} = \frac {\| \| \mathbf {r} _ {2} ^ {\prime} \| _ {2} \mathbf {r} _ {1} ^ {\prime} - \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \mathbf {r} _ {2} ^ {\prime} \| _ {2}}{\| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \| \mathbf {r} _ {2} ^ {\prime} \| _ {2}} = \frac {\| \| \mathbf {r} _ {2} ^ {\prime} \| _ {2} \mathbf {r} _ {1} ^ {\prime} - \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \mathbf {r} _ {1} ^ {\prime} + \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \mathbf {r} _ {1} ^ {\prime} - \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \mathbf {r} _ {2} ^ {\prime} \| _ {2}}{\| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \| \mathbf {r} _ {2} ^ {\prime *} \| _ {2}} \\ \leq \frac {\left| \| \mathbf {r} _ {2} ^ {\prime} \| _ {2} - \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \| \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} + \| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \| \mathbf {r} _ {1} ^ {\prime} - \mathbf {r} _ {2} ^ {\prime} \| _ {2} \right.}{\| \mathbf {r} _ {1} ^ {\prime} \| _ {2} \| \mathbf {r} _ {2} ^ {\prime} \| _ {2}} \leq \frac {2 (\| \mathbf {A} \| _ {2} \left(\varepsilon_ {2} + \varepsilon_ {1}\right))}{\frac {1}{\| \mathbf {A} ^ {- 1} \| _ {2}} - \| \mathbf {A} \| _ {2} \left(\varepsilon_ {2} + \varepsilon_ {1}\right)} \\ \leq \frac {2 \frac {\lambda_ {\max } (A)}{\lambda_ {\min } (A)} (\varepsilon_ {2} + \varepsilon_ {1}))}{1 - \frac {\lambda_ {\max } (A)}{\lambda_ {\min } (A)} (\varepsilon_ {2} + \varepsilon_ {1})} \leq 4 \frac {\lambda_ {\max } (A)}{\lambda_ {\min } (A)} (\varepsilon_ {2} + \varepsilon_ {1}) \text {i f} \frac {\lambda_ {\max } (A)}{\lambda_ {\min } (A)} (\varepsilon_ {2} + \varepsilon_ {1}) \leq \frac {1}{2} \\ \end{array}
$$
C Proof of Theorem 1.5
For the setting of Theorem 1.5, we provide Algorithm 4. It uses as a subroutine a polynomial time procedure PrincipalEigenVector for the principal eigen-vector of a symmetric matrix.
For notation, let  $\Gamma \coloneqq \Sigma^{1/2}$ , then by Lemma E.13, we can write our linear threshold function  $\mathrm{pos}(\mathbf{r}_*^\top \mathbf{X} + c_*) = \mathrm{pos}(\mathbf{u}_*^\top \mathbf{Z} - \ell)$  where  $\mathbf{Z} \sim N(\mathbf{0}, \mathbf{I})$  where  $\ell \coloneqq -\frac{c_* + \mathbf{r}_*^\top \boldsymbol{\mu}}{\|\mathbf{r}_{*}\|_2}$ ,  $\mathbf{u}_* \coloneqq \mathbf{f}\mathbf{r}_* / \| \mathbf{f}\mathbf{r}_*\|_2$ . For any  $\mathbf{r} \in \mathbb{R}^d$ , define  $\mathbf{u} \coloneqq \mathbf{f}\mathbf{r} / \| \mathbf{f}\mathbf{r}\|_2$ . Let  $\phi$  and  $\Phi$  be the standard gaussian pdf and cdf respectively.
The following geometric bound is obtained by the algorithm.
Lemma C.1. For any  $\varepsilon, \delta \in (0,1)$ , if  $m \geq O\left((d / \varepsilon^4)O(\ell^2)\log (d / \delta)\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^4 q^4\right)$ , then  $\hat{\mathbf{r}}$  computed in Step 3 of Alg. 4 satisfies  $\min \{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\} \leq \varepsilon$ , w.p.  $1 - \delta / 2$ .
Algorithm 4 PAC Learner for LTFs in over  $N(\mu, \Sigma)$
Input:  $\mathcal{O} = \mathsf{Ex}(f,\mathcal{D} = N(\pmb {\mu},\pmb {\Sigma}),q,k),m,s$  where  $f(\mathbf{x}) = \mathrm{pos}\left(\mathbf{r}_*\mathsf{T}\mathbf{x} + c_*\right)$ $\| \mathbf{r}_{*}\|_{2} = 1$
1. Compute  $\hat{\Sigma}_B, \hat{\Sigma}_D$  using MeanCovsEstimator with  $m$  samples.
2.  $\bar{\mathbf{r}} = \hat{\Sigma}_B^{-1 / 2}$  PrincipalEigenVector(  $\hat{\Sigma}_B^{-1 / 2}\hat{\Sigma}_D\hat{\Sigma}_B^{-1 / 2})$  if  $\hat{\Sigma}_B^{-1 / 2}$  exists, else exit.
3. Let  $\hat{\mathbf{r}} = \overline{\mathbf{r}} /||\overline{\mathbf{r}} ||_2$
4. If  $k = q / 2$
a. Sample a collection  $\mathcal{M}$  of  $s$  bags from  $\mathcal{O}$ .
b. For each bag  $B_{j}$  in  $\mathcal{M}$
i. Project each vector  $\mathbf{x}_j^{(i)}$  in  $B_{j}$  on  $\hat{\mathbf{r}}$ .
ii. Order these projections in a descending order  $\hat{\mathbf{r}}^{\top}\mathbf{x}_j^{(1)} > \ldots >\hat{\mathbf{r}}^{\top}\mathbf{x}_j^{(q)}$
ii. Define  $h_j = \mathrm{pos}(\hat{\mathbf{r}}^\mathsf{T}\mathbf{x} + \hat{\mathbf{r}}^\mathsf{T}\mathbf{x}_j^{(k)})$
c. Return  $h^* \in \{h_1, \dots, h_s\}$  which has lower  $\mathrm{BagErr}_{\mathrm{sample}}(h^*, \mathcal{M})$ .
5. else
i. Project each vector  $\mathbf{x}_j^{(i)}$  in  $B_{j}$  on  $\hat{\mathbf{r}}$ .
ii. Order these projections in a descending order  $\hat{\mathbf{r}}^{\top}\mathbf{x}_j^{(1)} > \dots >\hat{\mathbf{r}}^{\top}\mathbf{x}_j^{(q)}$
ii. Define  $h_j = \mathsf{pos}(\hat{\mathbf{r}}^\top \mathbf{x} + \hat{\mathbf{r}}^\top \mathbf{x}_j^{(k)})$
c. For each bag  $B_{j}$  in  $\mathcal{M}$
i. Project each vector  $\mathbf{x}_j^{(i)}$  in  $B_{j}$  on  $-\hat{\mathbf{r}}$
ii. Order these projections in a descending order  $-\hat{\mathbf{r}}^{\mathsf{T}}\mathbf{x}_{j}^{(1)} > \ldots > - \hat{\mathbf{r}}^{\mathsf{T}}\mathbf{x}_{j}^{(q)}$
ii. Define  $\tilde{h}_j = \mathrm{pos}(-\hat{\mathbf{r}}^\top \mathbf{x} - \hat{\mathbf{r}}^\top \mathbf{x}_j^{(k)})$
d. Return  $h^* \in \{h_1, \dots, h_s, \tilde{h}_1, \dots, \tilde{h}_s\}$  which has lower  $\text{BagErr}_{\text{sample}}(h^*, \mathcal{M})$ .
The above lemma, whose proof is deferred to Sec. C.1, is used in conjunction with the following lemma.
Lemma C.2. Let  $\varepsilon, \delta \in (0,1)$  and suppose that  $\hat{\mathbf{r}}$  computed in Step 3 of Alg. 4 satisfies if  $k \neq q/2$ ,  $\min \{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\} \leq \varepsilon$ . Then, with  $s \geq O\left(d(\log q + \log(1/\delta)) / \varepsilon^2\right)$ ,  $h^*$  computed in Step 4.c or Step 5.d satisfies
$$
\Pr_ {\mathcal {D}} \left[ h ^ {*} (\mathbf {x}) \neq f (\mathbf {x}) \right] \leq \frac {8 q \varepsilon}{\Phi (\ell) (1 - \Phi (\ell))} \left(c _ {0} \sqrt {\frac {\lambda_ {\operatorname* {m a x}}}{\lambda_ {\operatorname* {m i n}}}} + c _ {1} \frac {\| \boldsymbol {\mu} \| _ {2}}{\sqrt {\lambda_ {\operatorname* {m i n}}}}\right)
$$
and if  $k = q / 2$
$$
\min  _ {\mathscr {D}} \left[ h ^ {*} (\mathbf {x}) \neq f (\mathbf {x}) \right], \Pr_ {\mathscr {D}} \left[ (1 - h ^ {*} (\mathbf {x})) \neq f (\mathbf {x}) \right]) \leq \frac {8 q \varepsilon}{\Phi (\ell) (1 - \Phi (\ell))} \left(c _ {0} \sqrt {\frac {\lambda_ {\operatorname* {m a x}}}{\lambda_ {\operatorname* {m i n}}}} + c _ {1} \frac {\| \boldsymbol {\mu} \| _ {2}}{\sqrt {\lambda_ {\operatorname* {m i n}}}}\right)
$$
w.p.  $1 - \delta /2$  , where  $c_{0},c_{1}$  are the constants from Lemma 2.3.
With the above we complete the proof of Theorem 1.5 as follows.
Proof. (of Theorem 1.5) Let the parameters  $\delta, \varepsilon$  be as given in the statement of the theorem. We use  $O\left(\varepsilon \frac{(\Phi(\ell)(1 - \Phi(\ell)))\sqrt{\lambda_{\min}}}{q(\sqrt{\lambda_{\max}} + \| \boldsymbol{\mu}\|_2)}\right)$  for the error bound in Lemma C.1. We now take  $m$  to be of  $m = O\left((d / \varepsilon^4)\frac{O(\ell^2)}{(\Phi(\ell)(1 - \Phi(\ell)))^2}\log(d / \delta)\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^4\left(\frac{\sqrt{\lambda_{\max}} + \| \boldsymbol{\mu}\|_2}{\sqrt{\lambda_{\min}}}\right)^4q^8\right)$  in Alg. 4 we obtain the following bound:  $\min \left\{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\right\} \leq \varepsilon \frac{(\Phi(\ell)(1 - \Phi(\ell)))\sqrt{\lambda_{\min}}}{q(c_0\sqrt{\lambda_{\max}} + c_1\|\boldsymbol{\mu}\|_2)}$  with probability  $1 - \delta/2$ . Using  $s \geq O\left(d(\log(q) + \log(1/\delta))\frac{q^2(\sqrt{\lambda_{\max}} + \| \boldsymbol{\mu}\|_2)^2}{\varepsilon^2(\Phi(\ell)(1 - \Phi(\ell))^2\lambda_{\min}}\right)$ , Lemma C.2 yields the desired misclassification error bound of  $\varepsilon$  on  $h^*$  w.p.  $1 - \delta$ .
Proof. (of Lemma C.2) Define  $h(\mathbf{x}) \coloneqq \mathrm{pos}(\hat{\mathbf{r}}^{\mathsf{T}}\mathbf{X} + c_{*})$  and  $\tilde{h} (\mathbf{x}) \coloneqq \mathrm{pos}(-\hat{\mathbf{r}}^{\mathsf{T}}\mathbf{X} + c_{*})$ . Applying Lemma 2.3, we obtain that at least one of  $h$ ,  $\tilde{h}$  has an instance misclassification error of at most
$O(\varepsilon (\sqrt{\lambda_{\max} / \lambda_{\min}} +\| \pmb {\mu}\| _2 / \lambda_{\min}))$  . WLOG assume that  $h$  satisfies this error bound i.e.,  $\operatorname *{Pr}_{\mathcal{D}}[f(\mathbf{x})\neq h(\mathbf{x})]\leq \varepsilon (c_0\sqrt{\lambda_{\max} / \lambda_{\min}} +c_1\| \pmb {\mu}\| _2 / \sqrt{\lambda_{\min}}) = : \varepsilon '.$  Note that,  $\operatorname *{Pr}_{\mathcal{D}}[f(\mathbf{x}) = 1] = \Phi (\ell),\operatorname *{Pr}_{\mathcal{D}}[f(\mathbf{x}) = 0] = 1 - \Phi (\ell)$  . Thus,
$$
\operatorname * {P r} _ {\mathcal {D}} [ h (x) \neq f (x) \mid f (\mathbf {x}) = 1 ] \leq \varepsilon^ {\prime} / \Phi (\ell), \quad \operatorname * {P r} _ {\mathcal {D}} [ [ h (x) \neq f (x) \mid f (\mathbf {x}) = 0 ] \leq \varepsilon^ {\prime} / (1 - \Phi (\ell)).
$$
Therefore, taking a union bound we get that the probability that a random bag from the oracle contains a feature vector on which  $f$  and  $h$  disagree is at most  $q\varepsilon^{\prime} / \Phi (\ell)(1 - \Phi (\ell))$ . Applying Chernoff bound (see Appendix B.1) we obtain that with probability at least  $1 - \delta /6$ ,  $\mathsf{BagErr}_{\mathrm{sample}}(h,\mathcal{M})\leq 2q\varepsilon^{\prime} / \Phi (\ell)(1 - \Phi (\ell))$ . Therefore,  $h$  satisfies  $\mathsf{BagErr}_{\mathrm{sample}}(h^{*},\mathcal{M})\leq 2q\varepsilon^{\prime} / \Phi (\ell)(1 - \Phi (\ell))$ . Hence, there exists at least one  $h_j(\mathbf{x}) = \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x} + \hat{\mathbf{r}}^\top \mathbf{x}_j^{(k)})$  will satisfy  $\mathsf{BagErr}_{\mathrm{sample}}(h_j,\mathcal{M})\leq 2q\varepsilon^{\prime} / \Phi (\ell)(1 - \Phi (\ell))$ . Since,  $h^{*}(\mathbf{x})$  has the minimum sample bag error among all  $h_j(\mathbf{x})$ ,  $\mathsf{BagErr}_{\mathrm{sample}}(h^{*},\mathcal{M})\leq 2q\varepsilon^{\prime} / \Phi (\ell)(1 - \Phi (\ell))$ .
On the other hand, applying Theorem 2.2, except with probability  $\delta /3$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h^{*}(\mathbf{x})] \leq 8q\varepsilon' / \Phi(\ell)(1 - \Phi(\ell)) = \frac{8q\varepsilon}{\Phi(\ell)(1 - \Phi(\ell))}(c_{0}\sqrt{\lambda_{\max} / \lambda_{\min}} + c_{1}\|\pmb{\mu}\|_{2} / \sqrt{\lambda_{\min}})$  if  $k \neq q / 2$  and  $\min(\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h^{*}(\mathbf{x})], \operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq (1 - h^{*}(\mathbf{x}))]) \leq 8q\varepsilon' / \Phi(\ell)(1 - \Phi(\ell)) = \frac{8q\varepsilon}{\Phi(\ell)(1 - \Phi(\ell))}(c_{0}\sqrt{\lambda_{\max} / \lambda_{\min}} + c_{1}\|\pmb{\mu}\|_{2} / \sqrt{\lambda_{\min}})$  if  $k = q / 2$ . Therefore, except with probability  $\delta /2$ , the bound in Lemma C.2 holds.
We use these to generalize a few quantities we had defined earlier. We obtain their bounds using E.2,
Definition C.3. Define:
$$
\kappa_ {1} := \left(\frac {\phi (\ell) \left(\frac {k}{q} - (1 - \Phi (\ell))\right)}{\Phi (\ell) (1 - \Phi (\ell))}\right) ^ {2} - \frac {\ell \phi (\ell) \left(\frac {k}{q} - (1 - \Phi (\ell))\right)}{\Phi (\ell) (1 - \Phi (\ell))}, \quad \kappa_ {1} \geq - \ell^ {2} / 4
$$
$$
\kappa_ {2} := \frac {2}{q - 1} \frac {k}{q} \left(1 - \frac {k}{q}\right) \left(\frac {\phi (\ell)}{\Phi (\ell) (1 - \Phi (\ell))}\right) ^ {2}, \quad \kappa_ {2} \geq 2 \ell^ {2} / q ^ {2} \quad \text {w h e n} \ell > 1
$$
$$
\kappa_ {3} := \frac {\kappa_ {2}}{(1 - \kappa_ {1}) (1 - \max  (0 , \kappa_ {1}))}, \qquad \kappa_ {3} \geq \frac {2 \ell^ {2}}{q ^ {2} (1 + \ell^ {2} / 4)} \qquad w h e n \ell > 1
$$
$$
\theta := \frac {2 \lambda_ {\max}}{\lambda_ {\min}} \left(\frac {1}{2 - \max (0 , 2 \kappa_ {1} - \kappa_ {2})} + \frac {1}{1 - \max (0 , \kappa_ {1})}\right), \quad \frac {3 \lambda_ {\max}}{\lambda_ {\min}} \leq \theta \leq \frac {3 \lambda_ {\max}}{(1 - K _ {2}) \lambda_ {\min}}
$$
Where  $K_{i}$ 's are some finite functions of  $K$ .
Similar to Lemma 4.4, we again show in the following lemma that  $\hat{\mathbf{r}}$  in the algorithms is indeed  $\pm \mathbf{r}_{*}$  if the covariance estimates were the actual covariances.
Lemma C.4. The ratio  $\rho (\mathbf{r})\coloneqq \mathbf{r}^{\top}\pmb{\Sigma}_{D}\mathbf{r} / \mathbf{r}^{\top}\pmb{\Sigma}_{B}\mathbf{r}$  is maximized when  $\mathbf{r} = \pm \mathbf{r}_*$ . Moreover,
$$
\rho (\mathbf {r}) = 2 + \frac {\gamma (\mathbf {r}) ^ {2} \kappa_ {2}}{1 - \gamma (\mathbf {r}) ^ {2} \kappa_ {1}} \text {w h e r e} \gamma (\mathbf {r}) := \frac {\mathbf {r} ^ {\top} \boldsymbol {\Sigma} \mathbf {r} _ {*}}{\sqrt {\mathbf {r} ^ {\top} \boldsymbol {\Sigma} \mathbf {r}} \sqrt {\mathbf {r} _ {*} ^ {\top} \boldsymbol {\Sigma} \mathbf {r} _ {*}}} a n d
$$
Proof. Using the transformations to  $\mathbf{Z}$ , we can let  $\mathbf{X}_B = \boldsymbol{\Gamma}\mathbf{Z}_B$  as a random feature-vector sampled uniformly from a random bag sampled from  $\mathcal{O}$ . Also, let  $\mathbf{X}_D = \boldsymbol{\Gamma}\mathbf{Z}_D$  be the difference of two random feature vectors sampled uniformly without replacement from a random bag sampled from  $\mathcal{O}$ . Observe that the ratio  $\rho(\mathbf{r}) = \mathrm{Var}[\mathbf{r}^{\mathsf{T}}\mathbf{X}_D] / \mathrm{Var}[\mathbf{r}^{\mathsf{T}}\mathbf{X}_B] = \mathrm{Var}[\mathbf{u}^{\mathsf{T}}\mathbf{Z}_D] / \mathrm{Var}[\mathbf{u}^{\mathsf{T}}\mathbf{Z}_B]$ .
Let  $g^{*} \coloneqq \mathbf{u}_{*}^{\top}\mathbf{Z}$  which is  $N(0,1)$ . For  $a \in \{0,1\}$ , let  $\mathbf{Z}_a$  be  $\mathbf{Z}$  conditioned on  $\text{pos}(\mathbf{u}_*^\top \mathbf{Z} - \ell) = a$ . Let  $g_{a}^{*} \coloneqq \mathbf{u}_{*}^{\top}\mathbf{Z}_{a}$ ,  $a \in \{0,1\}$ .  $g_0^*$  lower-tailed one-sided truncated normal distributions truncated at  $\ell$  and  $g_{1}^{*}$  upper-tailed one-sided truncated normal distributions truncated at  $\ell$ . Hence,  $\mathbb{E}[g_1^*] = \phi(\ell) / (1 - \Phi(\ell))$ ,  $\mathbb{E}[g_0^*] = -\phi(\ell) / \Phi(\ell)$ ,  $\mathbb{E}[g_1^{*2}] = 1 + \ell \phi(\ell) / (1 - \Phi(\ell))$ ,  $\mathbb{E}[g_0^{*2}] = 1 - \ell \phi(\ell) / \Phi(\ell)$ .
With this setup, letting  $g_B^* \coloneqq \mathbf{u}_*^\top \mathbf{Z}_B$  and  $g_D^* \coloneqq \mathbf{u}_*^\top \mathbf{Z}_D$  we obtain (using Lemma B.2 in Appendix B.2)
Now let  $\tilde{\mathbf{u}}$  be a unit vector orthogonal to  $\mathbf{u}_{*}$ . Let  $\tilde{g} = \tilde{\mathbf{u}}^{\mathrm{T}}\mathbf{Z}$  be  $N(0,1)$ . Also, let  $\tilde{g}_a = \tilde{\mathbf{u}}^\top \mathbf{Z}_a$  for  $a\in \{0,1\}$ . Since  $\mathbf{Z}_a$  are given by conditioning  $\mathbf{Z}$  only along  $\mathbf{u}_{*}$ ,  $\tilde{g}_a\sim N(0,1)$  for  $a\in \{0,1\}$ . In particular, the component along  $\tilde{u}$  of  $\mathbf{Z}_B$  (call it  $\tilde{g}_B$ ) is  $N(0,1)$  and that of  $\mathbf{Z}_D$  (call it  $\tilde{g}_D$ ) is the difference of two iid  $N(0,1)$  variables. Thus,  $\operatorname {Var}[\tilde{g}_B] = 1$  and  $\operatorname {Var}[\tilde{g}_D] = 2$ . Moreover, due to orthogonality all these gaussian variables corresponding to  $\tilde{\mathbf{u}}$  are independent of those corresponding to  $\mathbf{u}_{*}$  defined earlier.
Now let  $\mathbf{u} = \alpha \mathbf{u}_{*} + \beta \tilde{\mathbf{u}}$ , where  $\beta = \sqrt{1 - \alpha^2}$  be any unit vector. From the above we have,
$$
\begin{array}{l} \frac {\mathrm {V a r} \left[ \mathbf {u} ^ {\mathsf {T}} \mathbf {Z} _ {D} \right]}{\mathrm {V a r} \left[ \mathbf {u} ^ {\mathsf {T}} \mathbf {Z} _ {B} \right]} = \frac {\mathrm {V a r} \left[ \alpha g _ {D} ^ {*} + \beta \tilde {g} _ {D} \right]}{\mathrm {V a r} \left[ \alpha g _ {B} ^ {*} + \beta \tilde {g} _ {B} \right]} = \frac {\alpha^ {2} \mathrm {V a r} \left[ g _ {D} ^ {*} \right] + \beta^ {2} \mathrm {V a r} \left[ \tilde {g} _ {D} \right]}{\alpha^ {2} \mathrm {V a r} \left[ g _ {B} ^ {*} \right] + \beta^ {2} \mathrm {V a r} \left[ \tilde {g} _ {B} \right]} \\ = \frac {2 \alpha^ {2} (1 - \kappa_ {1}) + \alpha^ {2} \kappa_ {2} + 2 \beta^ {2}}{\alpha^ {2} (1 - \kappa_ {1}) + \beta^ {2}} \\ = 2 + \frac {\alpha^ {2} \kappa_ {2}}{1 - \alpha^ {2} \kappa_ {1}} \tag {7} \\ \end{array}
$$
Lemma 4.5 shows that ratio maximization can be treated as an eigenvalue decomposition problem of the matrix  $\boldsymbol{\Sigma}_B^{-1/2}\boldsymbol{\Sigma}_D\boldsymbol{\Sigma}_B^{-1/2}$ . We now prove that the error in the estimate of  $\hat{\mathbf{r}}$  given to us by the algorithm is bounded if the error in the covariance estimates are bounded. The sample complexity of computing these estimates gives the sample complexity of our algorithm.